PAIR: Leveraging Passage-Centric Similarity Relation for
Improving Dense Passage Retrieval

The task of dense passage retrieval Karpukhin et al. (2020) is described as follows. Given a query $q$, we aim to retrieve $k$ most relevant passages $\{p_{j}\}_{j=1}^{k}$ from a large collection of $M$ passages.

For this task, the dual-encoder architecture is widely adopted Karpukhin et al. (2020); Qu et al. (2020), where two separate encoders $E_{Q}(\cdot)$ and $E_{P}(\cdot)$ are used to represent the query $q$ and the passage $p$ into $d$-dimensional vectors in different representation spaces. Then a dot product is performed to measure the similarity between $q$ and $p$ based on their embeddings:

Previous studies mainly capture the query-centric similarity relation. As shown in Figure 1, passage-centric similarity relation reflects important evidence for improving the retrieval performance. Therefore, we extend the original query-centric learning framework by leveraging the passage-centric similarity relation.

To develop our approach, we need to address the issues described in Section 1, and we consider three aspects to extend. First, we design a new loss function that considers both query-centric and passage-centric similarity relations. Second, we utilize knowledge distillation to obtain large-scale and high-quality pseudo-labeled data to capture more comprehensive similarity relations. Third, we design a two-stage training procedure to effectively learn the passage-centric similarity relation and improve the final retrieval performance.

Our approach considers two kinds of losses, namely query-centric loss and passage-centric loss, as shown in Figure 2. The two kinds of losses are characterized by the two different similarity relations, query-centric similarity relation and passage-centric similarity relation.

By optimizing both query-centric loss and passage-centric loss, we can capture more comprehensive similarity relations. However, more similarity relation constraints require large-scale and high-quality training data for optimization. Additionally, there might be a large number of unlabeled positives even in the existing manually labeled datasets Qu et al. (2020), and it is likely to bring false negatives when sampling hard negatives. Hence, we propose to generate pseudo-labeled training data via knowledge distillation.

Although passage-centric similarity relation (Eq. (5)) is able to incorporate additional relevance evidence, it is not directly related to the final task goal (i.e., query-centric similarity relation). Therefore, we design a two-stage training procedure that incorporates the passage-centric loss in the pre-training stage, and then only optimize the tasks-specific loss (i.e., query-centric loss) in the fine-tuning stage. We present an illustration for the two-stage training procedure in Figure 3. Next, we present the detailed training procedure.

We conduct experiments with the deep learning framework PaddlePaddle (Ma et al., 2019) on up to eight NVIDIA Tesla V100 GPUs (with 32G RAM).

We consider both sparse and dense passage retrievers for baselines. The sparse retrievers include the traditional retriever BM25 Yang et al. (2017), and four traditional retrievers enhanced by neural networks, including doc2query Nogueira et al. (2019b), DeepCT Dai and Callan (2019), docTTTTTquery Nogueira et al. (2019a) and GAR Mao et al. (2020). Both doc2query and docTTTTTquery employ neural query generation to expand documents. In contrast, GAR employs neural generation models to expand queries. Different from them, DeepCT utilizes BERT to learn the term weight. The dense passage retrievers include DPR Karpukhin et al. (2020), DPR-E, ANCE Xiong et al. (2020a), ME-BERT Luan et al. (2021), NPRINC Lu et al. (2020), ColBERT Khattab and Zaharia (2020) and RocketQA Qu et al. (2020). DPR-E is our implementation of DPR using ERNIE Sun et al. (2020) instead of BERT, to examine the effects of pre-trained LMs.

Table 2 presents the main experimental results.

(1) We can see that PAIR significantly outperforms all the baselines on both MSMARCO and NQ datasets. The major difference between our approach and baselines lies in that we incorporate both query-centric and passage-centric similarity relations, which can capture more comprehensive semantic relations. Meanwhile, we incorporate the augmented data via knowledge distillation.

(2) We notice that baseline methods use different pre-trained LMs, as shown in the second column of Table 2. In PAIR, we use the ERNIE-base. To examine the effects of ERNIE-base, we implement DPR-E by replacing BERT-base used in DPR as ERNIE-base. From Table 2, we can observe that PAIR significantly outperforms DPR-E, although they employ the same pre-trained LM.

(3) Another observation is that the dense retrievers are overall better than the sparse retrievers. Such a finding has also been reported in prior studies Karpukhin et al. (2020); Xiong et al. (2020a); Luan et al. (2021), which indicates the effectiveness of the dense retrieval approach.

In this section, we conduct ablation study to examine the effectiveness of each strategy in our proposed approach. We only report the results on the NQ, while the results on the MSMARCO are similar and omitted here due to limited space.

Here, we consider five variants based on our approach for comparison:

(a) w/o PSR removes the loss for passage-centric similarity relation in the pre-training stage;

(b) w/o KD removes the knowledge distillation for obtaining pseudo-labeled data and only uses the labeled data (MSMARCO and NQ) for both pre-training stage and fine-tuning stage;

(c) w/ PSR FT adds the loss for passage-centric similarity relation in the fine-tuning stage;

(d) w/o SP uses separate encoders for queries and passages instead of encoders with shared parameters;

(e) w/o PT removes the pre-training stage.

Table 3 presents the results for the ablation study. We can observe the following findings:

$\bullet$ The performance drops in w/o PSR, demonstrating the effectiveness of learning passage-centric similarity relation;

$\bullet$ The performance drops in w/o KD, demonstrating the necessity and effectiveness of the knowledge distillation for obtaining large-scale and high-quality pseudo-labeled data, since the passage-centric loss tries to distinguish highly similar but semantically different passages;

$\bullet$ The performance slightly drops in w/ PSR FT, because passage-centric loss is not directly related to the target task (i.e., query-based retrieval), which suggests that passage-centric loss should be only used in the pre-training stage;

$\bullet$ The performance drops in w/o SP, demonstrating the effectiveness of dual-encoders with shared parameters;

$\bullet$ The performance significantly drops in w/o PT, demonstrating the importance of our pre-training procedure.

The previous results demonstrate the effectiveness of our proposed approach PAIR. Here, we further analyze the effect of passage-centric loss (Eq. (5)) in a more intuitive way. To examine this, we prepare two variants of our approach, namely the complete PAIR and the variant removing the passage-centric loss (Eq. (5)) denoted by PAIR${}_{\neg\text{PSR}}$.

We first analyze how the passage-centric similarity relation (PSR) influences the similarity relations among query, positive passage and negative passage. Figure 4 shows the comparison of PAIR and PAIR${}_{\neg\text{PSR}}$ for computing the similarities of $s(p^{+},p^{-})$ and $s(p^{+},q)$. We obtain $s(p^{+},p^{-})$ and $s(p^{+},q)$ by the averaging the similarity of top 100 retrieved passages for each query in the testing data of Natural Questions. We can see that before incorporating passage-centric similarity relation (PSR), $s(p^{+},p^{-})$ is higher than $s(p^{+},q)$. As a result, the negatives are close to the positives. After incorporating PSR, $s(p^{+},p^{-})$ becomes lower than $s(p^{+},q)$. It indicates that passage-centric loss pulls positive passages closer to queries and push them farther away from negative passages in the representation space. The comparison result is consistent with passage-level similarity relation in Eq. (4).

Next, we further present two examples in Table 4 to understand the performance difference between PAIR and PAIR${}_{\neg\text{PSR}}$. In the first example, the top-1 passage retrieved by PAIR has the same topic “H1N1” as the query. In contrast, the top-1 passage retrieved by PAIR${}_{\neg\text{PSR}}$ has an incorrect but highly relevant topic “H5N1”. Actually, the sentences among the positive passage (retrieved by PAIR) and the negative passage (retrieved by PAIR${}_{\neg\text{PSR}}$) share many common words. Such a negative passage is likely to mislead the retriever to yield incorrect rankings. Hence, these two passages should be far away from each other in the representation space. This problem cannot be well solved by only considering the query-passage similarity in existing studies. Similar observations can be find from the second example. The top-1 passage retrieved by PAIR has the same topic “gall bladder” as the query, while the top-1 passage retrieved by PAIR${}_{\neg\text{PSR}}$ is about “urinary bladder”. These results show that passage-centric similarity relations are particularly useful to discriminate between positive and hard negative passages (highly similar to positive passages).

In this section, we examine the influence of the thresholds on pseudo-labeled data via knowledge distillation, including the data quality and the retrieval performance. We conduct the analyses by using different positive thresholds $s_{pos}$ and negative thresholds $s_{neg}$ (See Section 3.3).

We first manually evaluate the quality of the pseudo-labeled data via knowledge distillation w.r.t. different threshold settings (i.e., the combinations of $s_{neg}$ and $s_{pos}$). For each threshold setting, we randomly select 100 queries, each of which corresponding to a positive passage and a hard-negative passage. In total, we have 4 threshold settings (as shown in Table 5) and 800 query-passage pairs. We ask two experts to manually annotate the query-passage pairs and evaluate the quality of pseudo-labeled data, the Cohen’s Kappa of experts is 0.9. As shown in the first two columns of Table 5, we can observe that when $s_{pos}=0.9$ and $s_{neg}=0.1$, the data quality is relatively good. Additionally, when setting a low value of $s_{pos}$ and a high value of $s_{neg}$, the data quality becomes worse.

The last three columns of Table 5 also present the retrieval performance w.r.t. different threshold settings. When choosing a low value of $s_{pos}$ and a high value of $s_{neg}$, the retrieval performance drops. Hence, our approach is configured with a strict threshold setting ($s_{pos}$ = 0.9, $s_{neg}$ = 0.1) in experiments to achieve good performance.