Pseudo Relevance Feedback with Deep Language Models and Dense Retrievers: Successes and Pitfalls

Pseudo Relevance Feedback (PRF) assumes the top-ranked passages from any phase of retrieval contain relevant signals and thus modifies the query by exploiting these signals in a bid to reduce the effect of query-passage vocabulary mismatch and improve search effectiveness (Azad and Deepak, 2019). Previous research has considered PRF in the context of traditional bag-of-words retrieval models such as probabilistic (Robertson and Zaragoza, 2009), vector space (Rocchio, 1971), and language models (Lv and Zhai, 2014; Zhai and Lafferty, 2001; Lavrenko and Croft, 2017). PRF methods such as Rocchio (Rocchio, 1971), relevance models (Lavrenko and Croft, 2017), RM3 (Lv and Zhai, 2009), and KL expansion models (Zhai and Lafferty, 2001) analyse the top-ranked passages to expand the query or to modify the query weights. The query and the passage are represented as either text or vectors, hence the categorisation of text-based and vector-based PRF approaches hereafter. Empirically, these approaches improve the initial retrieval effectiveness (Azad and Deepak, 2019).

Recently, Transformer (Vaswani et al., 2017) based deep language models (Devlin et al., 2019; Yang et al., 2019a; Dai et al., 2019; Radford et al., 2019; Raffel et al., 2020) have been adopted with promising results in information retrieval (Yates et al., 2021; Guo et al., 2020). Seminal in this context is the work of Nogueira and Cho (2019) who fine tuned BERT (Devlin et al., 2019) as a reranker on top of BM25. In this article, we investigate how to integrate PRF signals, effective for bag-of-words models, with deep language model rerankers, e.g. BERT (other models such as RoBERTa (Liu et al., 2019), query likelihood models (Zhuang et al., 2021; Zhuang and Zuccon, 2021; dos Santos et al., 2020) can be applied as well), and dense retrievers, (specifically ANCE (Xiong et al., 2020), RepBERT (Zhan et al., 2020), TCT-ColBERT V1 (Lin et al., 2020), TCT-ColBERT V2 HN+ (Lin et al., 2021c), DistilBERT KD (Hofstätter et al., 2020), DistilBERT Balanced (Hofstätter et al., 2021), and SBERT (Reimers et al., 2019)); extensive evaluations are done towards the proposed PRF methods along the side.

Our experiments investigate two alternative paths to integrate PRF signals with deep language models: text-based and vector-based. The text-based PRF approach is an obvious direction as the concatenation of the query text and the PRF passages text is used as the new formulated query to feed into the deep language model (e.g., BERT). However, this approach has two significant impediments: (i) the lengthy concatenated text would often exceed the allowed input size (input vector length) of these deep language models (Yang et al., 2019b; Ding et al., 2020) and (ii) it is computationally expensive or infeasible as it requires additional deep language model inferences at query time (Jiao et al., 2020; Liu et al., 2020). To solve the first challenge, we propose three different text handling methods to generate text partitions from the full concatenated text such that each of the partitions is within the length limit of the deep language models. Furthermore, because we split the concatenated text into partitions, we also propose three different score aggregation methods (Average, Borda, and Max) to aggregate the scores from each partition to calculate the final scores for each passage.

To address the computational complexity challenge, we use model pre-generated embeddings to represent text (Khattab and Zaharia, 2020; Zhan et al., 2020; Xiong et al., 2020; Qu et al., 2020). Query latency is reduced to the time of generating the query embeddings because the passages embeddings are pre-generated. In the context of PRF, we further utilise these pre-generated passages embeddings to efficiently integrate the relevance signals while eliminating the input size limit of deep language models, which we refer to as vector-based PRF approach. Each feedback passage is pre-generated as embeddings (vectors) in this approach. We adopt two different vector fusion methods (Average and Rocchio) to integrate the feedback vectors into the query vectors. The Rocchio method has two parameters: the query vector and the feedback passage vector weights. We empirically investigate the influence of query and feedback passages through weighting within the Rocchio PRF approach.

To evaluate these PRF approaches, we use the TREC Deep Learning Track Passage Retrieval Task (TREC DL) 2019 (Craswell et al., 2020) and 2020 (Craswell et al., 2021), the TREC Conversational Assistance Track 2019 (TREC CAsT) (Dalton et al., 2020), the Web Answer Passages (WebAP) (Keikha et al., 2014), and the DL HARD (Mackie et al., 2021). TREC CAsT and WebAP are used for the passage retrieval task rather than their original tasks (e.g., for CAsT, we do not consider the multi-turn conversational relationship between queries).

For the text-based PRF approach, we find that our models significantly outperform the baselines across several evaluation metrics on TREC DL 2019 while having mixed results on TREC DL 2020, TREC CAsT and WebAP. For DL HARD, the proposed approach does not have any significant improvements. The results suggest that TREC DL 2019 queries are easier – the results from the initial ranking contain less noise – hence, the PRF can add more relevant information to the queries. On the other hand, the queries of TREC DL 2020, TREC CAsT, and WebAP are more challenging —the results from the initial ranking contain more noise—hence, adding these PRF signals into the query will cause query drift and lead to worse performance. DL HARD is created by selecting queries from TREC DL 2019 and TREC DL 2020 based on the performance systems at TREC had (i.e., select queries for which systems cannot perform well) and the characteristics of the queries (Mackie et al., 2021). Our results show that text-based PRF did not work on DL HARD, suggesting that the feedback passages do not contain relevant signals or more noise than valuable signals.

Another challenge for text-based PRF is its computational complexity for the full ranking pipeline. It requires at least two inferences, depending on the text partitioning method. At least, this doubles the total run time compared to that of deep language rerankers without PRF.

For the vector-based PRF approach, we find that our models improve the respective baselines (seven dense retrievers) across all evaluation metrics and all datasets for the retrieval task; the proposed approach also outperforms the strong BM25+BERT ranker across several metrics. This result suggests that encoding the PRF feedback passages into embedding vectors better models the relevance signals exploited by the PRF mechanism. Unlike text-based PRF, the passage vectors are pre-generated and indexed, so the inference steps on passages are not required at retrieval or rerank time. This makes vector-based PRF very efficient: it takes only 1/20th of the time of the BM25+BERT reranker and only about double the time of the simple bag-of-words BM25. In addition, since our proposed approach works directly with the vector embeddings of queries and passages, they can be applied on top of any choice of dense retriever. For the reranking task, we find that our models outperform the BM25 and BM25+RM3 baselines across all metrics and datasets, while they have only mixed improvements over the strong BM25+BERT reranker. Overall, the vector-based PRF approach for retrieval tends to improve deep metrics, while for reranking, they tend to improve shallow metrics.

To summarize in this article we make the following contributions:

• •

  We thoroughly investigate the PRF effectiveness under different conditions, in particular how sensitive the effectiveness is to PRF depth, text handling/vector fusion, and score estimation;

• •

  We conduct a thorough comparison of text-based and vector-based approaches within the same reranking task;

• •

  We conduct a thorough comparison of different vector-based approaches within the same retrieval task;

• •

  We study the efficiency of the proposed text-based and vector-based PRF approaches.

Pseudo-Relevance Feedback (PRF) is a classic query expansion method that modifies the original query in an attempt to address the mismatch between the query intent and the query representation (Clinchant and Gaussier, 2013; Wang et al., 2020b). A typical PRF setting uses the top-ranked passages from a retrieval system as the relevant signal to select query terms to add to the original query or to set the weights for the query terms. PRF approaches including Rocchio (Rocchio, 1971), KL expansion (Lv and Zhai, 2014; Zhai and Lafferty, 2001), query-regularized mixture model (Tao and Zhai, 2006), RM3 (Lv and Zhai, 2009), relevance-feedback matrix factorization (Zamani et al., 2016), and relevance models (Lavrenko and Croft, 2017) have been well studied. The use of PRF on top of efficient bag-of-words retrieval models is common in information retrieval systems, and it is an effective strategy for improving retrieval effectiveness (Clinchant and Gaussier, 2013; Lavrenko and Croft, 2017). Traditional PRF approaches (Rocchio, 1971; Zhai and Lafferty, 2001; Lv and Zhai, 2009; Lavrenko and Croft, 2017) are simple, but more effective, robust and generalisable, in comparison to more complex models (Tao and Zhai, 2006; Zamani et al., 2016), which instead achieve marginal gains, may be harder to implement/reproduce or maybe problematic to instantiate across different datasets or domains from those in which they have been originally evaluated. This study employs the most popular PRF method in existing research (RM3) as a baseline. The RM3 considers the original query and the feedback passages when creating a new query by assigning different weights to the original query and feedback terms. RM3 is effective and robust compared to other query expansion methods (Lv and Zhai, 2009), and it is used as a baseline in several pseudo relevance feedback studies (Miao et al., 2012; Lv and Zhai, 2010; Cao et al., 2008).

Recent research has studied PRF in different settings. Lin (2019) considered document ranking as a binary classification problem, combining PRF with text classification by introducing positive and negative pseudo labels. The positive pseudo labels are obtained from the top-$k$ documents, while the negative labels are from the bottom-$n$ documents. The final score is a linear interpolation of the classifier and retriever scores. Li et al. (2018) proposed a neural PRF framework, which was further extended by Wang et al. (2020a), that utilises a feed-forward neural network to determine the target document’s relevance score by aggregating the target query and the target feedback relevance scores. However, these proposed models have achieved marginal improvements over the BM25 baseline. Furthermore, the efficiency (run time) of the proposed models have not been reported, and thus it is difficult to establish whether these marginal improvements in effectiveness may be at the cost of efficiency.

Deep language models based on transformers (Vaswani et al., 2017), such as BERT (Devlin et al., 2019), T5 (Raffel et al., 2020), and RoBERTa (Liu et al., 2019), has surpassed the existing state-of-the-art effectiveness in different search tasks. BERT, in specific, has shown to improve over previous state-of-the-art for ad hoc retrieval (Nogueira and Cho, 2019). Recent research has also considered integrating PRF with deep language models. Padaki et al. (2020) integrated RM3 with BERT. The results, however, showed that the selection of highly weighted terms from the feedback passages via RM3 to expand the original query could significantly hurt the ranking quality of a fine-tuned BERT reranker. Yu et al. (2021a) presented a framework that integrates PRF into a Graph-based Transformer (PGT). It represents each feedback passage as a node, and the PRF signals are captured using sparse attention between graph nodes. While this approach handles the input-size limit of deep language models, it achieves marginal improvements compared to the BERT reranker approach across most evaluation metrics at the cost of efficiency. Specifically, compared to our results, PGT achieves a lower nDCG@10 than our simplest text-based PRF reranking approach; it also achieves lower effectiveness in reranking and similar effectiveness in retrieval than our vector-based PRF with ANCE, but at a much higher computational cost.

Wang et al. (2020b) argued that existing PRF research mainly considers relevance matching where terms are used to sort feedback documents. On the contrary, they propose a model that considers both relevance and semantic matching. The relevance score is obtained using BM25. For semantic matching, they split the top-$k$ PRF documents into sentences. For each sentence, they use BERT to estimate the semantic similarity with the query. Scores from the top-$m$ sentences of each document are considered as the semantic score for this document. The final scores of each document are calculated from a linear interpolation of the relevance and semantic scores. The expansion terms are then extracted from the reranked top-$k$ PRF documents and added to the original query for a second retrieval stage. Although the improvements are marginal, they demonstrate that BERT can identify relevance signals from the feedback documents at the sentence level to enhance retrieval effectiveness. However, this marginal improvement is at the expense of efficiency because expansion terms are identified through BERT.

Zheng et al. (2020, 2021) presented a three-phase BERT-based query expansion model: BERT-QE. The first phase is a standard BERT reranking (Nogueira and Cho, 2019) step. In the second phase, the top-$k$ passages are selected as feedback passages, further split into overlapping partitions using a sliding window. Together with the original query, these partitions are fed into BERT to get the top-$m$ partitions with the highest scores per passage. The top-$m$ partitions and the candidate passage are fed into BERT in the third phase. The score of a candidate passage in this phase is calculated as a weighted sum, where the weight is the relevance score of each partition in the top-$m$ partitions from phase two, and the score is the relevance score between the top-$m$ partitions and the candidate passage. The final score of a candidate passage is calculated by linear interpolation of the first phase BERT relevance score between the query and the passage, and the third phase weighted sum score between the top-$m$ partitions and the candidate passages. Although BERT-QE achieves significant improvements in effectiveness over BERT reranker, it requires 11.01x more computations than BERT, making it computationally infeasible in many practical applications.

Recently, Yu et al. (2021b) proposed a PRF framework based on ANCE (Xiong et al., 2020), which trains a new query encoder from ANCE that takes in the top-$k$ passages from the first-round ANCE retrieval, then concatenate the passage texts with the original query text to form the new PRF query, without changing ANCE’s passage encoder. This newly formed PRF query is passed to the trained query encoder to produce the PRF query representation, and to retrieve the results from the original passage collection index. Although the improvements are significant across several datasets over different metrics, according to a recent reproducibility paper from Li et al. (2022c), the proposed model does not generalise well to other dense retriever models and the training process needs to be adjusted accordingly with different dense retriever models, which makes it difficult to achieve the same effectiveness as the one proposed in the original paper.

Integrating PRF signals with deep language models implies a trade-off between effectiveness and efficiency. While current approaches ignored efficiency, the majority still achieved marginal improvements in effectiveness. In this study, we propose three approaches to integrate PRF signals to improve effectiveness while maintaining efficiency: (i) by concatenating the feedback passages text with the original query to form the new queries that contain the relevant signals, (ii) by pre-generating passage collection embeddings and performing PRF in the vector space, because embeddings promise to capture the semantic similarity between terms (Dalton et al., 2019; Diaz et al., 2016; Kuzi et al., 2016; Naseri et al., 2018; Roy et al., 2016; Zamani and Croft, 2016, 2017; Naseri et al., 2021), which makes it feasible as a method for first stage retrieval as well, (iii) by combining the previous two approaches into a hybrid approach.

The overarching research question we seek to answer with our experiments is: What are the successes and pitfalls of integrating PRF into deep language model rerankers and dense retrievers in terms of effectiveness and efficiency? Each of the following subsections addresses a more specific sub-question.

To complement our analysis of the results and separate from our core research questions, we further analysed the results for a subset of the queries to gain a qualitative understanding of when, and possibly why, the PRF methods work – or don’t. For this, we limit our analysis to the retrieval task and the ANCE and Vector-based PRF with Rocchio with $k=5$.

Figure 14(top) presents a query-by-query analysis of the gains and losses of Vector-based PRF (Rocchio) with respect to ANCE on the combined set of TREC DL 2019 and 2020 queries. The effectiveness measure used is nDCG@5 and gains and losses are represented by bars in the figure. We used nDCG@5 for this analysis and not the rank cut-offs we have reported for other analyses to align the evaluation cut-off with the cut-off used for the feedback signal, i.e. the PRF depth $k$, which was also 5. The observations that follow, based on nDCG@5, are also found when other rank cut-offs are considered.

As one would expect, there are a number of queries for which the PRF method experiences losses, and other queries for which gains are observed; interestingly there is a considerable amount of queries for which neither gains nor losses are found. We further verified whether the encoding of the query obtained via PRF is different from that of the original query. This is done especially for the cases in which neither gains nor losses are observed: does this happen because there is no difference between the representation of the original query and of the query after PRF? We performed this analysis by comparing the two dense representations using the inner product between the encoding vectors. We found that the similarity between the initial query encoding and the one obtained with PRF varies across queries and that queries for which neither gains nor losses are obtained, are no different in terms of difference between query representations compared to queries that exhibit gains or losses (figure not shown here, but available in the online appendix⁶⁶6 https://github.com/ielab/Neural-Relevance-Feedback-Public). In other words, the amount by which PRF changes the initial query representation is not directly associated with a gain or a loss, and queries for which no difference in effectiveness is found are often characterised by differences between the original and PRF representations of the query.

For each query, Figure 14 (top) also reports the amount of overlap between the top $k$ passages retrieved by the first stage of retrieval (ANCE) – this is effectively the signal that is used as input to the PRF method – and the top $k$ passages retrieved by the PRF method. The overlap is reported in percentage of the top $k$ passages that are in common between the two sets; $k=5$ and thus an overlap of $80\%$ means that 4 out of 5 passages are in common. This statistic helps us to analyse whether the PRF method ended up just re-ranking the top $k$ passages from the first stage, or it did push into the top $k$ ranks passages that were not there before. Losses in effectiveness for the PRF method tend to be associated with re-ranking of the top $k$ passages from the first stage of retrieval: the average overlap for these queries is 85.2%. And thus, these losses are not caused by retrieving less of the relevant passages, or passages that are less relevant (e.g. marginally relevant instead of highly relevant); they are instead caused by re-ordering the passages from the first stage in a less effective manner. On the other hand, gains instead have a higher tendency to be associated with the ability of the PRF method to bring into the top $k$ additional, new relevant passages compared to what the first stage of retrieval could do: the average overlap for these queries is 82.8%. Differences, however, are not statistically significant (Pearson’s Chi-squared test).

Figure 14 (mid) presents an analysis of the relevancy of the top $k$ passages in the first stage of retrieval, while Figure 14 (bottom) presents an analysis of the relevancy of the top $k$ passages after PRF. Remember that the relevancy of the top $k$ passages in the first stage of retrieval in all effects represents the quality of the feedback signal that is passed as input to the PRF method. Each stacked column refers to a specific query, and the column is aligned to the gain-loss plot (Figure 14 (top)) so that it is possible to directly compare the relevancy of the PRF signal and the gain or loss produced by the PRF method. No obvious pattern appears at first sight. PRF signals that contain non-relevant documents produce results that can go either way: at times losses are obtained, but it is also common to obtain gains (or even have no effect at all). However, a more scrupulous examination of this signal leads to an interesting observation. We do this by examining the five queries with largest losses, and then the five queries with the largest gains for which there is at least one non-relevant passage. We observe the following. Non-relevant passages for queries that display losses are largely off topic: they have a very weak relationship to the query, if at all. For example, for query 1110678: what is the un fao, the PRF signal includes 4 non-relevant passages, and these passages are related to the card game Uno; they have nothing to do with the United Nations FAO program. A subset of these passages (along with other examples) are presented in Table 6. Another similar example is query 169208: does mississippi have an income tax, for which the PRF signal includes 3 non-relevant passages and, while they are related to tax and the state of Mississipi, they are about unemployment or start-up tax and thus not relevant to the need of information about the more general income tax. When this type of feedback is used in the PRF mechanism, the new query representation displays query drift, and it does score higher passages that are non-relevant, but are related to those in the feedback signal (e.g., for query 1110678 many top retrieved passages by PRF are related to Uno, even beyond the top $k$). This situation instead does not occur when we examine the PRF signal of queries for which PRF provides a gain. In particular, while this signal does still contain non-relevant passages, the non-relevancy nature of these passages is different from that of passages for which losses are found. Let us unwrap this with actual examples. The PRF method exhibits a large gain for query 1136047: difference between a company’s strategy and business model is, despite the PRF signal for this query displays several non-relevant passages. These non-relevant passages, shown in Table 6, however, do display a strong aboutness relation (Maron, 1977) with the query, for example by mentioning the key terms of the query – and when this is used in the PRF mechanism it would help reinforce the query representation. We also note that the non-relevant label assigned to some of these passages by the original TREC assessors may be somewhat questioned: for example passage 8724036 for query 1136047 appears to us as at least marginally relevant. Other example queries that display a pattern similar to that of query 1136047 are included in Table 6.

Integrating PRF with deep language models has two main challenges: computational cost and input size limit, which can be roughly mapped to efficiency and effectiveness, respectively. Previous research has focused mainly on effectiveness, and applying PRF as a second stage that considers a subset of the data collection, to mitigate the computational cost. For instance, Zheng et al. (2020) proposed a PRF framework which is divided into three phases, each involving BERT, to rerank passages and chunks of text after the initial retrieval. Similarly, Wang et al. (2020b) applied PRF to the second stage retrieval, utilising BERT for reranking sentences. Other work that applied PRF to the first stage retrieval has limited the second phase of retrieval to a subset of the collection, either to depth of 500 (Yu et al., 2021a) or 1000 (Li et al., 2018). We argue that the main limitation with these approaches is related to the first stage retrieval model they employed — if the first stage is basic bag-of-words then the overall method inherits many of this models limitations.

To address the applicability, effectiveness, and efficiency of text-based, vector-based and hybrid PRF approaches, we compared them over two different tasks, retrieval for vector-based PRF and reranking for both text-based and hybrid PRF. While the text-based and hybrid PRF approaches handle the input size limit, we showed that they enhance the ranking effectiveness only marginally. This finding aligns with previous research, but we demonstrated it happens at the cost of efficiency. The BERT model is slow (ranking 1,000 passages for each query takes 9,209 ms), rendering text-based and hybrid PRF approaches infeasible and inapplicable to real-time search engines for this task, as shown in Figure 13. On the other hand, our proposed vector-PRF approach is substantially more efficient: it only takes 163-174 ms to process one query. In addition, it does so while also improving the effectiveness of dense retrievers in terms of all metrics compared to ANCE and RepBERT (for retrieval), and in terms of shallow metrics compared to BERT (for reranking).

We also apply the BERT reranker on top of Vector-based PRF approaches. Although this indeed improves effectiveness across all evaluation metrics, only improvements on MAP are significant on all datasets, and R+PRF+B significantly improves nDCG@10 on WebAP only. On the other hand, R@1000 is significantly improved across several datasets, but this improvement comes from the gain produced by the Vector-based PRF compared to the respective dense retrievers.

The BERT reranker used as a strong baseline in our experiments is an off-the-shelf model, originally produced by Nogueira and Cho (2019). This model is fine-tuned on the training set of the MS MARCO Passage Retrieval Dataset. The query length in this training set is much shorter than the newly formed queries from the text-based PRF approaches. This causes a mismatch between training and testing data. Theoretically, if the testing data is significantly different from the trained data data used to fine-tune the model, the model itself will be less effective. Despite this, in our experiments we found that the formed long PRF queries still significantly improve results for some metrics across several datasets (although improvements are often only marginal) – but in the majority of cases their effectiveness is worse than that of BM25+BERT. It will then be interesting to investigate whether the effectiveness of text-based PRF can be further improved by fine-tuning the BERT model with long PRF queries, hence eliminating the training/testing data mismatch. On the other hand, vector-based PRF is not affected by this issue, because the query vector is of fixed length: no matter how many PRF passages are added to the original query vector, the query vector formed via PRF maintains the same length, and thus no training/testing data mismatch occurs. Another advantage of vector-based PRF over the text-based is that, although its effectiveness might be improved significantly after fine-tuning on long PRF queries, the text-based PRF approach is still inefficient and it is infeasible to apply it to real-time settings.

As complementary experiments, we also executed the retrieval task and re-ranking task on the MS MARCO dev queries. We omit the detailed results for these experiments from the manuscript in the interest of space and focus; however we make them available in the online appendix⁸⁸8https://github.com/ielab/Neural-Relevance-Feedback-Public/blob/master/Vector_Based_PRF/README.md and we provide a summary of the observations next.

For both retrieval and re-ranking tasks on the MS MARCO dev queries, we found no improvement over the non-PRF baselines. This is unlike in the TREC DL datasets, where improvements are observed. We believe that the reason for this difference is to be found on the largely sparse annotated nature of MS MARCO vs. the more complete relevance annotations in TREC DL. One may argue that other neural PRF methods, such as ANCE-PRF, have shown improvements on MS MARCO too, and not just on TREC DL. However, we stress an important difference. Both the baseline dense retrievers and the learnt PRF methods (i.e. ANCE-PRF and the likes) are trained so as to build representations of the single “gold” (i.e. relevant) MS MARCO passage that are close to the query (for ANCE-PRF: representations of the query after PRF that are close to the gold MS MARCO passage). Thus, in the context of trained PRF methods like ANCE-PRF, the query representation is changed to become closer to the single MS MARCO target gold passage. Our PRF method changes the query representation (and consequently perturbs the ranking) based on the representation of the top $k$ passages alone: there is no notion of a gold passage towards which such new representation should be moved. This means these non-learnt vector-based PRF methods are not suitable to situations in which a single golden passage exists, for example for the task of known item retrieval – and this is shown by the MS MARCO results too. However, this does not mean these methods are not suitable when there is an array of relevant passages to be retrieved; and this in fact is what the TREC DL results show.

This article investigated the integration of PRF within transformer-based deep language models and dense retrievers for retrieval and reranking. In this context, a text-based PRF approach, applicable to the reranking task only, and a vector-based PRF approach, applicable to both retrieval and reranking tasks, were proposed to leverage the relevance signals from the feedback passages. The proposed approaches are based on the BERT reranker and the dense retrievers ANCE and RepBERT. In this context, we studied the impact of vector representation (for vector-based PRF), PRF depth, text handling techniques (for text-based PRF), and different score estimation (vector fusion) methods (for vector-based PRF) .

When analysing results for the vector-based PRF approaches, we found they differed depending on whether we considered the task of retrieval or reranking.

In terms of which vector-based representation is most effective for retrieval, we empirically found that performing PRF using ANCE as a representation (A+PRF-A) is better than when using RepBERT (R+PRF-R) across metrics and datasets, with the exception of R@1000. Indeed, our results show that R+PRF-R tends to improve deep metrics, while A+PRF-A tends to improve shallow metrics. When considering the reranking task, substantial improvements only occurred on a few datasets and for a limited amount of metrics, and overall ANCE-based PRF performs better than RepBERT-based.

In terms of the impact of depth of PRF signal for vector-based PRF, we found that when retrieval is considered, substantial improvements are recorded when 3 to 5 passages are given as feedback, while deeper feedback (10 passages) hurts effectiveness most of the times. When reranking is considered, PRF depth is not a factor substantially influencing effectiveness in a consistent manner across datasets and metrics.

In terms of score estimation methods (vector fusion) for the vector-based PRF, we found that, for the retrieval task, $\mathcal{RC}_{\beta}$ and $\mathcal{RC}_{\alpha,\beta}$ achieve the best effectiveness across all datasets. Moreover, R+PRF-R often performs best when using either approximately evenly distributed weights between query/feedback passages, or when the query retains most of the weight. This is similar for A+PRF-A, although for this method there are cases where the best effectiveness is achieved by giving a higher weight to feedback passages. When the task of reranking is considered, we found that $\mathcal{RC}_{\alpha}$ and $\mathcal{RC}_{\alpha,\beta}$ record the most improvements. Substantial improvements, however, only occur on TREC DL 2019 and for only MAP, RR, and nDCG@1.

To validate the generalisability of our proposed Vector-based PRF models, we considered five additional dense retrievers (TCT-ColBERT V1, TCT-ColBERT V2 HN+, DistilBERT KD, DistilBERT Balanced, and SBERT) other than ANCE and RepBERT. Experimental results showed that our vector-based PRF approaches consistently improved the effectiveness over deep evaluation metrics.

The text-based PRF approaches were only applicable to the reranking task. In terms of depth of PRF, we found that, in most cases, substantial improvements are achieved when using 3, 5, and 10 passages for PRF. Deeper PRF signals (15 and 20 passages) were found to hurt effectiveness: this is likely because the addition of more feedback passages substantially contributed to query drift. In terms of text handling methods, we found that CA performs best in most cases, but only marginal improvements over the baselines are recorded across datasets and metrics.

We also experimented with applying the BERT reranker on top of Vector-based PRF models: this improves the effectiveness for all evaluation metrics, but only significantly for MAP. The significant improvements on R@1000 are purely produced by the Vector-based PRF models: the reranking step does not contribute to these improvements.

Vector-based and text-based PRF approaches could be compared only in the reranking task. In terms of effectiveness, neither of these two types of approaches clearly and consistently outperformed the other across datasets and metrics. However, they greatly differed in terms of efficiency (query latency). When the vector-based PRF was used for retrieval, latency was close to that of the bag-of-words PRF method BM25+RM3, and about double that of BM25. When used for reranking, the latency of the vector-based PRF remained the same. However, to this latency one needs to add that of the preceding part of the retrieval/ranking pipeline: in the case of our experiments that was the latency of BM25+BERT, which was substantial. On the other hand, the latency of text-based PRF was substantially higher. To the latency of the initial retrieval/ranking pipeline, in fact, the text-based PRF added the latency associated with the text handling methods (e.g., Concatenate and Aggregate) – and this operation involved the execution of many more, costly BERT inferences, rendering the overall latency prohibitive for real-time search engines. The extensive experiments provided in this paper serve as a guide for using state-of-the-art neural retrieval models into low latency environments.

This work opens up a number of avenues for future research. First, we note that the Vector-based PRF method, and especially the Rocchio approach, has a number of limitations that future work could address. For example, Rocchio has parameters ($\alpha$, $\beta$) that need to be properly tuned to achieve the highest performance; we performed this tuning with respect to a validation set. How these parameters could instead be set on a per query basis (e.g., using some form of weight prediction) is still unclear.

Second, the use-case analysis in Section 6 revealed that the PRF methods are susceptible to the quality of the PRF signal (i.e. the passages retrieved by the first stage method and then passed as input to PRF). Recent work has made initial inroads to investigate this aspect. Specifically, Li et al. have studied the effect of signals of different qualities on the PRF process (Li et al., 2022a) – although they only explored a limited set of possible signals and PRF methods. Zhuang et al. have devised effective methods to de-noise the feedback signal in the case of implicit feedback (Zhuang et al., 2022). The extent to which different PRF methods differ in terms of susceptibility to PRF signals of different qualities is still however unclear, especially beyond the specific methods that we examined in the use-case analysis and the signals and settings explored in the aforementioned recent work.

Third, this paper only considered the passage retrieval and ranking tasks. PRF techniques have, however, long been explored for document retrieval tasks and one wonders how the methods designed here for passages, and that exploit pre-trained language models, generalise when considering documents. We note that adapting these specific passage PRF methods to documents is not straightforward. Documents are sensibly longer in length than passages, and most likely their text exceeds the maximum input length that pre-trained language models like BERT can consider. This is certainly a problem for models like ANCE-PRF and our text-based PRF methods, where the text of the PRF signal needs to be encoded. But, this is also a problem for our Vector-based PRF methods: while the text associated to the PRF signal is not required by the PRF process itself, these documents need still to be encoded so that their dense vector can be used (for the initial round of retrieval, and for the PRF). Tackling this limitation of pre-trained language models for effective document retrieval is not new (Yang et al., 2019b; Yilmaz et al., 2019a, b; Mass and Roitman, 2020)– progress so far in this space has not yet solved this problem (especially if also efficiency considerations are made) and PRF in this context is largely unexplored (Wang et al., 2020b; Xu et al., 2009; Wang et al., 2020a).

Fourth, PRF is typically integrated in wider pipelines, including multiple rankers and a multi-stage ranking architecture. A recent trend with neural rankers is to create hybrid systems that comprise a dense and a sparse rankers (Wang et al., 2021; Arabzadeh et al., 2021; Lin et al., 2021b). In this paper, we have not considered such architecture type, and how PRF could be integrated. We however refer the interested reader to the recent work of Li et al. that has extended our methods to hybrid rankers (Li et al., 2022b).

The implementations of our PRF methods are made publicly available at http://ielab.io/publications/li-tois-2022, along with the full empirical results.