CAME: Competitively Learning a Mixture-of-Experts Model for First-stage Retrieval

The ensemble of several models has become a standard technique for improving the effectiveness of information retrieval (IR) tasks. It has been studied extensively in the TREC evaluations (Harman, 1995) and is the basis of Web search engines (Ling et al., 2017). Overall, there are two approaches to developing ensemble IR models (Croft, 2002), both with the motivation to improve retrieval performance by combining multiple evidences. One approach is to create models that explicitly utilize and combine multiple sources of evidence from raw data (Fisher and Elchesen, 1972). For example, existing studies have shown that combining information from multiple fields of the document can facilitate relevance estimation (Fisher and Elchesen, 1972). The other approach is to combine the outputs of several retrieval models by voting or weighted summation, where each prediction provides an evidence about relevance (Fox and France, 1987).

Recently, with the development of pre-trained language models (PLMs), a variety of neural retrievers based on different relevance hypotheses have been proposed (Guo et al., 2022; Zhao et al., 2022). Although these retrievers have achieved impressive performance on retrieval benchmarks (Formal et al., 2021; Hofstätter et al., 2021; Zhou et al., 2022; Liu and Shao, 2022), their focuses (i.e., identifying relevance patterns) are distinct due to the differences in the model architecture. To combine the benefits from different retrieval models, the idea of building ensemble retrieval models has been explored by researchers (Kuzi et al., 2020; Gao et al., 2021b; Lewis et al., 2022; Shen et al., 2022). A straightforward way is to train multiple retrievers independently and combine their predicted scores linearly to obtain the final retrieval results (Luan et al., 2021; Kuzi et al., 2020; Chen et al., 2021; Lin and Lin, 2021). Some work has been inspired by the boosting technique (Gao et al., 2021b; Lewis et al., 2022). For example, Gao et al. (2021b) proposed CLEAR to learn a BERT-based retriever from the residual of BM25, and then combined them to build an ensemble retrieval model. Also, the knowledge distillation technique is introduced into building ensemble retrievers, which tries to leverage the capability of companion retrievers to enhance the ensemble results (Shen et al., 2022). For example,  Shen et al. (2022) proposed UnifieR that optimizes a KL divergence loss between the predicted scores of two retrieval models besides the ranking task loss.

The aforementioned ensemble retrievers always show better performance than a single-model retriever in practice (Nguyen et al., 2016; Craswell et al., 2020). However, their component retrievers, no matter trained independently or jointly, are not handled specially according to their strengths and weaknesses in certain relevance patterns with respect to model architectures. In contrast, built on the Mixture-of-Experts framework, our approach encourages the experts to compete with each other and specialize on suitable samples.

The Mixture-of-Experts (MoE) framework (Jacobs et al., 1991) is one of the techniques for building ensemble models. It builds upon the “divide-and-conquer” principle, in which the problem space is divided for several expert models, guided by one or a few gating networks. According to how and when the gating network is involved in the dividing and combining procedures, MoE implementations could be classified into two groups (Masoudnia and Ebrahimpour, 2014), i.e., the mixture of implicitly localized experts (MILE) and the mixture of explicitly localized experts (MELE). MILE divides the problem space implicitly using a tacit competitive process, such as using a special error function. MELE conducts explicit divisions by a pre-specified clustering method before the expert training starts, and each expert is then assigned to one of these sub-spaces.

The MoE framework has been widely used in various machine learning tasks (Yuksel et al., 2012; Zhao et al., 2019), such as natural language processing and computer vision, showing great potential. A major milestone in language modeling applications appears in (Shazeer et al., 2017), which encourages experts to disagree among themselves so that the experts can specialize on different problem sub-spaces. Recently, MoE has also been applied in search tasks. For example,  Nogueira et al. (2018) built an MoE system for query reformulation, where the input query is given to multiple agents and each returns a corresponding reformulation. Then, the aggregator collects the ranking lists of all reformulated queries and gets the fusion results.  Dai et al. (2022) proposed MoEBQA to alleviate the parameter competition between different types of questions. In addition, given the practical needs in product search, where the data comes from different domains, search scenarios, or product categories, the MoE framework is employed in various e-commerce platforms (Zou et al., 2022; Sheng et al., 2021). For example,  Sheng et al. (2021) claimed that different domains may share common user groups and items, and each domain has its unique data. Thus, it is difficult for a single model to capture the characteristics of various domains. To address this problem, they built a star topology adaptive recommender for multi-domain CTR prediction.

Similarly, due to various relevance matching patterns between queries and documents, we propose an MoE-based retrieval model that aims to exploit the expertise of different retrieval models to identify diverse relevance patterns. As far as we know, this is the first work to utilize the MoE framework to first-stage retrieval.

Given a query $q$ and a collection $\smash{\mathcal{C}}$ with numerous documents, the first-stage retrieval aims to find as many potentially relevant documents as possible. Formally, given a labeled training dataset $\smash{\mathcal{D}=\{(q_{i},D_{i}^{+})\}_{i=1}^{N}}$, where $q_{i}$ denotes a textual query, $\smash{D_{i}^{+}}$ is the set of relevant documents for $q_{i}$, and $N$ is the total number of queries. Let $\smash{d_{i}^{+}\in D_{i}^{+}}$ be one of the relevant documents. The essential task is to learn a model from $\mathcal{D}$ that gives high scores to relevant query-document pairs ($\smash{q_{i}}$, $\smash{d_{i}^{+}}$) and low scores to irrelevant ones. Then, all documents in the collection $\smash{\mathcal{C}}$ can be ranked according to the predicted scores. In the rest of this paper, we omit the subscript $i$ for a specific query if no confusion is caused.

To capture diverse relevance matching patterns, we include three representative retrievers in our MoE model, which are lexical, local, and global matching experts. Due to the efficiency constraint in first-stage retrieval, we adopt the typical bi-encoder architecture (Bromley et al., 1993) that encodes the query $q$ and the document $d$ separately, and then employs a simple interaction function to compute their relevance score based on the obtained representations:

where $\operatorname{Enc_{q}}$ and $\operatorname{Enc_{d}}$ denote the encoder modules for the query and document respectively, $\operatorname{f}$ is the interaction module, and the whole model is parameterized with $\bm{\theta}$. Following (Xiong et al., 2020; Zhan et al., 2021), the two encoders in CAME have the same architecture and shared parameters. For each encoder, the bottom layers are shared for all the experts and the upper layers are private for them, as shown in Figure 2 (a). The relevance score of each query-document pair is calculated based on all the component expert predictions with a result fusion module, as shown in Figure 2 (b). We will elaborate on each part next.

In the real world, human experts are often trained through different stages of education, i.e., standardized training to gain general knowledge and specialized training to obtain expertise in a specific domain. Inspired by this process, our competitive learning mechanism has two similar stages, as shown in Figure 2 (a). Remarkably, our specialized training is through competition.

We first introduce the basic training objective of general retrievers as a preliminary. They usually sample a set of negatives $\smash{D^{-}}$ from $\smash{\mathcal{C}}$ for a given relevant query-document pair $(q,\smash{d^{+}})$ in the training data. For $q$ and any $\smash{d\!\in\!\{d^{+}\}\!\cup\!D^{-}}$, the retriever parameterized by $\bm{\theta}$ computes the relevance score $s(q,d;\bm{\theta})$ with Eq. (1). Then, the probability distribution over the documents $\{\smash{d^{+}}\}\!\cup\!\smash{D^{-}}$ is defined as:

Finally, a contrastive learning loss is used to optimize the retriever:

As in (Karpukhin et al., 2020), in-batch negatives are generally used to facilitate the retriever learning. This objective is the basis for our two-stage competitive learning, which we will introduce next.

Standardized Learning Stage. At the beginning, to encourage the experts to develop decent relevance matching capabilities and show potentials in certain types of samples, in the first few steps, we update the component experts by considering their loss equally important for every training sample in the standardized learning loss $\smash{L_{st}}$:

At this stage, we sample the negative instances $\smash{D^{-}}$ in Eq. (8) from the top retrieved results of BM25 (Robertson et al., 2009) excluding the relevant documents, denoted as $\smash{\mathcal{N}^{(bm25)}}$.

Specialized Learning Stage. After the first stage, the experts show superiority on certain types of samples than the others. To enhance their expertise, in the second stage, we let them compete with each other on every training sample $(q,\smash{d^{+}},\smash{D^{-}})$. The specialized learning loss $\smash{L_{sp}}$ is the weighted combination of the individual loss from each expert:

where $\text{\scriptsize\Ecommerce{}}\in\{lex,loc,glob\}$, and $\smash{w_{\text{\tiny\Ecommerce{}}}}$ is the weight of the expert ℮ in the overall loss, computed according to its relative performance across all the experts on this sample:

In Eq. (11), $Rank\left(d^{+}\!\mid\!q,d^{+},D^{-};\bm{\theta}_{\text{\tiny\Ecommerce{}}}\right)$ denotes the rank of $d^{+}$ in expert ℮’s predictions, which represents how effectively this expert can perform on the training instance $(q,\smash{d^{+}},\smash{D^{-}})$; $\tau$ is the temperature to control the degree of soft competition, where a smaller $\tau$ means a more intense competition; and the softmax function is applied on all the experts so that $\sum_{\text{\tiny\Ecommerce{}}}w_{\text{\tiny\Ecommerce}}=1$ for each training sample and the experts compete for the update weights.

Similar to (Xiong et al., 2020; Zhan et al., 2021), we train CAME according to Eq. (10) first on BM25 negatives until convergence and then on hard negatives to enhance its performance. We collect hard negatives from CAME learned on BM25 negatives. Specifically, for each query, we first obtain the top retrieval results of every expert ℮ excluding the relevant ones, denoted as $\mathcal{N}^{(\text{\tiny\Ecommerce{}})}$. Then, we sample negatives $D^{-}$ from $\mathcal{N}^{({hard})}=\mathcal{N}^{({lex})}\cup\mathcal{N}^{(loc)}\cup\mathcal{N}^{(glob)}$ and further train CAME with Eq. (10).

During inference time (as shown in Figure 2 (b)), each expert that specializes on a specific relevance pattern can judge from its perspective to determine the relevance scores of documents. Since which expert is more trustworthy for an unseen query-document pair is unknown, we simply consider them equally authoritative. Specifically, for a test query $q$, to obtain its top-$K$ retrieval results of CAME, we first get the top-$K$ results of every component expert ℮ from the whole collection $\smash{\mathcal{C}}$ according to the predicted score $\smash{s(q,d,\bm{\theta}_{\text{\tiny\Ecommerce{}}})}$, then we sum up the document scores from each expert to determine the final top-$K$ results:

The score of a document that is beyond top-$K$ results of an expert is set to the score of the $K$th document this expert has on $q$. This simple method performs similarly or better compared to other fusion methods, e.g., combining expert results based on the summation of reciprocal ranks and learning a linear combination of the expert scores (see Table 4). We leave the study of more complex fusion methods in the future, e.g., learning to predict the probability of each relevance pattern given the query and document, and assign the authority weights to each expert for the fusion.

We conduct experiments on three retrieval benchmarks:

• •

  MS MARCO (Nguyen et al., 2016): MS MARCO passage ranking task is introduced by Microsoft. It focuses on ranking passages from a collection with over 8.8M passages. It has about 503k training queries and about 7k queries in the dev/test set for evaluation. Since the test set is not available, we report the results on the dev set.

• •

  TREC DL (Craswell et al., 2020): TREC 2019 Deep Learning Track has the same training and dev set as MS MARCO, but replaces the test set with a novel set produced by TREC. It contains 43 test queries for the passage ranking task.

• •

  NQ (Kwiatkowski et al., 2019): Natural Questions collects real questions from Google’s search logs. Each query is paired with an answer span and golden passages in Wikipedia pages. There are more than 21 million passages in the corpus. It also has over 58.8k, 8.7k, and 3.6k queries in training, dev, and test sets respectively. In our experiments, we use the version of NQ created by Karpukhin et al. (2020).

We consider two types of baselines for performance comparison, including representative single-model and ensemble retrievers.

We use the official metrics of the three benchmarks. For retrieval tasks (i.e., MS MARCO and TREC DL), the retrieval performance of top 1000 passages is compared. We report the Recall at 1000 (R@1000) and Mean Reciprocal Rank at 10 (MRR@10) for MS MARCO, and R@1000 and Normalized Discounted Cumulative Gain at 10 (NDCG@10) for TREC DL. For open-domain question answering tasks (i.e., NQ), the proportion of top-$n$ retrieved passages that contain the answers (Top-$n$) is compared (Karpukhin et al., 2020), and we report Top-20 and Top-100. Statistically significant differences are measured by two-tailed t-test.

As in UnifieR (Shen et al., 2022), for the Transformer and MLM layers in CAME, we initialize the parameters with the coCondenser (Gao and Callan, 2021) checkpoint released by Gao et al.¹¹1https://github.com/luyug/Condenser. Unless otherwise specified, the number of Transformer layers in the shared layers and MoE layers are set to 10 and 2 respectively. We adopt the popular Transformers library²²2https://github.com/huggingface/transformers for implementations. For MS MARCO and TREC DL, we truncate the input query and passage to a maximum of 32 tokens and 128 tokens respectively. We train CAME with the official BM25 top 1000 negatives for 5 epochs (including 1 epoch for standardized learning and 4 epochs for specialized learning), and then mine hard negatives from the top 200 retrieval results to train 3 more epochs. We use a learning rate of 5e-6 and a batch size of 64. The temperature $\tau$ is tuned on a held-out training subset and is set to 0.5. For NQ, we set the query and passage length to 32 and 156 respectively. We first train CAME with the official BM25 top 100 negatives for 15 epochs (including 3 epochs for standardized learning and 12 epochs for specialized learning), and then mine hard negatives from the top 100 retrieval results for 10-epoch further training. We use a learning rate of 1e-5 and a batch size of 256. The temperature $\tau$ is tuned on the dev set and is set to 2.0. For all these three datasets, each positive example is paired with 7 negatives for training. We control the sparsity of the lexical representations in the lexical matching expert with $\lambda=0.01$ for the FLOPS regularizer (Formal et al., 2021), and the output dimension of local representations in the local matching expert is set to 128. We use the AdamW optimizer with $\beta_{1}=0.9$, $\beta_{2}=0.999$, $\epsilon=10^{-8}$, and set the coefficient to control linear learning rate decay to 0.1. We run all experiments on Nvidia Tesla V100-32GB GPUs.

For all the baseline models, we use the official code or checkpoint to reproduce the results, except for CLEAR, DrBoost, and UnifieR whose code is not available, so we report the evaluation results from their paper directly³³3We report the performance of CLEAR, DrBoost, and UnifieR from their original papers and we could not conduct significance test against them since their code is not released and their result lists are not available..

To answer RQ1, we compare the retrieval performance of CAME with all the baselines described in Section 4.2 and record their evaluation results in Table 1.

Comparisons with Single-model Retrievers. The top block of Table 1 shows the performance of the representative baseline retrieval models. From the results on three datasets, we find that: (1) Neural retrieval models built upon pre-trained language models perform significantly better than term-based retrieval models, i.e., BM25, except for R@1000 on TREC DL. (2) Among all the neural retrievers, RocketQA and AR2 have the best performance on MS MARCO and NQ probably due to their more sophisticated training strategies, such as the data augmentation (in RocketQA) and knowledge distillation (in AR2). (3) For the other methods without complex training strategies, we find that the models that achieve the best performance on each dataset are of different types. Local retriever ColBERT has the best performance on MS MARCO while SPLADE and COIL perform the best on TREC DL. SPLADE is a lexical retriever and COIL is the variant of ColBERT that only keeps the similarities between the exact matched query and document terms. Global retriever DPR has the overall best performance on NQ. This observation also supports our claim that various relevance matching patterns exist. (4) By leveraging multi-types of retrievers, CAME outperforms all the single-model baselines by a large margin in terms of almost all the metrics, showing the superiority of capturing various relevance patterns. Note that CAME should achieve even better performance if using the knowledge distillation and data augmentation techniques proposed in RocketQA and AR2.

Comparisons with Ensemble Retrievers. The performance of the ensemble retrievers are presented in the bottom block of Table 1. We have the following observations: (1) An ensemble retriever that consists of only a single type of model underperforms ensemble retrievers that have multi-types of component models. This can be seen from that DrBoost (only using global retrievers) performs worse than the others. (2) Among the methods that combine various types of retrievers (i.e., ensemble models except DrBoost), DSR and UnifieR perform the best. The superiority of DSR and UnifieR should mostly come from the elaborate joint learning strategies, which shows the importance of the learning mechanism. (3) Compared to the existing ensemble retrievers, CAME achieves significantly better performance on the three datasets in terms of almost all the metrics. For example, its performance improvements over UnifieR are 1.5% and 0.9% on MRR@10 and NDCG@10 for MS MARCO and TREC DL respectively, and the improvements over DSR are 5.1% and 2.8% regarding Top-20 and Top-100 for NQ. These performance gains further confirm CAME’s advantages from employing multi-types of retrievers to capture various relevance patterns and the competitive learning mechanism that facilitates the component expert training.

To answer RQ2, we conduct ablation studies on each stage of the learning pipeline. We only report the results on MS MARCO and NQ in Table 2 due to the space limit, and TREC DL has similar trends. Since R@1000 on MS MARCO is already near 100% and slightly differs between the variants, we also report R@100 to show the differences.

We first remove the training based on hard negatives (i.e., w/o hard negatives) and see that the performance decreases by a large margin, especially on the top results (e.g., MRR@10 on MS MARCO and Top-20 on NQ). This observation is consistent with previous studies, showing that hard negatives could greatly enhance neural retrieval models (Qu et al., 2020). It should be noted that, even without using hard negatives for further learning, CAME still outperforms most baselines (see Table 1), and UnifieR (Shen et al., 2022), the best baseline which is reported to have 98.0 and 38.3 for R@1000 and MRR@10 under the same setting of using BM25 negatives only (Shen et al., 2022). It demonstrates the superiority of competitive learning in CAME.

Based on the above variant of CAME without hard negatives, we run another two ablation studies by: (1) removing the specialized learning stage and training CAME until convergence with Eq. (9) only (i.e., w/o specialized); (2) removing the standardized learning stage and training CAME with Eq. (10) from the beginning (i.e., w/o standardized). The first ablation shows that training without specialized learning leads to a substantial drop in retrieval performance, indicating that it is important to encourage component experts to compete with each other and focus on the samples they are skilled at. The second one shows that training only with the specialized learning stage would also lower the performance. Without the standardized learning process, the component experts are not readily prepared to specialize on certain relevance patterns. Then, the update weights calculated with Eq. (11) cannot reflect accurately how each training sample corresponds to the relevance matching patterns, which further makes it difficult to facilitate the experts to develop their own advantages based on suitable samples.

To answer RQ3, we conduct ablation studies and compare with other alternatives for each component in CAME. Since NQ and TREC DL lead to similar conclusions, we only report the performance on MS MARCO. Note that we only show the model performance based on BM25 negatives training due to the cost of collecting hard negatives.

Ablation Study of Encoder Module. For the ablation study of the encoder module, we: (1) remove the shared layers, that is, each expert has private 12 Transformer encoding layers; (2) remove one expert in the MoE layers separately; (3) remove two experts in the MoE layers simultaneously, that is, the remaining one expert is trained independently. We report the performance of overall and each component expert under every ablation setting in Table 3. By removing the shared layers, we find that MRR@10 of the fusion result decreases significantly, and the performance of every component expert is close to that of their own independent training. This shows that the shared bottom layers in CAME act as multi-task learning that can facilitate the semantic and syntactic knowledge to be better learned for relevance estimation. When removing any expert in CAME, the fusion performance degrades dramatically. Meanwhile, using two experts in MoE layers could achieve better performance than using either of them independently, e.g., the lexical expert and global expert achieve 35.1 and 35.9 on MRR@10 independently (in the bottom block), while using both of them (i.e., the variant of w/o local expert) could achieve 37.8. Furthermore, the rank-biased overlap (RBO) (Webber et al., 2010) between top 1000 results retrieved by any two independently trained experts is 42%-48%. It shows that documents may have multiple relevance patterns and the retrieval results of different experts could complement each other.

Study of Various Fusion Methods. To study the result fusion module, we compare several variants with CAME to verify our choice (see the top block of Table 4): (1) using summation/maximum of min–max normalized scores (NormSum/NormMax) instead of the summation of raw scores (Sum) in Eq. (12); (2) using reciprocal ranks (SumRR/MaxRR) instead of scores for fusion; (3) conducting learnable linear combination of model scores (LinearLayer). We can see that NormSum and LinearLayer do not outperform the simple score summation, i.e., Sum, in CAME. We also find that the fusion of reciprocal ranks has worse MRR@10 than others, probably due to their faster decay than scores. Also, maximum always yields worse results than summation, probably because multiple relevance patterns co-exist in some documents and the judgment from each expert is important to evaluate the overall relevance.

Study of Alternative Expert Learning Methods. To show the advantages of competitive learning, we compare it with alternative options of training an ensemble model with SPLADE, ColBERT, and DPR. We: (1) learn the three models independently on the entire training data that contains both suitable and unsuitable samples for each model (All-Independent) and fuse their results with score summation, reciprocal rank summation, and a learned linear combination layer; (2) learn the three independent models on their suitable samples (Suit-Independent), determined by whether the model has the best MRR@10 for the sample among all experts in CAME. As shown in the bottom block of Table 4, the fusion of reciprocal ranks performs better than scores, which is different from our observations on fusing expert results in CAME. The possible reason is that the scores obtained from the independently learned models are less comparable than those achieved by jointly learned experts in CAME. Both of them are significantly worse than CAME and the fusion variants based on competitive learning. In addition, when fusing the model results with linear combination, models trained only on suitable samples (Suit-Independent) have better ensemble performance than models trained on the whole data (All-Independent), which is consistent with the performance gains produced by the specialized learning mechanism in CAME.

To answer RQ4, we evaluate CAME with different hyper-parameter settings to investigate their impact on retrieval results. Again, we conduct analysis on the models trained on BM25 negatives due to the cost of collecting hard negatives.

Impact of the Temperature. As shown in Figure 3 (a), we examine the effect of temperature $\tau$ in Eq. (11) by varying it from 0.2 to 3.0. The results indicate that $\tau$ has a great impact on retrieval performance, with different optimal values for the two datasets (0.5 for MS MARCO and 2.0 for NQ). MS MARCO has shorter documents than NQ on average, which could lead to that the performance of the local and global matching experts may not differ much on MS MARCO, thus it may require a smaller $\tau$ to make the competition between the experts more intensely to enhance their expertise.

Impact of the Standardized Learning Ratio. Figure 3 (b) shows how MRR@10 changes with respect to the standardized learning ratio, where the ratio means the proportion of the standardized learning steps in the training based on BM25 negatives. For both datasets, the performance reaches the top with 20% standardized learning steps. The performance is relatively stable when the ratio is between 13.3% and 26.7%, while decreasing dramatically at smaller values. We speculate that the component experts cannot develop their expertise if they have not seen enough training samples, and once the standardized learning is enough, the ratio has a small effect on the retrieval effectiveness.

Impact of the Number of Shared Layers. As shown in Figure 3 (c), we investigate the model performance by varying the number of shared layers from 6 to 12. On both MS MARCO and NQ, the performance decreases dramatically when the number of shared layers is set to 12. This indicates that the component experts cannot develop their expertise when their private parameters are too few. In addition, we observe that with fewer shared layers, the performance becomes slightly better on MS MARCO, while 8 shared layers are better than 6 on NQ. The reason may be that fewer private layers are sufficient to learn effective experts for NQ due to its smaller size. Overall, the performance varies slightly from 6 to 10 shared layers (i.e., within 0.2%). Therefore, considering the balance between the effectiveness and efficiency of encoding, we set the number of shared layers as 10 in CAME.

In Figure 4, we show three representative cases in MS MARCO that have different relevance patterns to compare CAME, its component experts, and their counterparts trained independently. We find that, for each case, only one or two models could rank the relevant passage at the top positions. This supports our claim that different types of retrieval models are skilled at identifying different relevance patterns, and a single model could not handle all the cases. By contrast, CAME works well on all the three types of cases by leveraging the power of various types of retrieval experts.

We also compare the results of the component experts in CAME (denoted as “Expert-”) and their counterparts as single-model retrievers, and have some interesting observations: (1) When there are clear disagreements between different types of retrievers, the retriever that ranks the ideal passage to top positions when trained individually performs similarly or even better when it is trained as a component in CAME. The other single-model retrievers that are not good at the cases, by contrast, could have fluctuating performance when trained as experts in CAME. This indicates that the joint training of the experts in CAME with competitive learning mechanism can maintain or enhance their ability to identify their dedicated relevance patterns. (2) In the first two cases, although one component expert in CAME ranks the relevant passage at top-1, the fusion results (i.e., CAME) are not optimal. We impute it to the imperfect result fusion method. In the future, we need to explore better methods to fuse the retrieval results.