Simple Yet Effective Neural Ranking and Reranking Baselines for Cross-Lingual Information Retrieval

A very simple approach to cross-lingual information retrieval is known as document translation: Given $q_{e}$ in language $e$ and the corpus $\mathcal{C}_{f}$ in language $f$, we can translate the entire corpus into language $e$, i.e., $\{\textsc{Translate}(d_{i})\}$, and then perform mono-lingual retrieval in language $e$. This design is shown in Figure 1(b), where the primary addition is a document translation phase that feeds into the document side of the bi-encoder architecture.

While translating the entire corpus can be time-consuming, it only needs to be performed once and can be viewed as an expensive pre-processing step, like other computationally demanding document expansion techniques such as doc2query (Nogueira et al., 2019b). Any translation technique can be used, including off-the-shelf MT systems. Generally, since documents are comprised of well-formed sentences, automatic translation output can be quite fluent, depending on the quality of the underlying system. This stands in contrast to query translation (see below), where quality often suffers because queries are usually much shorter (hence lacking context) and systems are not usually trained on such inputs.

Once $\mathcal{C}_{f}$ has been translated into $\mathcal{C}_{e}$, we now have a mono-lingual retrieval task since queries are also in $e$. In our case, the three corpora are in Persian, Russian, and Chinese, and we used the English translations provided by the NeuCLIR Track organizers, generated by the SockEye MT system. From the NeuCLIR topics, we extracted three types of English queries: only the “title” field, only the “description” field, and both. Our experiments used two retrieval models and pseudo-relevance feedback:

The flip side of document translation is known as query translation: Given $q_{e}$ in language $e$ and the corpus $\mathcal{C}_{f}$ in language $f$, we can translate the query into language $f$, i.e., $\textsc{Translate}(q_{e})=q_{f}$, and then perform mono-lingual retrieval in language $f$. This design is shown in Figure 1(c), where we add a query translation component that feeds the query side of the bi-encoder architecture.

Query translation is much more computationally efficient than document translation, but has the disadvantages already discussed—queries may be more difficult to translate given that they may not be well-formed sentences. However, this approach enables more rapid experimentation since the introduction of a new translation model does not require re-translation of the entire corpus.

One challenge of query translation is that we need a good mono-lingual retrieval model in $f$, which by definition is non-English. While BM25 can provide a baseline (in the bag-of-words space of language $f$), effective learned retrieval models are more difficult to come by since less manually labeled data are available in non-English languages.

Our experiments consider both human and machine translations of the topics provided by the track organizers. From each type of translation, we can create three types of queries: “title”, “description”, and “both” (similar to the document translation case above). Thus, we have a total of six variations: {human translation, machine translation} $\times$ {title, description, both}. With these conditions, we experimented with two different retrieval models as well as pseudo-relevance feedback:

Starting from the bi-encoder design for mono-lingual retrieval shown in Figure 1(a), one might wonder if it were possible for the document and query encoders to generate some sort of language-independent semantic representation that would support direct relevance matching across languages. With the advent of pretrained multilingual transformers, this is indeed possible. For example, we can apply the document encoder to documents in $\mathcal{C}_{f}$ (in language $f$), and apply the query encoder to a query in $e$, and directly conduct relevance ranking on the representations. Thus, we can perform cross-lingual retrieval without explicit query or document translation. This is shown in Figure 1(d).

The most straightforward implementation of this approach is to train a DPR model (Karpukhin et al., 2020), but starting from a multilingual transformer backbone such as mBERT. To our knowledge, Asai et al. (2021) was the first to propose such an approach. More recently, Zhang et al. (2022) built on this basic design and introduced different approaches to exploit cross-lingual transfer by “pre–fine-tuning” on English data before further fine-tuning on the target languages using non-English data. Although Zhang et al. focused on mono-lingual retrieval in non-English languages, many of the lessons learned are applicable to the cross-lingual case as well.

Specifically, for this work, we pre–fine-tuned a multilingual DPR model initialized from an XLM-R (Conneau et al., 2020) backbone,²²2https://huggingface.co/xlm-roberta-large dubbed xDPR. The model was trained on the MS MARCO passage dataset (Bajaj et al., 2018), where both query and passage encoders share parameters.

With this trained model, we separately encoded the corpora in Persian, Russian, and Chinese. It is perhaps worth emphasizing that the same model was used in all three cases. For query encoding, we have a number of design choices. Similar to document translation and query translation, we can use “title”, “description”, or “both”. Furthermore, we can encode queries either in $e$ or $f$. In the first case, we are asking the encoder to directly project $e$ queries into the semantic space occupied by the $f$ documents. In the second case, the query starts off in $f$, so the model is encoding a sequence in $f$ into the semantic space occupied by $f$ documents. Thus, for each language, we arrive at a total of nine variations: {original query, human translation, machine translation} $\times$ {title, description, both}.

Finally, on top of xDPR retrieved results, we can apply pseudo-relevance feedback using Rocchio’s method, following the work of Li et al. (2021); Li et al. (2022). Thus, combined with Liu (2022), we are able to implement Rocchio feedback consistently across both dense and sparse retrieval models.

In a standard multi-stage ranking architecture, the first-stage retriever generates a ranked list of candidates that are then processed by one or more reranking stages that aim to improve the ranking. Reranking is also applicable in the cross-lingual case, but depending on the first-stage retriever, the candidate query/document pairs may either be in $e$ or $f$. In cases where both the queries and documents are in $e$, we can use a mono-lingual English reranker.

For the first-stage retrievers based on document translation, our experiments used monoT5, which is based on T5 (Raffel et al., 2020). Reranking is performed in English with the following prompt:

The model is asked to generate either the “true” or “false” token, from which we can extract the probability of relevance used to sort the candidates. When the monoT5 model is fine-tuned on the MS MARCO passage dataset, it achieves state-of-the-art results on the TREC Deep Learning Tracks (Craswell et al., 2019; Craswell et al., 2020), as well as impressive zero-shot effectiveness on BEIR (Jeronymo et al., 2023) and many other datasets (Roberts et al., 2019; Zhang et al., 2020; Rosa et al., 2021, 2022).

For reranking first-stage retrievers based on query translation, we used a variant based on the multilingual version of T5 called mT5, which was pretrained on the multilingual mC4 dataset (Xue et al., 2021); otherwise, we use the same reranking approach. To fine-tune mT5 for reranking, we employed a similar strategy as Bonifacio et al. (2021) using mMARCO, the multilingual version of the MS MARCO dataset. For our experiments, we used the XXL model with 13B parameters.

Researchers have known for many decades that fusion techniques, which combine evidence from multiple individual runs, can improve effectiveness (Bartell et al., 1994; Vogt and Cottrell, 1999). Fusion works particularly well when the individual runs are based on different underlying techniques, such as in the case of dense vs. sparse retrieval models (Gao et al., 2021; Ma et al., 2022). Given that our first-stage retrievers are all based on very different approaches, we would expect fusion to yield substantial boosts in effectiveness, although this does not appear to be borne out experimentally.

Within a multi-stage architecture, there arises a natural question of when fusion should be performed. One possible approach is to independently rerank the output of each first-stage retriever, and then fuse those results; we call this late fusion. Another possible approach is to first fuse the output of the first-stage retrievers, and then rerank the combined results; we call this early fusion. The effectiveness difference between the two approaches is an empirical question, but late fusion is more computationally intensive because it requires more reranking.

We begin by examining the output of individual first-stage retrievers. Tables 1 and 2 present results in terms of nDCG@20 and recall@1000, respectively. Each block of rows is organized by the general approach. The columns show metrics grouped by language, and within each block, we report the results of using queries comprised of the “title” field, the “description” field, and both.

In the previous section, we examined first-stage retrieval settings for 18 $\times$ 3 $=$ 54 different conditions, for each language. It is impractical to report reranking results for every single condition, and thus we made a few choices to focus our attention: We considered only conditions that take advantage of both title and description fields, which appear to be more robust than title-only queries. We also focused on runs without PRF, since PRF represents additional computational costs (both latency and index size).

For each language, this reduces the number of first-stage retrievers under consideration to nine. We applied reranking on these runs, including the title and description fields in the input template to the reranking models. We informally, but not exhaustively, examined other conditions, but they did not appear to alter our overall findings. For example, we tried reranking the first-stage retrieval results with pseudo-relevance feedback, but the results were not noticeably better (even though they exhibited higher recall).

Reranking results are shown in Table 3. Under the effectiveness of the first-stage retriever (“1st” columns), we report (nDCG@20, recall@1000): the first quantifies candidate ranking quality and the second quantifies the upper bound effectiveness of a reranker. We see that reranking improves effectiveness by large margins, but this is expected as the effectiveness of cross-encoders in various settings is well known (see Section 3.4).

One interesting observation, however, is that reranking reduces the effectiveness gap between the best and worst first-stage retrievers. For example, starting with BM25, which is clearly less effective than SPLADE, the reranker is able to “make up” for the lower quality candidates, such that the end-to-end effectiveness is relatively close to reranking SPLADE results (at least in terms of nDCG). In fact, in some cases, reranking xDPR results yields scores that are even higher than reranking BM25 results. While “coupling effects” between the first-stage retriever and reranker have been previously noted in the literature (Gao et al., 2021; Pradeep et al., 2022), this finding affirms the need for further explorations.

With fusion, the design space of possible combinations is immense and impractical to exhaustively explore. To provide continuity, we focus only on the first-stage retrievers in the reranking experiments. In the space of fusion techniques, we settled on reciprocal rank fusion, which is a simple, effective, and robust approach (Cormack et al., 2009).

With these considerations, we experimented with the following fusion conditions in Table 4: (6a) document translation combining BM25 and SPLADE; (6b) query translation combining BM25 and SPLADE; (6c) combining document and query translation with BM25; (6d) combining SPLADE document and query translation; (6e) combining all lexical approaches; (6f) combining both dense approaches; (6g) combining everything. The top block of Table 4 repeats the effectiveness of the first-stage retrievers for convenience. In the bottom block of the table, cases in which the fusion results are worse than the best input are shown in red. In these cases, fusion provides no value over just selecting the best individual run.

From these results, it appears that for Persian and Russian, the best effectiveness can be achieved by fusing both document translation and query translation SPLADE models, row (6d), although for Chinese, the same fusion is a bit worse than just document translation SPLADE. Fusing all the lexical runs, row (6e), is a bit worse than fusing just SPLADE runs in Persian and Russian, but it improves Chinese. Finally, incorporating evidence from the language-independent dense retrieval techniques appears to provide value over simply fusing the lexical results, as we see comparing (6g) and (6e). This is surprising given that by themselves, the dense retrieval runs are quite poor.

Overall, we were somewhat surprised by the finding that fusion did not improve effectiveness as robustly as we had hoped. In Table 4, the figures in red represent all the cases in which fusion actually hurt effectiveness, i.e., fusion performed worse than the best single input run. We attribute this finding to the large differences in effectiveness between the runs, in that RRF does not work as well if one of the fusion inputs is much better than the others.

To more rigorously test this observation, we performed significance testing comparing the document translation SPLADE model, row (2a) in Table 4, against fusion of SPLADE models, row (6d), fusion of all lexical models, row (6e), and fusion of all lexical and dense models, row (6g). These comparisons answer the following questions, starting from the single best first-stage retriever: Does SPLADE fusion provide any additional value? What about BM25? Dense retrieval?

The conclusion, reported in Table 4 with the symbol ^†, is that most of the fusion combinations are not statistically significantly better than document translation with SPLADE, the single best first-stage retriever. For nDCG@20, the largest ensemble is significantly better than DT–SPLADE only on Russian; for recall@1000 we see more significant improvements, but only on Persian and Chinese. Notably, combining evidence from both document and query translation with SPLADE, row (6d), is not significantly better than DT–SPLADE alone.

In our final set of experiments, we compared the effectiveness between early and late fusion for a subset of the conditions in Table 4. These results are reported in Table 5. In this case, we use QT–SPLADE as the point of comparison, which appears to provide the best single-stage retriever and reranking combination. For Persian, late fusion appears to be either about the same or slightly better, with the exception of (6f); this appears to be the case for Russian also, although the late fusion margin of improvement seems to be smaller. Chinese results are a bit more mixed, with early beating late in some cases. To more rigorously compare early vs. late fusion, we performed significance tests comparing all pairs. Only a few of these differences are significant, and they only happen for cases where early fusion is better than late fusion. Two of the three cases, however, occurred for the dense models, which are less effective to begin with. Overall, these experiments are inconclusive with respect to the question of which fusion strategy is better.

To provide additional context, the best runs from the NeuCLIR 2022 evaluation were from members of our group, but were generated under the time pressure of deadlines and thus it was not possible to carefully consider all configurations as we did in Table 5. The best runs were (nDCG@20 scores): (i) Persian: p2.fa.rerank, 0.588; (ii) Russian: p3.ru.mono, 0.567; (iii) Chinese: p2.zh.rerank, 0.516. Comparing those runs to the best conditions reported here, we verify that just by carefully studying the various effects of different system components, improvements are possible across all languages, achieving new state-of-the-art effectiveness with (i) Persian: 6d late-fusion 0.612 ($+$0.024); (ii) Russian: 6d late-fusion 0.592 ($+$0.025); (iii) Chinese: 6e early-fusion 0.539 ($+$0.023).