Resources for Brewing BEIR: Reproducible Reference Models
and an Official Leaderboard

One recent conceptual innovation in information retrieval is the recognition that the “classic” task of ad hoc retrieval can be framed as a representation learning problem. Karpukhin et al. (2020) demonstrated that transformers can be used to encode queries and documents into a dense vector space, where top-$k$ retrieval translates into the problem of nearest neighbor search. This led to the development of many so-called dense retrieval models (Karpukhin et al., 2020; Xiong et al., 2021; Hofstätter et al., 2021; Khattab and Zaharia, 2020; Qu et al., 2021; Ren et al., 2021). Separately, Zamani et al. (2018) showed that neural networks can be used to learn sparse representations of queries and documents that are amenable to retrieval using standard inverted indexes. Later, researchers applied transformers for learning these sparse lexical representations, which led to a long line of so-called sparse retrieval models (Bai et al., 2020; Jang et al., 2021; Lin and Ma, 2021; Formal et al., 2022; Choi et al., 2022; Formal et al., 2021; Li et al., 2023).

Lin (2021) recently pointed out that learned dense representations, learned sparse representations, and even traditional lexical retrieval models such as BM25 can be viewed as parametric variations of a bi-encoder architecture (see Figure 1). In this design, both queries and documents are fed to “encoders” that generate vector representations. Retrieval boils down to the problem of efficiently finding the top-$k$ most similar document representations given a query representation and a similarity function, usually the inner product.

The design of encoders in such a bi-encoder architecture is dictated primarily by two choices: (1) the basis of the vector space and (2) how the vector weights are assigned. For example, both dense models such as DPR and sparse models such as SPLADE use pretrained transformers to encode queries and documents into vectors; both take advantage of large amounts of manually labeled data. However, the critical difference is the representational basis of their vectors—DPR generates dense vectors, typically with the same width as the output contextual embeddings (for many models, 768 dimensions), whereas SPLADE “projects” the scalar weights of each dimension back into the input vocabulary space, generating, in essence, bag-of-words vectors. BM25 can be understood in this bi-encoder architecture as having a document “encoder” that was heuristically designed (i.e., the BM25 scoring function) and a query “encoder” that generates multi-hot vectors.

Viewing retrieval as representation learning not only helps us understand the relationship between different models, but immediately illuminates open research questions. The dominant approach today is based on supervised learning with (manually) labeled datasets such as the MS MARCO test collections (Bajaj et al., 2018). This naturally begs the question: What happens when models are applied to out-of-distribution data? Examples include applying retrieval models trained on one type of text (e.g., passages from the web) to another type of text (e.g., text from the medical or legal domain), or differences between training and test queries (e.g., keyword queries vs. well-formed natural language questions).

This is where BEIR (Benchmarking IR) (Thakur et al., 2021) comes in. BEIR is a benchmark for zero-shot evaluation of information retrieval models that enables exactly the types of explorations outlined above. For example, BEIR has shown that traditional lexical retrieval models such as BM25 remain competitive baselines—and in fact, the first dense retrieval models evaluated on BEIR were worse overall than BM25 in a zero-shot setting. With BEIR, researchers have discovered that sparse retrieval models appear to achieve better cross-domain generalization than dense retrieval models. The dataset has spurred entirely new lines of research—for example, on unsupervised representation learning (Izacard et al., 2021), where BEIR served as the benchmark for demonstrating model effectiveness.

Nevertheless, we can identify two shortcomings in the state of the current BEIR ecosystem. First, the variety and complexity of modern neural retrieval models create barriers to entry for researchers who wish to explore the research questions that BEIR enables. Successfully executing an end-to-end retrieval run requires coordinating heterogeneous software components that differ depending on the model type. It would be desirable to have reproducible implementations of retrieval models that are easily accessible to everyone, particularly newcomers.

Second, there does not exist an authoritative nexus for sharing and comparing the effectiveness of models on BEIR. When the benchmark was first introduced, the creators shared a publicly readable spreadsheet that tabulated results from different models. Since there was no mechanism for updating this spreadsheet beyond personal communications, it soon fell into disuse.

Today, the de facto method for disseminating and comparing BEIR results is via tables in research papers, with copied-and-pasted effectiveness figures from previous work. This, needless to say, is a laborious and error-prone process. Over time, issues emerged, the most concerning of which is the reporting of results on partially overlapping subsets of the 18 datasets that comprise BEIR. This practice erodes the benefits of BEIR—the entire point of the benchmark is to cover the widest possible range of domains—and makes comparisons of different models difficult. It would be desirable if there existed a single, “official” nexus that gathered all BEIR evaluation results from the community for comparison purposes. In other words, it would be nice to have a self-service central leaderboard.

The BEIR benchmark, introduced by Thakur et al. (2021), is designed to evaluate information retrieval systems across diverse combinations of tasks and domains. It was designed to focus on the “zero-shot” retrieval setting, i.e., evaluation on tasks and domains without any training data or supervision signals. The benchmark enables researchers to explore the out-of-domain generalization capabilities of retrieval models and has steered innovation toward more robust and adaptable retrieval methods.

An overview of the tasks and domains covered by BEIR across 18 distinct datasets is shown in Figure 2. In terms of task diversity, BEIR spans the gamut from traditional ad hoc retrieval tasks (e.g., the TREC 2004 Robust Track) to, for example, Natural Questions (NQ) (Kwiatkowski et al., 2019), an open-domain question answering task involving retrieval of English Wikipedia passages that can answer natural language questions. BEIR tasks further include argument retrieval (e.g., ArguAna, Tóuche-2020) and fact checking (e.g., FEVER, SciFact), both of which are related to, but distinct from, traditional ad hoc retrieval. BEIR also covers various domains, including scientific articles, news, Wikipedia, tweets, and so on. Finally, queries take different forms and can vary in length, from a few keywords (Boteva et al., 2016) to as long as a paragraph (Wachsmuth et al., 2018).

Table 1 summarizes the 18 datasets that comprise BEIR. The datasets range in the amount of relevance judgments available. A few datasets have “dense” judgments, such as TREC-COVID (Roberts et al., 2020), with 66k judgments for 50 test queries, but many have “sparse” judgments, such as SciFact, with only 339 judgments. The corpora associated with the datasets also vary in size, some containing millions of passages, whereas others have only a few thousand. BEIR also standardizes its evaluation metric and uses nDCG@10 and recall@100 across all datasets to compare the effectiveness of each system on an equal footing. Individual dataset scores are macro-averaged across all datasets for a final cumulative score.

The BEIR authors provided additional resources with the benchmark. They shared a GitHub repository¹¹1https://github.com/beir-cellar/beir that contains source code for the evaluation framework along with example usage. The code is written in Python and is available on PyPI (pip install beir). With the original release of BEIR, the authors hosted a leaderboard in the form of a Google spreadsheet.²²2https://docs.google.com/spreadsheets/d/1L8aACyPaXrL8iEelJLGqlMqXKPX2oSP_R10pZoy77Ns/edit There were many issues with this that led to the replacement we describe in Section 6.

This work provides reproducible reference implementations of five different retrieval models for BEIR. These comprise a “bag-of-words” BM25 baseline, two learned dense retrieval models (TAS-B and Contriever), and two learned sparse retrieval models (uniCOIL without expansion and SPLADE). In this section, we provide an overview of these models as they are presented in the literature, but explore different model variants in Section 7.

The effectiveness of the five models presented in the previous section is shown in Table 2, with nDCG@10 in the left group of columns and recall@100 in the right group of columns. Each row corresponds to one of the BEIR datasets, and the rows are ordered in the same manner as Thakur et al. (2021).

We have seen previous papers report BEIR results using slightly different layouts and organizations, which make comparisons difficult. Moving forward, we offer a few best practices to promote consistency in how results are shared: We feel that presenting the datasets in rows and effectiveness metrics in columns feels more natural, and urge the community to also adopt this layout. The alternative of showing the different datasets in columns feels more awkward to us. Furthermore, we recommend that researchers order the rows exactly as Thakur et al. (2021), which we have done here. Other reasonable alternatives, for example, alphabetical sorting, discard the “semantic grouping” of the datasets.

Encouraging consistency in the presentation of BEIR results is an important first step to gaining insight when comparing retrieval models. However, there is no hiding the fact that BEIR scores comprise a complex aggregation of diverse datasets, and the standard approach of comparing macro-averaged nDCG@10 scores (as we have done in the final row of Table 2) is deficient in many ways.

It is well known that averages often hide important individual differences, but teasing apart these differences from a large table of numbers such as Table 2 can be difficult. For example, the results show that uniCOIL and BM25 achieve a similar level of effectiveness overall (0.428 vs. 0.429), but what can we say about effectiveness on individual datasets? Glancing down the rows, we see many differences—some large, some small—so it is possible to conclude that although uniCOIL and BM25 are “about the same” averaged across the BEIR datasets, effectiveness on individual datasets differ. How can we gain more insight easily? The same question applies when comparing BM25 and TAS-B, where the average nDCG@10 scores are comparable. Consider the SPLADE and Contriever results: we see that both achieve a higher average across all the datasets, but is this due to consistent gains across many datasets or a few big gains? It’s difficult to tell from Table 2.

We present a potential solution to these challenges in terms of radar charts: Figure 3 shows a visualization comparing the effectiveness of the five retrieval models, with nDCG@10 on the left and recall@100 on the right. Each radar chart comprises 18 axes, arranged radially, in the same order as the rows in Table 2. The effectiveness of a model is plotted on each of the 18 axes and connected by line segments to form a polygon. The effectiveness of BM25, which serves as a baseline, is scaled to half of the radius of the entire chart area, so the effectiveness of BM25 across all 18 datasets is captured by the dotted polygon. The effectiveness of the other models on each corpus is scaled relative to BM25. That is, points further away from the center represent higher scores and points closer to the center represent lower scores, where the distance to the midpoint of the axis is proportional to the score difference with respect to BM25.

The radar charts allow us to easily compare the effectiveness of the models across all datasets, and differences that are obscured by averages come readily to light. Focusing on nDCG@10, consider the question above about BM25 vs. uniCOIL (orange): We can see that uniCOIL excels on HotpotQA and NQ in terms of nDCG@10, but otherwise achieves effectiveness that is either on par with BM25 or worse. In particular, on Signal-1M, Quora, and Tóuche-2020, uniCOIL is substantially worse. We see a similar situation with TAS-B, which is more effective on some datasets but performs terribly on others, most notably TREC-COVID, BioASQ, and Tóuche-2020. Inconsistent effectiveness is similarly observed with Contriever as well, even though on average the model scores higher than BM25. It appears that both dense retrieval models perform rather poorly on BioASQ and TREC-COVID, two datasets that focus on biomedical retrieval. For all the models examined, it appears that SPLADE exhibits the most consistent gains, with only a few datasets where nDCG@10 is worse than BM25.

Focusing on recall@100 (Figure 3, right), we believe that the radar chart is similarly helpful in highlighting effectiveness differences between the models that are obscured by only looking at the means. In particular, conclusions drawn from the nDCG@10 scores differ from those based on recall@100: both metrics are important, but for different reasons. Early precision metrics such as nDCG@10 directly quantify output quality if results from a first-stage retriever are directly presented to the user. On the other hand, recall@100 quantifies the upper bound of reranking effectiveness. Based on the radar chart, we clearly see that neural models do not consistently increase effectiveness. We observe large gains—for example, all four models perform very well on NQ and DBPedia—but we also observe many cases where at least some of the neural models perform poorly—for example, on Signal-1M and BioASQ.

We have some explanations for these findings. First, let us consider the poor effectiveness of dense retrieval models on TREC-COVID and BioASQ. BioASQ involves scientific paper retrieval given a biomedical query that often involves specialized terminology (and similarly for TREC-COVID as well). As an example, consider the BioASQ query, “Is AZD5153 active in prostate cancer?” Here, “AZD5153” is a specialized biomedical term, which the model most likely has seen only rarely during training. This would cause issues for dense retrievers, as they are unable to represent rare terms well within the embedding space, and hence retrieval effectiveness would suffer. That is, poor effectiveness can be explained by domain shifts between training data (general web) and the test data (biomedical texts). On the other hand, sparse models such as SPLADE or uniCOIL produce representations that retain lexical matching, and thus are less impacted by domain shifts involving vocabulary differences. The relatively poor effectiveness of models on Signal-1M (tweets) can be explained similarly.

Second, we observe that all transformer models achieve big gains on NQ. Previous work has found that effectiveness on NQ and effectiveness on the TREC 2019 Deep Learning (DL) track have the highest correlation among all BEIR datasets in terms of scores (Zhan et al., 2022) (that is, models that perform well on TREC DL also perform well on NQ). The close connections between the MS MARCO datasets and the TREC Deep Learning tracks suggest that transfer from MS MARCO is particularly advantageous for the NQ dataset. Since all models examined in this paper take advantage of MS MARCO, the big gains on NQ might be a data artifact and not a demonstration of robust domain transfer capabilities.

Finally, it appears that all transformer models perform worse than the BM25 baseline for Tóuche-2020, sometimes by sizeable margins. Deeper analysis suggests a number of factors at play. The dataset has the fewest number of queries (49) and the relevance judgments were created from very shallow (top 5) pools. As a point of contrast, TREC-COVID, the dataset with the next smallest number of queries (50), has much deeper pools. Tóuche-2020 queries are rather short, for example, “Is obesity a disease?” and phrased in a manner where relevant documents are highly likely to contain query terms. The task of argument retrieval is quite different from standard ad hoc retrieval tasks, thus potentially hurting zero-shot transfer effectiveness. Reading through the organizers’ description of the evaluation, it appears that most submissions focused on lexical techniques, and we believe that the relevance judgments are biased towards models based on lexical matches. Thus, the qrels might be unable to fairly assess models that retrieve semantically relevant content but do not necessarily contain query terms. We believe this is the case for many of the datasets in BEIR, although this phenomenon is most pronounced in Tóuche-2020.

Our point here is not to exhaustively explain all the effectiveness differences observed, although we do offer some analyses above. Instead, the main contribution of this paper is a methodological tool (i.e., visualizations using radar charts) that provides a starting point for further analyses. For researchers, it would be worthwhile to follow up any of the findings above with more detailed error analyses, but such a detailed study is beyond the scope of this work. Beyond support for error analyses, we believe that these radar chart visualizations are helpful for practitioners who might be interested in deploying neural models. From the perspective of real-world applications, it would make sense to ensure that a deployed model “performs no worse” than BM25, and thus these results lead us to conclude that none of the models are viable as a replacement yet, given that there are clearly situations where effectiveness is substantially lower.

We provide reproducible implementations of the retrieval models discussed in this paper. At a high level, our goal is to make it as easy as possible for researchers to reproduce the results in Table 2. To be precise, here we are using reproducibility in the sense articulated by the ACM in its Artifact Review and Badging Policy,⁴⁴4https://www.acm.org/publications/policies/artifact-review-and-badging-current characterized as “different team, same experimental setup”. Specifically, “this means that an independent group can obtain the same result using the author’s own artifacts.”

Our reproducible implementations conform to the aspirational ideal of “two-click reproductions” described by Lin (2022). The motivation is that a user should be able to reproduce an experimental result with only two clicks: a “copy” and a “paste” from a documentation page. That is, the user will arrive at the same nDCG@10 and recall@100 reported in Table 2.

Our reproducible implementations leverage previous efforts and infrastructure investments in the Pyserini IR toolkit (Lin et al., 2021a), which is built on Anserini (Yang et al., 2018). Anserini is an IR toolkit built on the open-source Lucene search library, and like Lucene, it was written in Java. To provide compatibility with Python, the dominant language for building neural retrieval models today, we developed Pyserini, which provides Python bindings for Anserini as well as many other non-Java capabilities.

In our design, BM25 baselines and sparse retrieval models are directly implemented in Anserini with Lucene inverted indexes, exposed in Pyserini in Python. The dense retrieval models are implemented using the Faiss library for efficient similarity search and clustering of dense vectors (Johnson et al., 2021) by Meta Research; for simplicity, we used flat indexes. Pyserini provides a uniform API to support retrieval using all the models, for example, abstracting over the Java-based implementation of retrieval using BM25 and the sparse retrieval models.

To provide a concrete example, performing a BM25 retrieval run over the test queries in the BioASQ corpus in BEIR can be accomplished by the following command:

    python -m pyserini.search.lucene \
      --index beir-v1.0.0-bioasq-multifield \
      --topics beir-v1.0.0-bioasq-test \
      --output run.beir-multifield.bioasq.txt \
      --output-format trec \
      --batch 36 --threads 12 \
      --hits 1000 --bm25 --fields contents=1.0 title=1.0

The --index argument specifies a pre-built inverted index for the BioASQ corpus that is stored on our group’s servers. On the first invocation of the above command, the driver automatically downloads the index and caches it on the local machine. The --topics argument specifies the BioASQ test queries, which are already included as part of Pyserini. With this design, the user does not need to separately figure out where to download the indexes and queries to successfully reproduce a result.

Pyserini also includes all the components necessary to evaluate the retrieval results. In this case, the nDCG@10 score can be computed as follows:

    python -m pyserini.eval.trec_eval \
      -c -m ndcg_cut.10 beir-v1.0.0-bioasq-test \
      run.beir-multifield.bioasq.txt

Modern IR evaluation methodology can be quite complex, but with our “two click reproductions”, the two commands above will produce the results in Table 2. We have built a landing page⁵⁵5https://castorini.github.io/pyserini/2cr/beir.html in the Pyserini documentation that provides an entry point to a “reproduction matrix” comprising all models and all datasets.

Another major contribution of this work is the introduction of an official self-service BEIR leaderboard that unifies community-wide evaluation efforts. This replaces an informal leaderboard that was initially maintained via a public Google spreadsheet with read-only permissions. Updating the leaderboard involved researchers directly contacting the organizers to share the effectiveness of each task, who would manually enter the new results into the spreadsheet. In the beginning, this was manageable because there were relatively few researchers, but after BEIR started to gain traction in the community, the overhead of communications and manual effort made leaderboard maintenance unsustainable. As a result, the spreadsheet leaderboard became stale as time went by. And as the pace of leaderboard updates slowed, researchers sought other means to disseminate results. This in turn led to disparate forms of reporting results. Many researchers opted to report their results on arbitrary subsets of BEIR, giving rise to scores on several different subsets containing anywhere from 11 to 15 datasets.

As a remedy to these issues, we have built an official leaderboard to standardize evaluation on BEIR, which is linked from the website at http://beir.ai/. The new leaderboard is hosted on EvalAI (Yadav et al., 2019), which is an open-source platform designed to enable human-in-the-loop and interactive agent evaluations. It has successfully hosted over 200 challenges to date from more than 30 organizers. The platform provides the capability to configure customized evaluation pipelines with flexible dataset splits and challenge phases that can be executed on cloud-based worker pools. Specifically, EvalAI defines a new challenge through a competition bundle, encapsulated in a GitHub repository, that contains the challenge configuration (e.g., start/finish date, submission limits, etc.), evaluation code, and information about data splits. Moreover, it supports the processing of large submission files, making it particularly suitable for the BEIR benchmark, which often involves large run files.

The new leaderboard offers an easy-to-use interface accompanied by documentation, providing a step-by-step description of the submission protocol. To evaluate a model on BEIR, users need to sign up for EvalAI first. Then, they can submit their run files through the EvalAI command-line library or through a web interface. Each run file may contain at least top-10 and up to top-100 retrieved results. Upon receiving a submission, an evaluation script validates each run file, promptly addressing any edge cases that may arise. For example, queries that are returned in retrieved documents will be discarded. The script subsequently measures the effectiveness of each task. The results can be readily viewed in the public leaderboard, although users have the option to keep their results private. We limit the submission rate to one per day.

Moving forward, this leaderboard will serve as the central “source of truth” for BEIR results, enforcing consistency in evaluation protocol. Furthermore, we have invited the community to submit results from their already published methods as an attempt to consolidate results that pre-date the leaderboard.

The models presented in Section 3 cover the major approaches to neural retrieval today. In this section, we further examine variants that help us better understand some of the strengths and limitations of those models.

The BEIR benchmark provides an important instrument for evaluating the cross-domain robustness of retrieval models and has gained traction due to the growing recognition of retrieval as a form of representation learning. The efforts described in this paper address the two shortcomings that we have identified with BEIR: challenges in reproducibility and in the sharing of results. Reproducible reference implementations in the Pyserini IR toolkit tackle the first challenge. An official self-service leaderboard and best practices for sharing results target the second challenge.

Looking ahead, we see two more challenges that need to be addressed: First, how do we perform significance testing for a benchmark like BEIR that is itself an aggregation of individual datasets? There is no agreed-upon methodology, and although one could perform tests on each dataset, significance testing across the final macro-average would not be meaningful since the datasets diverge so much in corpus size, number of queries, number of judgments, and many other dimensions. Thus, there is currently no way to answer the question: Is this retrieval model significantly better than that across a broad range of domains? While our radar charts can show if observed gains are consistent across the BEIR dataset, they currently lack statistical rigor.

The second major issue concerns potential systematic biases in the current relevance judgments used in the evaluations. Many of the datasets were created before the advent of modern transformer-based models, and may have relevance judgments biased towards lexical matching techniques. Voorhees et al. (2022) argued that for old TREC collections that used deep pools, this doesn’t appear to be an issue, but many of the datasets in BEIR used rather shallow pools (e.g., Tóuche-2020), where potential biases are still poorly understood. Operationally, these issues manifest as unjudged documents, sometimes called “holes”. We are not aware of any work that has examined these issues in detail, but we believe that such studies are sorely needed.

To end on a positive note, we are optimistic about the future of the BEIR benchmark. It is an important evaluation instrument for the community to address a number of timely research questions. This work mitigates two existing shortcomings, and while there remain more challenges ahead, BEIR has already and will continue to help advance the state of the art.

This research was supported in part by the Canada First Research Excellence Fund and the Natural Sciences and Engineering Research Council (NSERC) of Canada.