Open-domain Visual Entity Recognition:
Towards Recognizing Millions of Wikipedia Entities

Task Definition The input to an Oven model is an image-text pair $x=(x^{p},x^{t})$, with the text query $x^{t}$ expressing intent with respect to the corresponding image $x^{p}$. Given a unified label space $\mathcal{E}$ which defines the set of all possible entities, the knowledge base $\mathcal{K}=\{(e,p(e),t(e))\mid e\in\mathcal{E}\}$ is a set of triples, each containing an entity $e$, its corresponding text description $t(e)$ (i.e., name of the entity, description, etc.) and a (possibly empty) set of relevant images $p(e)$. For instance, an entity $e=\text{{{\small Q7395937}}}$ would have a corresponding textual description $t(e)=\text{`{\small Name: Sabatia campestris; Description:$\ldots$}'}$²²2In this paper, we only consider using the name of the entity as its textual representation, despite the fact that more textual descriptions are available. and a set $p(e)$ containing one or more images from the corresponding Wikipedia page³³3https://en.wikipedia.org/wiki/File:Sabatia_campestris_Arkansas.jpg of Sabatia campestris. We consider the combination of $t(e)$ and $p(e)$ the multimodal knowledge for the entity $e$. As Oven is a recognition task, we focus on recognizing and linking entities that are physically present in the image.⁴⁴4Extending this framework to entities that are not physically present in the image (e.g. the inventor of the airplane) is also valid and useful. See a follow-up works [10] for more details.

Data Split and Evaluation Due to Oven’s goal of evaluating pre-trained multi-modal models, we only provide a partial set of visual concepts (i.e., seen categories) for model training or fine-tuning. For evaluation, an Oven model is tested on generalization to entities not present in the fine-tuning data (thus unseen), without forgetting the seen concepts. The models need to either acquire information from the knowledge base, or make a prediction using knowledge obtained during pretraining. We evaluate Oven with a metric aiming to balance performance between seen and unseen entities using a harmonic mean, as shown below:

$$\small\textsc{hm}(\texttt{Acc}_{\textsc{seen}},\texttt{Acc}_{\textsc{unseen}})=2\;/\;(\frac{1}{\texttt{Acc}_{\textsc{seen}}}+\frac{1}{\texttt{Acc}_{\textsc{unseen}}})$$

(1)

Harmonic mean equally weighs the importance of the seen and unseen subsets, and penalizes models with a short barrel. Further details are provided in §3.

Oven versus recognition benchmarks Given that an Oven model need to generalize to unseen entities, it is required to predict over all KB entities, which can exceed 6 million in our experiments (e.g., the size of English Wikipedia). This is orders of magnitude larger than existing benchmarks. Second, the large label space has made the generalization to unseen entities the most critical criterion for a successful Oven model, which also allows future open-domain evaluation⁵⁵5One can collect and label a new set of entities from Wikipedia, to serve as a new evaluation data for Oven models. Third, Oven requires models to do multi-modal reasoning, i.e., comprehending the text query within its visual context, to predict the answer entity.

Oven versus Visual QA tasks Oven can be considered as a VQA task because its input format is the same as that of standard VQA models (e.g., text query + image). However, Oven is specialized and focuses solely on recognition, with the text input serving mainly for intent disambiguation. Moreover, Oven models are required to generate the name of an entity that exists in a given KB (like models for text entity linking tasks), while VQA models output free-form answers (such as yes/no for verification questions and numbers for counting questions).

From Oven to Knowledge-Intensive VQA Although this paper aims to evaluate pre-trained multi-modal models on universal visual entity recognition, we highlight that models that excel at Oven can serve as foundational components for systems that can answer knowledge-intensive questions. For example, given an image and a question “When was the church built?”, one could apply an Oven model to link the image to a concrete church’s Wikipedia page and then extract the answer from that document. A follow-up work has conducted a thorough study on the value of Wikipedia grounding for answering knowledge-intensive visual questions [10].

Label Disambiguation and Human Annotation Grounding the labels of 14 datasets to Wikipedia entities is challenging, and we perform the following steps to accomplish this. We first apply a state-of-the-art textual entity linking system [13] to recognize text labels and map them into Wikipedia. Human annotators are used to write rules to detect bad linking results or unlinkable labels (e.g. numbers), and correct entity linking errors. The union of original dataset labels were linked to 20,549 unique Wikipedia entities, each with a number of examples for the purpose of training and evaluation. Meanwhile, we construct the candidate label space using the English Wikipedia snapshot from Oct. 1 2022, by removing all disambiguation, redirect, and media file pages. As shown in Figure 1 (right), this left us with 6,063,945 Wikipedia entities in total. Note that we only consider using the first Infobox images [57] from each page to serve as the visual support for each Wikipedia entity; these are available for 2,032,340 entities.

Dataset Statistics Figure 3 (left) presents the general distribution of the super-categories for our final collection of Wikipedia entities that have positive examples. Figure 3 (right) shows detailed statistics for queries and entities for each of the fine-tuning (train), validation, test, and human splits. Note that the models do not know which entities are present in the val/test/human set, and must scan through the whole KB to make predictions. The # of seen/unseen examples indicates the # of examples of which the positive entity labels are in the seen/unseen split.

Evaluation Details As aforementioned, we evaluate models by asking them to predict one out of over 6 million English Wikipedia entries. While our data does not cover all 6 million labels as positive examples, models still need to consider all possible outputs due to the presence of unseen entities. We measure the models’ performance using both the Entity Split (ES) and Query Split (QS). Specifically, we first compute the harmonic mean of accuracy over examples from the seen and unseen classes, as $\texttt{Acc}_{\texttt{ES}}=\textsc{hm}(\texttt{Acc}_{\texttt{ES}\textsc{seen}},\texttt{Acc}_{\texttt{ES}\textsc{unseen}})$ and $\texttt{Acc}_{\texttt{QS}}=\textsc{hm}(\texttt{Acc}_{\texttt{QS}\textsc{seen}},\texttt{Acc}_{\texttt{QS}\textsc{unseen}})$ as the Equation 1. Then we further calculate the harmonic mean between splits $\textsc{hm}(\texttt{Acc}_{\texttt{ES}},\texttt{Acc}_{\texttt{QS}})$ to reward models that do well on both splits. We use the validation data, which contains examples from subsets of both seen and unseen entities, for model selection, and we measure performance on the test split and the human evaluation set.

Does fine-tuning always help generalization? Figure 4 presents the validation scores of the PaLI model (left) and the CLIP2CLIP model (right), during fine-tuning on Oven-Wiki’s training split. It shows that a longer training schedule does not lead to better generalization performance, particularly when evaluated on the unseen entities. Because of this, we employ the early stopping strategy for model selection, and pick the model with the best harmonic mean combined score on the validation set. However, due to this early stopping strategy, both fine-tuned models are not utilizing 100% of the examples in Oven’s training data because their unseen performance starts to degenerate within one epoch. This has indicated that more advanced fine-tuning strategies that use better regularization techniques to encourage generalization across Wikipedia entities, could be a promising research to explore in the future.

How would the number of entities in KB influence the model’s prediction? Figure 5 presents the accuracy of CLIP2CLIP, as a function of the # of total candidates to retrieve from. Here, we compute the accuracy by sub-sampling the negative candidates from KB to different sizes. We observe that when the retrieval candidate entities are only the positive entities (with the # of candidates being 20K), the performance of the CLIP2CLIP model is significantly higher than the open-domain setting (with 6M entities in total). Beyond this, as the KB size increases, model accuracy decreases. Concretely, it shows an approximately linear decline along the log-scale x-axis in Figure 5. This indicates that as the KB size increases, the models’ accuracy first drops significantly and then follows with a gradual decline. On the other hand, PaLI’s performance is generally more steady as the size of KB grows, potentially because its prediction has already matched up entity names inside KB, so narrowing down the set of candidates does not help the BM25 post-processing. One potential direction is to employ constrained decoding for the PaLI-based model, which we leave for future works.

How would models perform on head vs. tail entities? We evaluate the visual entity recognition performances of CLIP2CLIP and PaLI, on entities of different popularity. Particularly, Figure 6 presents a histogram according to models’ performance on the entity that has different average monthly Wikipedia page views in 2022 [31]. From the comparison, we can see that PaLI is significantly more accurate compared to CLIP2CLIP, on the head entities (that have more than 5K monthly page views). However, we observe that CLIP2CLIP can perform on par or even outperform PaLI on tail-ish entities (that have less than 2.5K monthly views). This suggests that the retrieval-based visual entity recognition model has its own advantages, in recognizing the difficult and tail entities. Meanwhile, this result also provides a hint that potentially a frequency calibrated evaluation should be developed to reward models more with strong recognition capability on the tail entities.

Error analysis To better understand the errors that CLIP2CLIP and PaLI models are making, we sampled a random 100 examples on the human evaluation set, and manually categorize and analyze the errors that PaLI and CLIP2CLIP are making. Particularly, we categorize the errors of the pre-trained models into four categories: (a) erroneous but relevant prediction, on concepts of the same granularity; (b) errors due to predicting very generic concepts; (c) errors due to misunderstanding the intent behind the query. (d) other miscellaneous errors. Note that errors type (d) are mostly mistakes that are unrelated and not easily interpretable. The results are shown in Table 3. Figure 4 has provided some concrete examples of the above types of mistakes made by CLIP2CLIP and PaLI. Interestingly, it shows that the two models, i.e., CLIP2CLIP and PaLI, are making very different types of errors in their predictions. Particularly, CLIP based model is good at capturing the right granularity of the entity, but often fails to understand the true intent of the text query. For instance, Figure 4 (c) shows that CLIP2CLIP ignores the text query and starts to predict the name of the barrel racer. In contrast, PaLI is good at following the text query, but can usually predict generic concepts when it does not know the answer confidently (see Figure 4 (b)).

Learning to Recognize unseen Categories There has been a significant amount of prior work [26, 53, 28] focusing on the generalization situation where information of novel categories are presented at test time. Zero-shot learning (ZSL) is one of such attempts that tackles learning new categories with zero images for training. To achieve such transfer, ZSL methods typically rely generating unseen image classifiers based on corresponding semantic representations, in the format of manually labeled attributes [26], unsupervised learned word vectors [6], or pre-trained sentence embeddings [23, 35]. Few-shot learning (FSL) [53] proposes a more realistic setup, where learners have access to a limited number of visual exemplars during the model deployment. With this goal, FSL methods aim to extract the inductive bias of learning from the seen classes, such that the model can leverage it in learning the unseen classes, to avoid severe over-fitting. Particularly, prior works either use adapted non-parametric classifiers [49, 42, 61], or meta-optimized linear classifiers [18, 37] to incorporate the few-shot unseen support examples. Comparing to them, our proposed task exposes different challenges as we ask the model to make the best use of open-world Web knowledge (i.e., Wikipedia pages with images & figures), which contains textual semantic information and visual appearance of the entities in the open world.

Vision and Language + Knowledge There have been efforts in combining knowledge into vision and language tasks, such as Visual QA [44, 32, 5, 11] and entity-focused image captioning [27, 1]. Among them, knowledge-based VQA is most related to Oven, but also differs in many aspects. Specifically, [5] presents a text QA dataset that requires understanding multi-modal knowledge in a KB. [44] propose to perform knowledge-based question answer tasks, centered around questions that resolve relational query over public Figures. Meanwhile, [32] propose to answer questions where the answer is outside of the image context, to assess model’s capability in understanding real-world knowledge More recently, [11] studies the zero-shot visual QA setting where some answers (out of a total of 500 frequent answers of general concepts) are unseen during the training, where a KB is supplied to assist the model in answering unseen answers. Comparing to them, Oven steps back to the more fundamental problem of establishing the link between visual content and entity in the KB, but at a larger scale and broader coverage. We believe that stronger models developed on Oven would benefit such knowledge-intensive visual QA tasks.

Entity Linking Entity linking (EL) is the task of grounding entity mentions in the text by linking them to entries in a given knowledge base. Supervised EL [33] has demonstrated its strong performance when all entities are in-distribution during the evaluation. Because KB is updating all the time, recent works [29, 3, 64, 13, 15] focus on a more realistic setting where entity linking needs to be achieved in the zero-shot, with a large portion of entities (to be evaluated) completely unseen during the training. Oven is a visual analog of zero-shot EL, and targets at developing generalizable models that recognize entities unseen in the training. Among all EL literature, visually assisted EL [65] is most relevant to this work, whose goal is to use the associated image of text to improve the precision of text EL. Oven is different as its text queries do not mention the name of the entities, which put visual understanding and reasoning into the central position.