Generative Context Pair Selection for Multi-hop Question Answering

Recently many reading comprehension datasets like HotpotQA Yang et al. (2018) and WikiHop Welbl et al. (2018) that require compositional reasoning over several disjoint passages have been introduced. This style of compositional reasoning, also referred to as multi-hop reasoning, first requires finding the correct set of passages relevant to the question and then the answer span in the selected set of passages. Most of these dataset are often collected via crowdsourcing, which makes the evaluation of such models heavily reliant on the quality of the collected held-out sets.

Crowdsourced datasets often present only a partial picture of the underlying data distribution. Learning complex latent sequential decisions, like multi-hop reasoning, to answer a given question under such circumstances is marred by numerous biases, such as annotator bias Geva et al. (2019), label bias (Dua et al., 2020; Gururangan et al., 2018), survivorship bias (Min et al., 2019; Jiang and Bansal, 2019), and ascertainment bias Jia and Liang (2017). As a result, testing model performance on such biased held-out sets becomes unreliable as the models exploit these biases and learn shortcuts to get the right answer but without learning the right way to reason.

Consider an example from HotpotQA in Figure 1, where the latent entity “Virgina Commonwealth University" can be used by the model Jiang and Bansal (2019) to bridge the two relevant passages (highlighted in green) from the original dev set and correctly predict the answer “1838”. However, upon adding an adversarial context (highlighted in pink) to the pool of contexts, the model prediction changes to “1938” implying that the model did not learn the right way to reason. This is because the discriminatively trained passage selector exploits lexical cues like “founded” in the second passage and does not pay attention to the complete question. The absence of such adversarial contexts at training allows the model to find incorrect reasoning paths.

In this work, we propose a generative context pair selection model, which tries to reason through the data generation process of how a specific question could have been constructed from a given pair of passages. We show that our proposed model is comparable in performance to the state-of-the-art systems, with minimal drop in performance on the adversarial held-out set. Our generative passage selector shows an improvement of 4.9% in Top-1 accuracy as compared to discriminatively trained passage selector on the adversarial dev set.

Given a set of contexts $C=\{c_{0},c_{1},...c_{N}\}$, the goal of multi-hop question answering is to combine information from multiple context passages to identify the answer span $a$ for a given question $q$. In single-hop QA, the goal is to identify the pair of contexts, from all possible pairs $\psi=\{(c_{i},c_{j}):c_{i}\in C,c_{j}\in C)\}$, that is appropriate for answering the question.

Existing models for multi-hop question answering Tu et al. (2020); Chen et al. (2019) consist of two components: a discriminative passage selection and an answering model. Passage selection identifies which pairs of contexts are relevant for answering the given question, i.e. estimates $p(c_{ij}|q,\psi)$. This is followed by the answering model to extract the answer span given a context pair and the question ($p(a|q,c_{ij})$). These are combined as follows:

The discriminative passage selector learns to select a set of contexts by conditioning on the question representation. This learning process does not encourage the model to pay attention to the entire question, which can result in ignoring parts of the question, thus, learning spurious correlations.

To predict the answer at test time, we do not sum over all pairs of contexts, but instead use the top scoring pair to answer the question¹¹1Summing over all context pairs, or maintaining a beam of highly ranked pairs, did not yield much higher performance, in particular, not worth the additional computation cost..

In other words, we use passage selection to pick the best context pair $c^{*}_{ij}$, which is used by the answering module to get the answer, $a^{*}=\operatorname*{argmax}p(a|q,c^{*}_{ij})$.

We experiment with two popular multi-hop datasets: HotpotQA Yang et al. (2018) and WikiHop Welbl et al. (2018). Most SOTA passage selection modules for HotpotQA use a RoBERTa Liu et al. (2019) based classifier to select top-k passages given the question, which has an accuracy of $\sim$94.5% Tu et al. (2020). We used a T5-based standard passage selector, $p(c_{ij}|q,\psi)$, as our baseline, which provides a comparable performance to SOTA passage selector (Table 1).

We also use a simple T5-based answering model that has a comparable performance to SOTA answering models to illustrate the effect of our generative selector on end-to-end model performance. The oracle EM/F1 of our answering model, $p(a|q,c_{gold})$, on HotpotQA and WikiHop are 74.5/83.5 and 76.2/83.9 respectively. The overall EM/F1 of WikiHop with generative model are 73.5/80.2.

Most passage selection models for HotpotQA and Wikihop’s distractor style setup employ a RoBERTA based context selectors given the question Tu et al. (2020); Fang et al. (2019). In an ideal scenario, the absence of latent entity in the question should not allow selection of all oracle passages. However, the high performance of these systems can be attributed to existing bias in HotpotQA Jiang and Bansal (2019); Min et al. (2019). Another line of work dynamically updates the working memory to re-rank the set of passage at each hop Das et al. (2019). With the release of datasets like SearchQA Dunn et al. (2017), TriviaQA Joshi et al. (2017), and NaturalQuestions Kwiatkowski et al. (2019), lot of work has been done in open-domain passage retrieval, especially in the full Wikipedia setting. However, these questions do not necessarily require multi-hop reasoning. A series of work has tried to match a document-level summarized embedding to the question Seo et al. (2018); Karpukhin et al. (2020); Lewis et al. (2020) for obtaining the relevant answers. In generative question answering, a few works  Lewis and Fan (2018); dos Santos et al. (2020) have used a joint question answering approach on single context.

We have presented a generative formulation of context pair selection in multi-hop question answering models. By encouraging the context selection model to explain the entire question, it is less susceptible to bias, performing substantially better on adversarial data than existing methods that use discriminative selection. Our proposed model is simple to implement and can be used with any existing (or future) answering model; we will release code to support this integration.

Since context pair selection scales quadratically with the number of contexts, it is not ideal for scenarios that involve a large number of possible contexts. However, it allows for deeper inter-document interaction as compared to other approaches that use summarized document representations. With more reasoning steps, selecting relevant documents given only the question becomes challenging, increasing the need for inter-document interaction.

This paper focuses on biases found in question answering models that make its reasoning capabilities brittle. It uses an existing method of testing model performance on adversarial held-out set as an evaluation metric. This work does not deal with any social impacts of biases in natural language processing systems.