How Can Context Help?
Exploring Joint Retrieval of Passage and Personalized Context

With the recent developments in AI, the world has witnessed an eruption of chatbots deployed with LLMs (large language models), such as ChatGPT, BARD Cohen et al. (2022), BlenderBot Shuster et al. (2022b), which often generate texts indistinguishable from human fluency. However, chatbots powered by parameter-based LLMs are known to generate factually incorrect statements - a problem regardless of the model size Shuster et al. (2021). By leveraging an external corpus of knowledge, retrieval augmented systems Dinan et al. (2018); Lewis et al. (2020); Karpukhin et al. (2020), including document-grounded dialogue systems Roller et al. (2020); Shuster et al. (2022b, a); Cohen et al. (2022), have demonstrated several advantages compared to pure parameter-based systems. For instance, grounding responses on external knowledge bases have been shown to reduce hallucinations across a variety of retrieval systems and model architectures Shuster et al. (2021).

A document-grounded conversational system, particularly in the enterprise setting, is likely to have access to a significant amount of contextual information, whether as a knowledge base or a library of API calls. This context may be temporal, such as the current date and time, or recent events; or it may be user-specific, such as information about the user’s account, profile, recent transactions, activity logs, etc. Without any such context, a user’s question such as, "Am I eligible for this rebate?" would receive the generic answer "You may be eligible for this rebate depending on where you live," grounded only on the relevant documents. If the right context were also retrieved and made available, the response could be instantly elevated to "Yes, you are eligible since you live in Singapore." Furthermore, retrieving the correct context information may serve to better understand the user’s intent, and therefore improve the likelihood of identifying the correct grounding document to use.

The significant challenge of choosing which context to retrieve has great potential business value, but has not been well-studied. Including too much contextual information may result in too much noise to the generation step, or exceeding the LLM’s allowed input size. Including irrelevant contextual information may degrade the generated response. Our motivating question can thus be posed as follows: Given a user query (which itself may be underspecified), a document collection, and a set of available contexts, how can a document-grounded conversational system retrieve a good subset of contexts to help answer the query, and can this process also help in retrieving the most relevant grounding document(s)?

To this end, we propose a new task of personalized context-aware passage retrieval for document-grounded dialogue, and create a dataset, ORCA-ShARC, for this setting. We provide several baseline approaches, as well as develop a novel approach, Personalized Context-Aware Search (PCAS), to address the task.

In order to showcase the efficacy of PCAS, we conduct extensive experiments on multiple well-known retrieval systems. The results illustrate that PCAS not only surpasses the baselines in retrieving the most relevant passage but also excels in identifying the pertinent context. We hope that our advancements in joint context-passage retrieval will serve as a catalyst, motivating future research endeavors in this highly promising field.

We now formally define the task of context-passage retrieval, which involves not only retrieving the relevant document from the external knowledge base, but also selecting the relevant piece of context from all the available contexts.

Formally, when a user $u$ engages in a conversation with the system, in addition to the static document corpus $\mathcal{D}$, all the available personalized context information $\mathcal{C}^{u}$ are accessible to the system. There is also the conversation history composed of utterances that have already occurred in this session between the user and the system: $H=\{r_{1}:X_{1},...,r_{i}:X_{i},...\}$ where $r_{i}$ is the speaker role, and $X_{i}$ is the utterances at the $i$-th turn, respectively. Since the focus of our work is in context-passage retrieval rather than the conversation history, in the rest of the paper, we simply consider a single turn user query $q$ rather than $H$.

Given the input of a user query $q$, the task is to select 1) the most relevant latent document $d$ from $\mathcal{D}$; and 2) the most relevant latent context $c$ from $\mathcal{C}^{u}$, to help the system generate a good response.

To evaluate the retrieved documents and context, we use standard retrieval metrics, including binary rank-aware metrics MAP (mean average precision) and decision support metrics Recall@$K$.

To the best of our knowledge, there is no existing open-retrieval content-grounded dialogue or QA datasets where each document-grounded example is annotated with a set of context. To this end, we curate a dataset for the proposed task in Section 2.

ShARC Saeidi et al. (2018) is a conversational QA dataset focusing on question answering from given text and one piece of given context (scenario). OR-ShARC Gao et al. (2021) is adapted from the ShARC dataset to an open-retrieval setting, where the task is to retrieve the relevant text snippet from the whole corpus. In OR-ShARC, each example is given one piece of relevant context (scenario).

We create a dataset, ORCA-ShARC (Open-Retrieval Context-Aware ShARC), by converting the OR-ShARC dataset into our task setting, where a set of contexts is provided for each example. To create the set, we use the example’s original relevant context, and expand the set by randomly sampling from all the contexts appearing in the OR-ShARC dataset, as long as there is no contradiction between contexts introduced (as judged by prompting FLAN_T5_3B model Chung et al. (2022)). We include 10 pieces of context for each example.¹¹1As the size of the context set grows, it naturally becomes harder to add context without contradictions. A few examples could only support 6-9 pieces of context. Table 1 summarizes the statistics of the ORCA-ShARC dataset and Table 2 provides an example.

We compare our approach with three baselines, and use some of the most popular neural retrieval systems to address the context-and-passage retrieval task on the newly constructed dataset.

We evaluate the baselines and PCAS in a $0$-shot context-and-passage retrieval task on the ORCA-ShARC dataset. We conduct experiments using several popular pretrained modern neural retrieval systems including a late-interaction retrieval model ColBERT Khattab and Zaharia (2020); Santhanam et al. (2021), single-vector retrieval models DPR Karpukhin et al. (2020), ANCE Xiong et al. (2021a) and Sentence BERT (S-BERT) Reimers and Gurevych (2019) with DistilBERT-TAS-B model Sanh et al. (2019); Hofstätter et al. (2021) .

For ColBERT, we adapt the code from the ColBERT repository ²²2https://github.com/stanford-futuredata/ColBERT. From the BEIR Thakur et al. (2021) repository³³3https://github.com/beir-cellar/beir, we get the pretrained model names, as well as the code for the other dense retrieval systems. We use the py_trec toolVan Gysel and de Rijke (2018) ⁴⁴4https://github.com/cvangysel/pytrec_eval for evaluation.

In Table 3, we present the document retrieval results and the context selection results (limited to the approaches that predict the context: B2, B3 and PCAS). For the same approach, the results largely varies across the retrievers, due to the distinct models and different pre-training data and processes. For example, DPR results are on the low side, especially for B3 document metrics, because it involves two chained retrieval steps that amplify this effect. B1 yields the lowest accuracy across the board, mainly due to the noise introduced by including all the contexts without discrimination. Importantly, we observe that when the original relevant context is unknown, our proposed PCAS approach achieves better retrieval results than all baselines, which indicates that jointly considering the documents and contexts can improve the performances of both document and context retrieval. Note that the PCAS results are close to the OR experiment in which the original relevant context is given. This suggests that PCAS can identify the relevant and important contexts for the query, with no need for users to specify any contexts. Furthermore, the comparison between B2 and B3 illustrates that the retrieval process of $q\rightarrow d\rightarrow c$ is better than $q\rightarrow c\rightarrow d$ , which supports the motivation of our PCAS design.

Our proposed task is different from the work on context-aware QA Seonwoo et al. (2020); Taunk et al. (2023) in which QA is done on a given document and no retrieval is involved. On the other hand, context-aware QA can be considered as the next step following our task.

Our work is closely related to the work on contextual information retrieval MERROUNI et al. (2019). The major difference from our work is that the tasks in this line of work do not involve selecting of relevant context. The form of context being used is structured and in a few pre-defined genres, whereas we leverage a set of unstructured contexts. There is also no joint relevance modeling of both context and content with respect to the query. Our task and approach is also different from contextual recommendation systems (on which Tourani et al. (2023) recently presents their work), which does not involve user questions or queries.

Our proposed data and task are also related to open domain question answering (Kwiatkowski et al., 2019; Lewis et al., 2020; Min et al., 2020; Qu et al., 2020; Izacard and Grave, 2021; Li et al., 2021; Xiong et al., 2021b; Yu et al., 2021), open-retrieval conversational QA (Qu et al., 2020; Gao et al., 2021), and open-retrieval document-grounded dialogues Feng et al. (2021). However, none of these datasets and tasks include external context information. The lone exception is OR-SHARC, which provides one relevant context for each example and does not involve any selection of relevant context from a larger set.

Finally, our work is related to Multi-Session Chat (MSC) Qian et al. (2021), a dataset consisting of multiple chat sessions, created for studying how to utilize information outside of the current conversation session. Similarly,  Xu et al. (2022) recently leveraged retrieval-augmented methods to select useful contexts from previous chat sessions. However, the datasets in both works are not document-grounded, and retrieving documents jointly with contextual information is not explored.

This work proposes the task of context-aware passage retrieval and creates a dataset based on OR-ShARC. We also present a novel approach for integrating external context information into the retrieval aspect of document-grounded conversational systems. The proposed PCAS method effectively combines both the document-query relevance and contextual relevance. We conduct experimental evaluations on popular retrieval systems, including ColBERT, DPR, ANCE, and S-BERT. The results demonstrate that incorporating external context information through PCAS brings significant improvement on passage retrieval and achieves higher MAP and Recall@$K$ than baseline models.

The proposed retrieval paradigm opens up avenues for future research and extensions. Several potential directions include extending the PCAS method to the training process, integrating downstream modules such as response generation, and creating real-world context datasets with the inclusion of human feedback. These topics offer promising directions for the community to explore and advance the field further.

Although the motivation for our work is to improve the quality of conversation systems that could be grounded on both document content and contextual information, in this work we focus exclusively on the task of retrieving the content and context. We do not evaluate the resulting generative responses to the user query, which is the actual final desired outcome, though we are working on doing so in the near future.

There are no publicly available datasets that are suited to the context-aware passage retrieval task. To the best of our knowledge, some of the related datasets do not have a passage retrieval setting, the rest of them either do not have annotated context set, or have a few limited genre contexts which does not serve our motivation with rich context from various business applications. This leads us to create a dataset by ourselves. Experimenting on only one dataset may limit the more generalizable conclusions we can draw.

The size of the context set that we construct is limited. In a realistic setting, it is likely that the number of available contexts is much greater, and more heterogeneous in nature rather than the short unstructured text available from the OR-ShARC dataset. More work is needed to discover how our approach scales to these settings.

To the best of our knowledge, we have not identified any immediate ethical concerns or negative societal consequences arising from this work, in accordance with the ACL ethics policy. The dataset we created does not involve generating any data with LLMs, hence does not impose any risk of non-factual or harmful content.

We hope our work may serve as an inspiration for the community to utilize the newly created dataset and further enhance retrieval performance. In the future, it may be necessary to tread carefully when incorporating user context in a real-world setting, for example to not cause consternation about how much a system may know about a given user.

This work was partly supported by the Rensselaer-IBM AI Research Collaboration (http://airc.rpi.edu), part of the IBM AI Horizons Network (http://ibm.biz/AIHorizons).