A Graph-guided Multi-round Retrieval Method for Conversational Open-domain Question Answering

Conversational question answering (QA) is an emerging topic in QA. It has been formulated as the tasks of conversational knowledge-based QA (KBQA) (Saha et al., 2018; Christmann et al., 2019; Shen et al., 2019), conversational search (Dalton et al., 2020; Ren et al., 2020), conversational machine reading comprehension (MCR) (Choi et al., 2018; Reddy et al., 2019), and conversational open-domain QA (Elgohary et al., 2018; Qu et al., 2020). It is worth emphasizing that the first one relies on knowledge graphs, and the rest three tasks reply questions based upon documents. Conversational search allows users to interact with the search engine to find the documents that contain potential answers to their questions. Different from conversational search, the gold passage that contains true answers is given in conversational MCR, and it further extracts a text span as the exact answer from the given passage. Comparatively, conversational open-domain QA is more challenging in a sense that it not only retrieves the relevant passages from the Web but also extracts the text answer spans.

The common problem that all above-mentioned conversational QA tasks have to address is how to understand the historical context in the conversation. For example, there are often questions with ellipsis and/or coreference resolution problems. The common solution in the existing methods is to utilize the historical context to reformulate the current question or refine its representation. A simple strategy is to pre-append all the historical context (Elgohary et al., 2018) or heuristically select some words or sentences from the context (Choi et al., 2018) to expand the current question. Elgohary et al. (Elgohary et al., 2019) constructed the CANARD dataset, which is able to transform a context-dependent question into a self-contained question. Based on this dataset, a series of question rewriting methods have been proposed. Specifically, some regard question rewriting as a sequence-to-sequence task to incorporate the context into a standalone question. For example, Vakulenko et al. (Vakulenko et al., 2020) used both retrieval and extractive QA tasks to examine the effect of sequence-to-sequence question rewriting on the end-to-end conversational QA performance. Differently, Voskarides et. al. (Voskarides et al., 2020) modeled query resolution as a binary term classification problem. For each term appearing in the previous turns of conversation, it decides whether or not to add this term to the question in the current turn. Given the gold passage in conversational MCR, the authors in (Qu et al., 2019) designed a history attention mechanism to implement “soft selection” from conversation histories. The current conversational QA methods have presented various ways to modeling the historical context, and utilizing the historical context to enhance the current question representation. However, they ignore relations among answers (or answer passages) across the conversation turns. Some efforts (Huang et al., 2018; Chen et al., 2020) in conversational MCR captures the conversational flow by incorporating intermediate representations generated during the process of answering previous questions. Differently, we attempted to model relations more directly and explicitly by exploring the passage graph.

Open-domain question answering (QA) (Chen et al., 2017; Wang et al., 2018a, b), which aims to answer questions from the Web, has attracted wide attention from the academic field in recent years. In 2017, Chen et al. (Chen et al., 2017) first introduced neural methods to open-domain QA using a textual source. They proposed DrQA, a pipeline model with a TF-IDF based retriever and a neural network based reader that was trained to find an answer span given a question and a retrieved paragraph. Later, Wang et al.  (Wang et al., 2018a) added a ranker between the retriever and reader to rank the retrieved passages more precisely in 2018. Our proposed method also follows the retriever, ranker and reader pipeline. In addition to TF-IDF based retrievers, dense based retrievers have also been well developed recently (Lee et al., 2019; Karpukhin et al., 2020; Chang et al., 2019), where all passages are offline encoded in advance to allow efficient large-scale retrieval. In (Karpukhin et al., 2020), the authors show that the retriever can be practically implemented using dense representations in open-domain QA, where embeddings are learned from a small number of questions and passages. Our method applies both dense based and TF-IDF based retrievers. The selected top-ranked passages are further extended and explored based on the structure of our constructed passage graph by our proposed explorer.

There are also researchers exploiting the graph-based retriever for open-domain QA. Min et al.  (Min et al., 2019) utilized the available knowledge base to build a passage graph and proposed a graph retriever to encode passages. In (Asai et al., 2020), authors introduced a graph based recurrent retrieval approach that is learned to retrieve reasoning paths over the Wikipedia graph to answer multi-hop open-domain questions. Distinctly, we aimed to capture the conversation flow with the help of the passage graph. Recently, multi-round retrieval methods have been proposed (Feldman and El-Yaniv, 2019; Das et al., 2018; Xiong et al., 2020). The authors in (Feldman and El-Yaniv, 2019) proposed the multi-hop retrieval method, which iteratively retrieved supporting paragraphs by forming a joint vector representation of both question and returned paragraph. Das et al. (Das et al., 2018) proposed a gated recurrent unit to update the question at each round conditioned on the state of the reader, where the retriever and the reader iteratively interact with each other. Our proposed method also includes a multi-round retrieval mechanism. Beyond existing methods, our focus is to explore relevance feedback, particularly the returned passages, to screen out the most useful information in the historical context to better formulate the current question.

Our proposed method contains multi-rounds of retrieval, as shown in Figure 3. In the first round, given the current question $q_{k}$ and historical context $H_{k}$, DHM encodes the text as a representation vector and then feeds it to the retriever from which the feedback, i.e., retrieved passages $\mathcal{P}_{retriever}$ is received. In the following rounds, DHM utilizes $\mathcal{P}_{retriever}$ and current question $q_{k}$ to attend to history $H_{k}$ towards the representation vector refinement iteratively.

The First Round of Retrieval. We pre-append all history questions $\{q_{1},q_{2},…,q_{k-1}\}$ to the current question $q_{k}$, denoted as $q_{k}^{*}$. To accommodate the BERT (Devlin et al., 2019) based question encoder, we introduce two special tokens, [CLS] and [SEP]. $q_{k}^{*}$ is represented as a text sequence “[CLS] $q_{1}$ [SEP], $\ldots$, $q_{k-1}$ [SEP] $q_{k}$ [SEP]”. We use ALBERT (Lan et al., 2019) to obtain the first-round question representation vector $\textbf{v}_{q}$ , which is formulated as,

where $\textbf{F}_{q}$ is the ALBERT based question encoder, $\textbf{W}_{q}$ is the question projection matrix, and $\textbf{v}_{q}\in\mathbb{R}^{d_{q}}$. Similarly, for each passage $p_{i}$ in $\mathcal{C}$ we obtain its representation $\textbf{v}_{p_{i}}$ as follows,

where $\textbf{F}_{p}$ is the ALBERT based passage encoder, $\textbf{W}_{p}$ is the passage projection matrix, $\textbf{v}_{p_{i}}\in\mathbb{R}^{d_{p}}$, and $d_{p}$ is equal to $d_{q}$. It is critical that paragraph encodings are independent of questions in order to enable storing precomputed paragraph encodings and executing the efficient maximum inner product search (MIPS) algorithm (Lee et al., 2019; Karpukhin et al., 2020). Otherwise, any new question would require re-processing the entire passage collection (or at least a significant part of it). Benefiting from this, we can calculate the similarity score via the inner product of $\textbf{v}_{q}$ and each passage $\textbf{v}_{p_{i}}$ efficiently, and select the top-$n_{1}$ passages $\mathcal{P}_{retriever}=\{p_{1}$, $p_{2}$, $\ldots$, $p_{n_{1}}\}$, where $n_{1}$ is the number of the passages in $\mathcal{P}_{retriever}$.

The Following Rounds of Retrieval. The returned passages in $\mathcal{P}_{retriever}$ are incorporated into the history questions $H_{k}$ and the current question $q_{k}$ in the form of triplet $\{q_{i}^{t}\}_{i=1}^{k-1}$, where each triplet $q_{i}^{t}$ contains the current question $q_{k}$, the top-$n_{r}$ passages in $\mathcal{P}_{retriever}$, and the $i$-th history question. It is formulated as $q_{i}$ “[CLS] $q_{k}$ [SEP] $p_{1}$, …, [SEP] $p_{n_{r}}$ [SEP] $q_{i}$ [SEP]”. As such, interactions among the history question, the current question and the retrieval feedback are encouraged. The representation of $q_{i}^{t}$ is calculated as follow,

A history attention network is then followed to aggregate the $k-1$ representation vectors into the refined question vector $\textbf{v}_{q}$ with learned attention weights. Formally, the attention layer is defined as follows,

where $\textbf{W}_{a}\in\mathbb{R}^{1\times d_{q}}$, $\alpha_{i}$ denotes the attention weight of the $i$-th triplet, and $\textbf{v}_{q}\in\mathbb{R}^{d_{q}}$. The attention weights help to capture certain contexts that are more relevant to the current round of retrieval. With the new question representation $\textbf{v}_{q}$, we re-calculate the similarity scores and select the new list of top-$n_{1}$ passages as before. The question representation vector $\textbf{v}_{q}$ is continuously updated when the returned passage changes in each round. After several rounds, the final rank list of passages $\mathcal{P}_{retriever}$ is delivered to the following Explorer.

Essentially, our proposed explorer is equipped with a graph-guided candidate answer passage expansion method. Based on the observation that answers are also highly related in conversation QA, we design Explorer to model relations of answers by constructing a passage graph. In general, the explorer goes through the following three steps: initialization, expansion, and graph modeling, as illustrated in Figure 4.

We first construct an initial passage graph $\mathcal{G}_{0}=<\mathcal{P}_{0},\mathcal{E}_{0}>$, where $\mathcal{P}_{0}$ are some selected passages and $\mathcal{E}_{0}$ are hyperlinks among them. $\mathcal{P}_{0}$ includes the passages containing gold/predicted answers in previous turns of the conversation and the passages retrieved by the dense retriever described in the previous section. To increase the pool of relevant passages, we also run a simple TF-IDF based retrieval model. Altogether we have

where $\mathcal{P}_{a}$, $\mathcal{P}_{retriever}$, and $\mathcal{P}_{T}$ denote the set of passages that contain previous gold/predicted answers, the passages returned by the dense retriever, and the passage returned by the TF-IDF retriever, respectively. Starting from the initial passage nodes $\mathcal{P}_{0}$, we expand the passage set to $\mathcal{P}_{m}$ by adding the hyperlink-connected passages via,

where $\mathcal{P}_{e}$ denotes the passages linked to the passages in $\mathcal{P}_{m-1}$ via the Web hyperlinks. Through expansion, a larger set of passages $\mathcal{P}_{m}=\{p_{i}\}_{i=1}^{n_{m}}$ is obtained, where $n_{m}$ refers to the number of passages in $\mathcal{P}_{m}$. Meanwhile, we obtain the corresponding expanded passage graph $\mathcal{G}_{m}$, whose nodes are passages in $\mathcal{P}_{m}$ and edges are hyperlinks among them.

To take advantage of the structure information, we apply a Graph Attention network (Veličković et al., 2018) to update the embeddings of the passages in $\mathcal{G}_{m}$. For each passage $p_{i}$ in $\mathcal{G}_{m}$, we input it to an ALBERT encoder to obtain its embedding by Eqn.(2) and then update the node embedding as follows,

Based on the updated embeddings, we re-calculate the similarity between the question and the passages in $\mathcal{G}_{m}$. Specifically, we obtain the question embedding $\textbf{v}_{q}$ by Eqn.(1) and calculate the similarity $S_{a_{i}}$ of passage $p_{i}$ to $\textbf{v}_{q}$ as follows,

Finally, the top-$n_{2}$ passages $\mathcal{P}_{explorer}$ are selected according to their ranks based on similarity calculation, where $n_{2}$ denotes the number of passages in $\mathcal{P}_{explorer}$. $\mathcal{P}_{explorer}$ is then passed to the following Ranker and Reader for further processing.

Given the list of passages $\mathcal{P}_{explorer}$, Ranker re-ranks the passages more precisely and Reader predicts a text span as an answer. We first construct the reformatted text sequence $[q,p_{i}]$ by concatenating the question text and the passage $p_{i}$ text. $[q,p_{i}]$ is “[CLS] $q_{k}^{*}$ [SEP] $p_{i}$ [SEP]”. For each token $token_{j}$ in $[q,p_{i}]$, a BERT encoder generates its representation $\textbf{v}_{t_{j}}$, and for the whole text sequence, BERT encoder generates its representation $\textbf{v}_{q,p_{i}}$.

Ranker conducts a listwise reranking of the top-$n_{2}$ passages received from Explorer and calculates the reranking score for each passage $p_{i}$ in $\mathcal{P}_{explorer}$ as,

where $\textbf{W}_{ra}\in\mathbb{R}^{d_{q}\times 1}$. Reader then predicts an answer span by computing two scores for each token $j$ in passages in $\mathcal{P}_{explorer}$ as the start token and the end token, respectively, formulated as,

where $n_{t}$ is the number of tokens, and both $\textbf{W}_{s}$ and $\textbf{W}_{e}\in\mathbb{R}^{d_{q}\times 1}$. Finally, the text span with the highest score is extracted as the answer. We will detail the inference process in the following section.

Training. Recall that we encode the large collection of passages $\mathbb{C}$ offline for efficient retrieval. Specifically, we follow the previous work (Lee et al., 2019; Qu et al., 2020) to pretrain a passage encoder so that it can provide reasonably good retrieval results to the subsequent components for further processing. After offline encoding, a set of passage vectors are obtained. Note that while the parameters of the passage encoder $\textbf{F}_{p}$ are fixed after pretraining, the parameters of the question encoder $\textbf{F}_{q}$ are updated along with those of the explorer, the ranker and the reader.

After passage offline encoding, we then train the retriever, the explorer, the ranker, and the reader. For each passage $p_{i}$ in $\mathcal{P}_{retriever}$, we calculate the retrieval score $S_{re_{i}}$ as follows,

To train the retriever, we set the retriever loss $L_{retriever}$ as follows,

where $y$ indicates whether the passage is a gold passage. If the gold passage is not presented in $\mathcal{P}_{retriever}$, we manually include it in the retrieval results. Similar to the retriever loss, we respectively define the explorer loss $L_{explorer}$ and the ranker loss $L_{ranker}$ as follows,

where $S_{a_{i}}$ and $S_{b_{i}}$ are the explorer score and ranker score of the passage $p_{i}$ in $\mathcal{P}_{explorer}$, respectively. The reader loss $L_{reader}$ is formulated as,

where $y_{1}$ and $y_{2}$ indicate whether the token is the start token and the end token, respectively. Considering the limitation of GPU memory, we first train the retriever, the ranker, and the reader jointly via the sum of their losses, and then train DHM and the explorer separately.

Inference. For each passage in $\mathcal{P}_{explorer}$, we obtain the explorer score $S_{a}$ and the ranker score $S_{b}$ by Eqn.(8) and Eqn.(9), respectively. For each token, the reader then assigns it the probabilities of being the start token $S_{s}$ and the end token $S_{e}$. Following the convention (Devlin et al., 2019; Qu et al., 2020), we consider the top 20 text spans only to ensure tractability. Invalid predictions, including the cases where the start token comes after the end token, or the predicted span overlaps with a question in the conversation context, are all discarded. The predicted score $S$ of a potential answer is then calculated as,

The model finally outputs the answer span with the maximum overall score to respond to the current question.

To conduct experiments, we used the public available dataset OR-QuAC  (Qu et al., 2020), which expands the QuAC dataset (Choi et al., 2018) to the open-domain QA setting. Each conversation in this dataset contains a series of questions and answers, and is supplemented with a collection of passages. There are totally 5,644 conversations covering 40,527 questions. The passage collection is from the English Wikipedia dump file of 10/20/2019, and it contains about 11 million passages. We further crawled the internal page links in the Wikipedia via Wikiextractor⁵⁵5https://github.com/attardi/wikiextractor. to facilitate passage graph construction. The statistics of the above dataset are summarized in Table 1.

Evaluation Protocols. Following the previous work (Choi et al., 2018; Qu et al., 2020), we employed the word-level F1 and the human equivalence score (HEQ), which are two metrics provided by the QuAC challenge (Choi et al., 2018) to evaluate the Conversational QA systems. HEQ computes the percentage of examples for which system F1 exceeds or matches human F1. It measures whether a system can give answers as good as an average human. This metric is computed on a question level (HEQ-Q) and a dialog level (HEQ-D). Besides, to evaluate the retrieval performance of the retrieval passage list, we also applied the Mean Reciprocal Rank (MRR) and Recall to evaluate the Retriever component of the baselines, our Explorer component, and the Ranker component.

Implementation Details. We utilized the pretrained passages embedding vectors released in (Qu et al., 2020) to make a fair comparison. The $d_{q}$, $d_{p}$ are both set to 128, and $n_{2}$ is set to 5, the same as (Qu et al., 2020). The number of retrieved passages from the retriever $n_{1}$ is 3. Limited by our GPU memory, both $m$ and $n_{r}$ is set to 1, and the retrieve rounds are 2. There are two GAT layers, where the numbers of heads are 128 and 64, respectively. The passage embeddings are stored in a 2080Ti card and the model is in another 2080Ti card.

To the best of our knowledge, only one published work (Qu et al., 2020), ORConvQA, aims to solve the conversational open-domain QA problem. We compared our proposed method with ORConvQA and the baselines in (Qu et al., 2020) as follows:

• •

  DrQA (Chen et al., 2017): This model uses a TF-IDF retriever and a RNN based reader. The original distantly supervised setting is not applied, since the full supervision is allowed in the dataset and adopted for all the methods.

• •

  BERTserini (Yang et al., 2019): This model uses a BM25 retriever implemented in Anserini ⁶⁶6http://anserini.io/. and a BERT reader. Their BERT reader is similar to ours, except that it does not support reranking and thus cannot benefit from joint learning.

• •

  ORConvQA (Qu et al., 2020): It is an open-retrieval conversational question answering method. It uses a dense retriever, ranker, and reader pipeline, which is similar to ours. ORConvQA utilizes a sliding window to append previous questions.

• •

  ORConvQA without history (ORConvQA w/o hist.): This is ORConvQA with the history window size $w=0$. However, the first question is always included in the reformulated current question.

Following the previous work (Qu et al., 2020), we evaluated all methods under the setting of no true history answers. The results of these baselines are public in (Qu et al., 2020). We also evaluated ORConvQA and ours with true history answers, denoted as ORConvQA-ta and “Ours-ta”, respectively.

The results of all methods are summarized in Table 2, from which we have the following findings.

(1) As expected, traditional single-turn open-domain QA methods, i.e. DrQA and BERTserini, perform worse than the conversational QA methods, since they consider no historical context. And thus they fail to solve the elliptical and coherence problems in conversations. Moever, the methods based on a BERT reader, i.e. BERTserini, ORConvQA, and “Ours”, have a significant improvement over DrQA, a RNN based reader. This indicates pretrained BERT is more powerful than RNN in producing semantic representations.

(2) The dense retriever based methods, including ORConvQA and “Ours”, surpass the BM25 retriever based methods. This is also verified in the single-turn QA (Lee et al., 2019; Karpukhin et al., 2020). On the one hand, the semantic similarities can be learned and reflected via the distances in the embedding space. On the other hand, the dense retrievers allow joint learning of different components in the conversational open-domain QA pipeline.

(3) Our method achieves the best performance, substantially surpassing all baselines. In specific, our method has a dense retriever and a BERT reader, outperforming DrQA and BERTserini consistently. Although our method has the same ranker and reader as ORConvQA, our method still considerably surpasses it. This verifies that our proposed graph-guided and multi-round dynamic retrieval method is able to provide a better list of passages containing more relevant passages to the following ranker and reader.

(4) Both ORConvQA and “Ours” with true history answers surpass with predicted history answers. This demonstrates that the history answers indeed provide useful and necessary information for the current question understanding, which is consistent with the observation on the QuAC dataset (Choi et al., 2018). “Ours-ta” significantly outperforms ORConvQA-ta, and it improves the recall to 0.84. That is because that our proposed Explorer fully takes advantages of the history answers by activates them in the passage graph. Therefore, when true history answers available, gold passages can be easily found via the graph structure.

To justify the performance of the retrieval passage list, we compared the performance of our model and the best baseline ORConvQA by varying the number of retrieved passages $n_{2}$. From Figure 5, our observations are as follows,

(1) Looking at both Figure 5(a) and 5(b), we can see that when the number of returned passages increases, both Recall and MRR rise. This is because more and more gold passages are retrieved with the increased number of returned passages. We can also find that the F1 score increases fast at the beginning and then gradually degrades. This shows that although increasing the number of returned passages benefits Recall and MRR, it also becomes a heavy burden to the reader as more noisy passages are also delivered to it.

(2) In terms of Rt-R, “Ours” outperforms ORConvQA significantly. The retrieved passages of ours contain more gold passages, because our proposed explorer find more relevant passages via the graph structure. But in terms of Rt-M, ORConvQA exceeds “Ours”. It may be because of the GAT layer, which allows the embeddings become more similar to the surroundings. Thus, it is easier for our explorer to retrieve more gold passages and their neighbors but cannot precisely distinguish them. However, compared to MRR, improving recall is more important for the retriever in the pipeline, because the following ranker can further re-rank the passages in the list. As shown in Table 2, the Rr-M of ours outperforms that of ORConvQA, demonstrating that our ranker works well and provides a complementation to our explorer.

We further tested the variants of our model to further verify the effectiveness of the components in the pipeline.

• •

  Ours w/o Ranker: The ranker is excluded from the pipeline. And the ranker’s score $S_{b}$ is removed from the final predicted score in Eqn.(15).

• •

  Ours w/o DHM: DHM is excluded from the pipeline. That is, the question representation vector is generated as described in the first round of retrieval. Since there is no dynamic history modeling, there is no multi-round retrieval process and no relevance feedback from the retriever.

• •

  Ours w/o Explorer: The Explorer is excluded from the pipeline. That is, the passage list $\mathcal{P}_{retriever}$ rather than the list $\mathcal{P}_{explorer}$ is delivered to the ranker and the reader.

We compared these three variants, and summarized the results in Table 3. By jointly analyzing Table 3, we gained the following insights.

(1) Removing the ranker component degrades the QA performance. To be more specific, “Ours w/o Ranker” drops by 0.3 in terms of F1. This statistic reveals the effectiveness of the ranker component. Although the ranker loss does not influence the retriever, the retriever performance also decreases. This is because that the joint training mechanism can improve all different components.

(2) “Ours” surpasses “Ours w/o Explorer”, which indicates that incorporating the passage graph is indeed beneficialin boosting the performance. Moreover, compared with “Ours w/o Ranker”, the performance of “Ours” drops more, which again reflects the effectiveness of our graph-guided retrieval component.

(3) “Ours” shows the consistent improvements over “Ours w/o DHM”. The improvement in terms of F1 is 0.8. Such phenomenon clearly reflects the great advantage of our novel multi-round dynamic retrieval method.

(4) By comparing three variants, it is clear that when the Explorer is excluded, the system performance drops the most. We believed that the improvement of our method comes from the graph guided retrieval. Meanwhile, when the Ranker is excluded, it only causes less impact on the overall performance.

Impact of the number of passages in $\mathcal{P}_{retriever}$. As mentioned in Section 3.2, the Retriever component retrieves $n_{1}$ passages and then Explorer utilizes these passages to activate the corresponding nodes in the passage graph. Therefore, $n_{1}$ is an import hyper-parameter that needs to be explored. To this end, we carried out experiments on the development set to verify the impact of the number of passages in $\mathcal{P}_{retriever}$. The choices of $n_{1}$ are from $0$ to $9$, and the comparison results are illustrated in Figure 6. It demonstrates that if the passages from the Retriever are not included to activate the nodes, the performance drops a lot. This is because that the retrieval result help locate an appropriate area in the big passage graph, and thus it contributes to the retrieval of the gold passage. Besides, as the number of passages from the Retriever increases, the F1 score rises and then gradually degrades. This experimental result tells us that more seed nodes do not represent better performance, because it also make it more difficult to retrieve the gold one when the number of nodes in the sub-graph gets larger.

Impact of the number of passages from TF-IDF. Similarly, the Explorer utilizes the passages from TF-IDF to activate the nodes in the passage graph. Therefore, we conducted experiments to explore the number of passages from TF-IDF, and the number is set from $0$ to $5$. The results are summarized in Figure 6. We can observe that selecting a certain amount of passages from TF-IDF as seed nodes benefiting the QA system. However, the performance drops when the number of passages is increased from $0$ to $1$. This may be due to the fact that the quality of the passage list from TF-IDF is not as good as that from the dense retriever, thus the position of the gold passage is not at the front. And as the number of passages increases, the F1 score gradually rises because more and more relevant passages are added.

Impact of the number of retrieve rounds. As claimed before, our proposed DHM makes the multi-round retrieval possible, thus it is worth to exploit the number of retrieval rounds. However, due to the limitation of the GPU memory, we only tuned this value on the dev set when evaluating the system. The results are shown in Figure 6. We found that increasing the retrieval rounds does not improve the performance continuously. On the one hand, we did not tune the number of retrieval rounds in the training process. On the other hand, multi-round retrieval may bring more noise and probably suffer from the error propagation problem.

To gain deeper insights into the performance of our proposed method in conversational open-domain QA, we listed examples in Table 4. Each example includes the current question, history questions, and attention weights. To illustrate our proposed graph-guided and multi-round retrieval method, we also listed the labels of the retrieved passages from the 1-round retrieval, 2-round retrieval, the Explorer, and the Ranker. From examples in Table 4, we had the following observations.

Current questions. (1) Analyzing the current questions, we found that the coreference problem is common. For example, the pronoun “he” in question “Did he play any live shows?” refers to the name “Hank Snow” in the first history question. (2) It is also observed that there may be two coreferences in one question. For example, for the question “Did he accept and appear on the show”, “he” and “the show” refer to “Sakis Rouvas” and “the first tv appearance” in the history question, respectively. (3) Moreover, the current question may depend on the history answers. For the second question “Why did he burn it?”, we cannot clarify what the word “it” means only based on the history questions, and which turns out to be “the Fantoft Stave Church” in the previous answer “On 6 June 1992, the Fantoft Stave Church, dating from the 12th century and considered architecturally significant, was burned.”. This observation is consistent with our experiment results that “Ours-ta” and ORConvQA-ta largely surpass the ones than that without the previous true answers.

Attention weights. (1) By jointly analyzing the history questions and attention weights, we found that the attention weights approximately reflect the importance of history questions. For example, in the first and second example, the first history question obtains the largest attention score. This adheres to our intuition because the first history question usually mentions the topic of the conversation. (2) It surprises us that the second history question of the second example also achieves a comparative attention score. As claimed before, in the second example, the gold passage and answer of the second history question provide important information to help understand the current question. Therefore, a comparative attention score becomes reasonable because it makes it easier to retrieve the corresponding evidence to clarify the current question. (3) The attention weights are not significantly discriminative. For example, in the first example, the attention weight of the fourth history question is 0.1858 although it may only provide little of useful information. This is partly due to the fact the questions in a conversation are always related more or less, since they usually have a same or similar topic.

Retrieval result. (1) From the result of the 1-round of retrieval, we found that it is easier for the dense retriever to handle the simple question, like the first example that contains one pronoun. However, for some more complex questions, it seems that it works not so well. (2) Analyzing the result of 2-round of retrieval, we found that it can make a supplement for the 1-round of retrieval in some cases. For example, in the third example, 1-round of retrieval does not retrieve the gold passage but 2-round of retrieval does. This may illustrate our dynamically multi-round retrieval method works partly due to its ability of handling the complex questions. And this observation is consistent with some recent studies (Xiong et al., 2020; Das et al., 2018) that focus on applying the multi-round retrieval method to answer the complex questions in open-domain QA. (3) It also can be observed that the Explorer obtains the best retrieval result, which is consistent with the experiment results in Section 4.4. For example, in the second example in Table 4, the Explorer retrieves the gold passage while both the 1-round and 2-round of retrieval do not. This is because our proposed Explorer models the relation among answers in the conversation. However, the position of the gold passage in the result of the Explorer is not as front as in the result of the Retriever. This is may be because of the graph neural network, but it is acceptable because the main target of retrieval is to improve the recall as much as possible. And fortunately, the following Ranker component refines the rank list.