Knowledge Base Question Answering by
Case-based Reasoning over Subgraphs

Given the input query $q$, Cbr-subg first retrieves other similar cases from the training set. We represent the query by embeddings obtained from large pre-trained language models (Liu et al., 2019). Inspired by the recent advances in neural dense passage retrieval (Karpukhin et al., 2020), we use a Roberta-base encoder to encode each question independently. A single representation is obtained by mean pooling over the token-level representations.

Generally, we want to retrieve questions that express similar relations rather than retrieving questions that are about similar entities. For example, for the query, ‘Who played Natalie Portman in Star Wars?’, we would like to retrieve queries such as ‘Who played Ken Barlow in Coronation St.?’ instead of ‘What sports does Natalie Portman like to play?’. To obtain entity-agnostic representation, we replace the entity spans with a special ‘[mask]’ token in the query, i.e. the original query becomes ‘Who played [mask] in [mask]?’. The entity masking strategy has previously been successfully used in learning entity-independent relational representations (Soares et al., 2019). The similarity score between two queries is given by the inner product between their normalized vector representations (cosine similarity). We pre-compute the representations of queries in the train set. During inference, the most similar query representations are obtained by doing a nearest neighbor search over the representations stored in the case-base.

A practical challenge for KBQA models is to select a subgraph around the entities present in the query. The goal is to ensure that the necessary reasoning patterns and answers are included while producing a graph small enough to fit into GPU memory for gradient-based learning. A naïve strategy to select a subgraph is to consider all edges in 2-3 hops around the query entities. This strategy leads to subgraphs which are independent of the query. Moreover, in large KGs like Freebase (Bollacker et al., 2008), considering the full 2 or 3-hop subgraph often leads to accumulation of millions of edges because of the presence of hub nodes.

We propose a nonparametric approach of query subgraph collection that utilizes the retrieved cases, $\mathrm{kNN}_{q}$, from the last step (Figure 2). For each of the retrieved case, chains of edges (or paths) in the graph that connect the entity in the retrieved query to its answers are collected by doing a depth-first search. (Note, since the retrieved queries are from the training set, we know the answer to them). Next, the sequence of relations are collected from each chain and they are followed starting from the entities of the input query. If a chain of relations do not exist from the query, then they are simply ignored. This process is repeated for each of the retrieved cases. Note that, not all chains collected from the nearest neighbors are meaningful for the query. For example, the last (3-hop) chain collected in Figure 2 is not relevant for answering the query and even though it ended at the answer for the retrieved query, the same is not the case for the input query. Such paths are often referred to as spurious paths or evidence (He et al., 2021). All the edges gathered by following this process form the subgraph for the input query. The underlying idea behind the subgraph selection procedure is simple — since the paths connect queries and answers of similar queries, they should also be relevant for answering the given query.

There is a class of prior work such as Graft-Net (Sun et al., 2018) and PullNet (Sun et al., 2019a) that learn a parametric model to choose a query-specific subgraph. These models employ a retrieval process where a parametric model classifies which edges would be relevant to the query. Being parametric, these models cannot generalize to new type of questions without re-training the model parameters. However, our nonparametric approach will work as long as it has access to similar queries, which it can retrieve on-the-fly. Our subgraph selection procedure is similar to the approach proposed in (Das et al., 2020a). However, Das et al. (2020a) do not use this approach to collect a query-specific subgraph. Rather it uses each of the paths to independently predict the answer to a query. In contrast, we collect all edges in the path to form a subgraph and then reason jointly over the subgraph of the query as well as the subgraph of retrieved cases as detailed in the next sub-section.

This section describes how Cbr-subg reasons across the subgraphs of the given query and the subgraphs of the retrieved cases. We use graph neural networks (GNNs) (Scarselli et al., 2008) to encode the local structure into the node representations of each subgraph. During training, the answer node representations of different subgraphs are made more similar to each other in comparison to other non-answer nodes. Inference reduces to searching for the most similar node in the query subgraph w.r.t the answer nodes in the KNN-subgraphs.

Modern GNNs employ a neighborhood aggregation strategy (message passing) where a node representation is iteratively updated by aggregating representations from its neighbors (Gilmer et al., 2017). Let $\mathcal{G}_{q}=(V_{q},E_{q})$ represent the subgraph for a query $q$ obtained from (§3.2). Let $\bm{X}_{v}$ denote the node feature vectors for each $v\in V_{q}$.

Input node representations. A property of nonparametric models is its ability to represent, reason and grow with new data. Knowledge graphs store facts about the world and as the world evolves, new entities and facts are added to the KG. Models developed for KG reasoning (Bordes et al., 2013; Schlichtkrull et al., 2018; Sun et al., 2019b, inter-alia) learn dense representations of a fixed vocabulary of entities and are hence unable to handle new entities added to the KG. Following our nonparametric design principles, each entity node is represented as a sparse vector of its outogoing edge (relation) types, i.e. $\bm{x}_{v}\in\{0,1\}^{|\mathcal{R}|}$ where $\mathcal{R}$ denotes the set of relation types in the KG. If entity $x_{v}$ has m distinct outgoing edge types, then the dimension corresponding to those types are set to 1. This is an extremely simple and flexible way of representing entities which also expresses the local structural information around each node. Also note that, as new entities are added or new facts are updated about an entity, the sparse representation makes it very easy to represent new entities or update existing embeddings.

Relative distance embedding. Each query-specific subgraph $\mathcal{G}_{q}$ has a few special entities — the entities present in the input query. This is because the reasoning pattern is usually in the immediate subgraph surrounding the query entity. We treat the query entities as ‘center’ entities and append a relative distance embedding to every other node in $\mathcal{G}_{q}$ (Zhang & Chen, 2018; Teru et al., 2020). Specifically, for each node, the representation $\bm{x}_{v}$ is appended with an one-hot distance embedding $\bm{x_{d}}\in\{0,1\}^{|d|}$ where the component corresponding to the shortest distance from the query entity is set to 1. In practice, we consider subgraphs upto 3-hops from the query entities, i.e. $d=4$. For queries with multiple query entities, the minimum distance is considered.

Message passing. Our GNN uses the graph structure and the sparse input node features $\bm{X}_{v}$ to learn intermediate node features capturing the local structure within them. We follow the general message-passing scheme where a node representation is iteratively updated by combining it with aggregation of it’s neighbors’ representation (Xu et al., 2019). In particular, the $l^{th}$ layer of a GNN is,

where, $\bm{a}_{v}^{l}$ denote the aggregated message from the neighbors of node $v$, $\bm{h}_{v}^{l}$ denote the node representation of node $v$ in the $l$-th layer and $\mathcal{N}(v)$ denotes the neighboring nodes of $v$. Since KGs are heterogenous graphs with labelled edges, we adopt the widely used multi-relational R-GCN model model (Schlichtkrull et al., 2018) which defines the aggregate step as: $\bm{a}_{v}^{l}=\sum_{r=1}^{\mathcal{R}}\sum_{s\in\mathcal{N}_{r}(v)}\frac{1}{|\mathcal{N}_{r}(v)|}W_{r}^{l}h_{s}^{l-1}$ and the combine step as $h_{v}^{l}=\text{ReLU}(W_{\textrm{self}}^{l}h_{v}^{l-1}+\bm{a}_{v}^{l})$. For each answer node, we consider the representation obtained from the last layer.

Training. Let $a_{i},a_{j}$ be an answer node in the corresponding query-subgraphs of $q_{i}$ and $q_{j}$ (i.e. $\mathcal{G}_{q_{i}}$, $\mathcal{G}_{q_{j}}$) respectively. Let $\mathrm{sim}(\bm{a}_{i},\bm{a}_{j})=\bm{a}_{i}^{\top}\bm{a}_{j}/\lVert\bm{a}_{i}\rVert\lVert\bm{a}_{j}\rVert$ denote the inner product between $\ell_{2}$ normalized answer representations (i.e. cosine similarity). In general there can be multiple answer nodes for a query. Let $\mathcal{A}_{j}$ denote the set of all answer nodes for query $q_{j}$ in its subgraph $\mathcal{G}_{q_{j}}$. Let $\mathrm{sim}(\bm{a}_{i},\mathcal{A}_{j})=\frac{1}{\lvert\mathcal{A}_{j}\rvert}\sum_{a_{j}\in\mathcal{A}_{j}}\mathrm{sim}(\bm{a}_{i},\bm{a}_{j})$, i.e. $\mathrm{sim}(\bm{a}_{i},\mathcal{A}_{j})$ represents the mean of the scores between $a_{q_{i}}$ and all answer nodes in $\mathcal{G}_{q_{j}}$. We aggregate the similarity score from all retrieved queries $\mathrm{kNN}_{q_{i}}$ for the current query $q_{i}$.

The loss function we use is,

where, $\mathcal{A}_{j}$ denotes the set of all answer nodes in $\mathcal{G}_{q_{j}}$ for a $q_{j}\in\mathrm{kNN}_{q_{i}}$, $x_{i}$ goes over all nodes in query-subgraph $\mathcal{G}_{q_{i}}$ and $\tau$ denotes a temperature parameter. In other words, the loss encourages the answer nodes in $\mathcal{G}_{q_{i}}$ to be scored higher than all other nodes in $\mathcal{G}_{q_{i}}$ w.r.t the answer nodes in the retrieved query subgraphs. This loss is an extension of the the normalized temperature-scaled cross entropy loss (NT-Xent) used in Chen et al. (2020).
Inference. During inference, message passing is run over each of the query-subgraph $\mathcal{G}_{q_{i}}$ and the subgraphs $\mathcal{G}_{q_{j}}$ of its $k$ retrieved queries $q_{j}\in\mathrm{kNN}_{q_{i}}$ to obtain the node representations and the highest scoring node in $\mathcal{G}_{q_{i}}$ w.r.t all the answer nodes in the retrieved query sub-graphs is returned as the answer.

We want to test whether Cbr-subg can answer queries requiring complex reasoning patterns. Note that, the reasoning patterns are always latent to the model, i.e. the model has to answer a given query from the query-subgraph and the retrieved KNN-subgraphs without any knowledge of the structure of the pattern.

To test the model capacity to identify reasoning patterns, we devise a controlled setting in which the model has to infer reasoning patterns of various shapes (Figure 3), inspired by Ren et al. (2020). Note that in their work, the task was to execute the input structured query on an incomplete KB, i.e, the shape of the input patterns are known to the model. In contrast, in our setting, the model has to find the answer node (marked ), which is nestled in each of the structured pattern without the knowledge of the pattern structure. Also note, there can be multiple nodes of same type as the answer type, so the task cannot be completed by solving easier task of determining entity types. Instead the model has to identify specific nodes which are at the end of reasoning patterns (there can be multiple nodes in the graph).

Data Generation Process. We first define a type system with a set of entity types $\mathcal{E}$ and relation types $\mathcal{R}$. The type-system also specifies a set of ‘allowed relation types’ between different pairs of entity types. For example, an ‘employee’ KB relation is defined between an ‘organization’ and ‘people’ entity types. Entities (or nodes $\mathcal{V}$) are then generated uniformly from the set of entity types $\mathcal{E}$. Next, relation edges (uniformly sampled from the allowed types) are joined between a pair of nodes with a probability $p$ following the Erdős-Rényi model (Erdos et al., 1960) of random graph generation. To ensure that models only rely on the graph structure, each graph has a ‘unique’ set of entities and no two graphs share entities. This also effectively tests how much the nonparametric property of Cbr-subg can reason with unseen entities. More details regarding the hyperparameters $\mathcal{E,R,V}$ are included in the appendix B.

Pattern Generation. A pattern is next sampled from the set of shapes shown in Figure 3. The sampled pattern merely suggests the structure of the desired pattern. ‘Grounding’ a pattern shape involves assigning each nodes with an entity present in a generated graph. Similarly each edge of the pattern type is assigned a relation from the set of allowed relation types. After grounding the pattern, we “insert” the pattern in the graph. Since the nodes of the grounded pattern already exists in the graph, inserting a pattern in the graph amounts to adding the edges of the grounded pattern to the graph that already did not exist in it. We also define a ‘pattern type’ — that refers to a pattern whose edges have been assigned relation types but the nodes have not been assigned to specific entities (bottom-left corner in Figure 4). Each pattern type is assigned an identifier and queries with the same pattern type are grouped together.

We generate 1000 graphs in each of the training, validation and test sets. We generate 200 pattern types whose shapes are uniformly sampled from the 5 shapes shown in Figure 3. Therefore, there are around 40 examples of each pattern shape and around 5 examples of each pattern type. This is consistent with real-world setting where a model will encounter a reasoning pattern only very few times during training. For a query with a particular pattern type, other training queries with the same pattern type form its nearest neighbors.

Baselines. Because of the inductive nature of this task where only new entities are seen at test time, most parametric KG reasoning algorithms (Bordes et al., 2013; Yang et al., 2015; Sun et al., 2019b) will not work out of the box. We extend the widely used KG reasoning model — TransE (Bordes et al., 2013) to work in the inductive setting. Specifically, instead of having a fixed vocabulary of entities, the objective function is computed on the dense representation obtained from the output of the GNN layers. This also makes the comparision with Cbr-subg fair since it also operates on the same representations albeit with a contrastive loss. KG completion algorithms also need a query-relation as input. Each pattern type for a query serves as the query relation. Apart from the parametric baseline, we also test a simple nonparametric approach, CBR-path in which path patterns which connect source and target entities are gathered and applied for the current query. Comparing Cbr-subg to CBR-path will help us understand the importance of modeling subgraph patterns rather than simple chains.

Does Cbr-subg have the right inductive bias? The first research question that we try to answer is, if Cbr-subg has the right inductive bias for this task. We test Cbr-subg that has undergone no training, i.e. the parameters of the GNN are randomly initialized. Note the model still takes as input the sparse representation of entities. This experiment will help us answer if the node representations actually capture the local structure around them and whether the answer node can be found by doing a search w.r.t the answer nodes in the KNN-query subgraphs.

Table 1 reports the strict hits@1 on this task, i.e. to score a query correctly, a model has to identify and rank all answer nodes above all other nodes in the graph. The first row of Table 1 shows the results. For comparison, a random performance on this task is $\frac{1}{\lvert\mathcal{V}\rvert}=\frac{1}{120}=0.83\%$. As it is clear from the results, an un-trained Cbr-subg achieves performance much higher than random performance. Its quite high for the simple 2p and 3p patterns. For other patterns that need the more complicated intersection operation, the performance degrades, but is still much higher than random.

Our Results. On training Cbr-subg, the performance of the model drastically improves for each pattern type reaching an average performance of 85.68%. The performance on pattern types which are more complex than chains (ip, pi) etc are worse than chain-type patterns (2p, 3p) suggesting that our task is non-trivial.
On comparison to parametric model. This experiment helps us understand whether a model can learn to memorize and store patterns effectively (for each query relations) when it has seen few examples of that pattern during training. Row 2 of Table 1 shows the performance of GNN + TransE model. We find that the parametric model performs worse than Cbr-subg on all the query types reaching an average performance of 13% point below Cbr-subg. This shows that a semiparametric model with a nonparametric component that retrieves similar queries at inference can make it easier for the model to reason effectively. In practice, we had to train this model for a much longer time than training Cbr-subg.
On comparison to path-based model. From Table 1, we can see that Cbr-subg outperforms CBR-path by more than 14% points suggesting that reasoning over subgraphs is a more powerful approach that reasoning with each paths independently. On the ‘2i’ pattern, CBR-path outperforms Cbr-subg since ‘2i’ can be seen as 2 independent paths intersecting at one node and CBR is able to model that perfectly. However, when the pattern needs composition and intersection and path-traversal, CBR-path struggles and performs much worse.

Next, we test the performance of Cbr-subg on various KBQA benchmarks — MetaQA (Zhang et al., 2018), WebQSP (Yih et al., 2016) and FreebaseQA (Jiang et al., 2019). MetaQA comes with its own KB. For other datasets, the underlying KB is the full Freebase KB containing over 45 million entities (nodes) and 3 billion facts (edges). Please refer to the appendix for details about each dataset (§C).

Our main baselines are the two semiparametric models that provide both a mechanism to gather query subgraphs for a given query and reason over them to find the answer — GraftNet (Sun et al., 2018), PullNet (Sun et al., 2019a). GraftNet uses personalized page rank to determine which edges are relevant for a particular query and PullNet uses a multi-step retriever that at each step, classifies if an edge is relevant to the current representation of the query. For their reasoning model, both works use a graph convolution model and treat the answer prediction as a node classification task. However unlike us, they do not use query-subgraphs of KNN queries. Followup KBQA works (Saxena et al., 2020; He et al., 2021, inter-alia) use the query-specific graphs provided by GraftNet from their open-source code and do not provide a mechanism to gather query-specific subgraphs. However, for completeness, we report and compare with those methods as well.

Table 2 reports the performance on WebQSP and all three partitions of MetaQA. When compared to GraftNet and PullNet, Cbr-subg performs much better on an average on both the datasets. On the more challenging 3-hop subset of MetaQA, Cbr-subg outperforms PullNet by more than 7 points and GraftNet by more than 15 points. This shows that even though these two models use a GNN for reasoning, using information from subgraphs from similar KNN queries leads to much better performance. On WebQSP, we outperform all models except the recently proposed NSM model (He et al., 2021). But as we noted before, NSM operates on the subgraph created by GraftNet and does not provide any particular mechanism to create its own query-specific subgraph (an important contribution of our model). Moreover NSM is a parametric model and will not have some advantages of nonparametric architectures such as ability to handle new entities and reasoning with more data. Table 3 reports the results on the FreebaseQA dataset, which contains real trivia questions obtained from various trivia competitions. Thus the questions can be challenging in nature. We compare with other KBQA models reported in Han et al. (2020). Most of the models are pipelined KBQA systems that rely on relation extraction to map the query into a KB edge. Cbr-subg outperforms all the models by a large margin. We also report the performance on two models that use large LMs and large-scale pre-training. Cbr-subg, which only operates on the KB has a performance very close to the performance of Entity-as-Experts model (Févry et al., 2020). We leave the integration of large LMs into our parametric reasoning component as future work.

How effective is our adaptive subgraph collection strategy? Table 4 reports few average graph statistics for the query-subgraphs collected by our graph-collection strategy. We also compare to GraftNet’s subgraphs. As can be seen, our adaptive graph collection strategy produces much more compact and smaller graphs while increasing recall of answers. We also consistently find that our graph contains relations which is more relevant to the questions than the subgraph produced by GraftNet (§E). Table 5 reports the performance of Cbr-subg when trained and tested on the subgraph obtained from Graftnet and our adaptive procedure, demonstrating the effectiveness of our adaptive subgraph collection method.

Can Cbr-subg reason with more evidence? A desirable property of nonparametric models is to be able to ‘improve’ its prediction as more evidence is made available. We test Cbr-subg by taking a trained model and issuing it an increasing number of nearest neighbor queries. As we see from Figure 5, the performance of Cbr-subg drastically improves as we increase the number of nearest neighbors from 1 to 7 and then increases at a lower rate and converges at around 10 nearest neighbors. This is because, the model has all the required information it needs from its nearest neighbors. However, in the much smaller WebQSP dataset we observe a different behavior (Table 6). This is because of the limited size of the dataset, irrelevant questions starts appearing in the context as we increase the number of KNNs beyond a certain limit.