R5: Rule Discovery with Reinforced and Recurrent Relational Reasoning

Path sampling. To predict the relation between two entities in graphs, we preprocess the data by sampling paths that connect the two entities from the graph. When the total number of paths is small, we enumerate all the paths. Otherwise, we randomly sample $L$ paths at maximum. Each path purely consists of relations. For instance, in Figure 2, given the input graph, we can get 4 paths between query nodes X and Y, including $r_{1}$-$r_{2}$-$r_{5}$-$r_{3}$-$r_{7}$-$r_{6}$ and so on.

Reasoning as sequential decision-making. Our method solves the problem of relational reasoning in terms of sequential decision-making. Specifically, we train a reasoning agent based on a policy value network combined with MCTS (Silver et al., 2017a) to recurrently reason over the extracted relation paths. The policy value network $f_{\theta}(s)$ is a neural network with parameters $\theta$. It takes the current state $s$ as input, and outputs the action probability distribution $\bm{\rho}$ and a state value $\nu$. MCTS utilizes the policy network $f_{\theta}(s)$ to guide its simulations, and outputs a vector $\bm{\pi}$ representing the improved search probabilities of the available actions. The policy network $(\bm{\rho},\nu)=f_{\theta}(s)$ is then trained to minimize the error between the predicted state value $\nu$ and the reward $z$ received after an episode, as well as maximize the similarity between two probability distributions $\bm{\rho}$ and $\bm{\pi}$. The loss function $l$ is $l=(z-\nu)^{2}-\bm{\pi}^{\intercal}\log{\bm{\rho}}+a\left\|\theta\right\|^{2}$, where $a$ is a hyper-parameter.

Action. At each step of an episode, the MCTS outputs an action $(a_{B,0},a_{B,1})$, which is a relation pair denoted as $a_{B}$. Furthermore, by looking up the dynamic rule memory, it obtains $a:a_{H}\leftarrow(a_{B,0},a_{B,1})$ that contains a head relation $a_{H}$ to be deducted to, which means that the relation pair $(a_{B,0},a_{B,1})$ in the path will be substitute with $a_{H}$. For example, in Figure 2 at step $1$, MCTS outputs an action $a:r_{3}\leftarrow(r_{1},r_{2})$, and the path $r_{1}$-$r_{2}$-$r_{5}$-$r_{3}$-$r_{7}$-$r_{6}$ is transformed into $r_{3}$-$r_{5}$-$r_{3}$-$r_{7}$-$r_{6}$. By recurrently applying different actions to a path between the query nodes, it will be transformed into a single relation at the end of an episode.

State. Instead of encoding the walking paths between the query nodes as other RL-based rule induction methods (Shen et al., 2018; Das et al., 2017; Xiong et al., 2017), we make use of the features of the possible relations pairs at the current state. As shown in Figure 2, we define the state $s\in\mathbb{R}^{(m+n)\times(m+n)\times k}$ as the representation of all the possible relations pairs among the current paths, where $m$ is the number of observed relations, $n$ is the number of invented relations we assumed, and $k$ is the dimension of features of each relation pair. As previously discussed, a relation pair may be deducted to an invented relation $r_{*}\in N$. Besides, $r_{*}$ can be paired with another relation $r$, and maybe further deducted to another invented relation $r^{\prime}_{*}\in N$. Our state space design enables us to represent such complicated rules with the invented relations that serve as intermediate predicates, which is essential to model complex relational data. $n$ invented relations are allowed in our design, where $n$ depends on the complexity of input. An example is presented in Appendix A.4.

Even if sometimes the input data can be explained by rules without invented relation, the invented relations in our state design can actually help to speed up model training. For example, when we observe that a relation pair $r_{1}$ and $r_{2}$ shows up frequently in the training data, we can sample an invented relation $r_{*}$ as the head, and form a candidate rule $r_{*}\leftarrow(r_{1},r_{2})$. In this way, our model learns to merge $r_{1}$ and $r_{2}$ quickly without the need to figure out what exactly $r_{*}$ is. The value of $r_{*}$ will be inferred by our model with Backtrack Rewriting mechanism, which we will introduce in more detail later. Without these invented relations, a reasoning agent has to infer what exactly $r_{*}$ is at an early state, otherwise it cannot proceed to merge the relation pairs in a path correctly. Thus, the invented relations actually serve as a buffering mechanism to allow an agent to learn “what to merge” first, rather than inferring the exact rule at an early stage.

The features we utilized to represent each relation pair can be summarized into two groups. The first group includes the general and statistical features of the pair, such as the number of occurrences, the index of the most viewed position among the paths, the number of occurrences at the most viewed position, and the types of the two relations consist the pair. The other group includes the rules-related features, such as whether the relation pair is in the obtained rules, whether the relation pair consists of two known relations, whether the rule head is a known relation if the pair is in the rules memory, and the score of the corresponding rule if the pair is in the rules memory.

Reward. During an episode, actions are applied to a relation path recurrently, until the path is deducted to a single relation $r$, and a reward $z_{T}\in\{-1,0,+1\}$ will be assigned to all the states in this episode. If $r$ is an known relation, but is not the target relation, $z_{T}=-1$. If $r$ is an invented relation, $z_{T}=0$. If $r$ is the target relation, $z_{T}=1$.

The above recurrent reasoning agent enables us to predict the relation between two queried nodes more efficiently than only using MCTS. To extract rules from observed data, we further design a dynamic rule memory module which stores and updates candidate rules and interact with the reasoning agent during training. Our memory design can be interpreted as two hash tables with the same keys: one hash table $D_{rl}$ in the form of $(r_{B0},r_{B1}):r_{h}$ is used to memorize the candidate rules, and another hash table $D_{rls}$ in the form of $(r_{B0},r_{B1}):\texttt{score}$ to track the rule scores. Note that in Figure 2, we union the two tables as they share the same keys. A newly added rule will have an initial score of $0$. In Figure 2, a rule $(r_{3})\leftarrow(r_{1},r_{2})$ was stored in the hash tables, where the rule body $(r_{1},r_{2})$ is the key, and the rule head $r_{3}$ and score $4.303$ are the values in $D_{rl}$ and $D_{rls}$, respectively.

We utilize a buffer $B_{\tt{unkn}}$ to store the $n$ intermediate invented relations as described in Sec. 2.1. When an invented relation type $r\in N$ is used to complete a rule, it will be removed from $B_{\tt{unkn}}$. If $r$ is freed from the rule memory $D_{rl}$, then it will be added to $B_{\tt{unkn}}$ again.

Recall that at each time step the MCTS will return an action body $a_{B}$. In order to complete the action, we get the action head $a_{H}$ by

Then, if $a_{B}\notin D_{rl}$, we update the memory $D_{rl}$ with $D_{rl}[a_{B}]=a_{H}$, and compute the score $c_{a}$ by

where $v_{0}>v_{1}>0>v_{2}>v_{3}>v_{4}$. We can see that an action consisting of three known relations will receive the largest positive score, and an action with an invented relation in the head and at least an invented relation in the body will receive the smallest negative score. This way we penalize the usage of invented relations to prevent overfitting.

After extracted candidate rules, at the end of each training episode, we further utilize Backtrack Rewriting to consolidate all the candidate rules and replace invented relations in a rule with observed relations if we found they are equivalent. Algorithm 1 describes our Backtrack Rewriting mechanism for updating the rule memory $D_{rl}$. Specifically, assume the action sequence we have taken in an episode is $\{a_{0},a_{1},...a_{t},...a_{T}\}$, and the head relation of the last action $a_{T}$ is $r:={a_{T}}_{H}=D_{rl}[{a_{T}}_{B}]$. As $y$ is the target relation, we can backtrack $r$ in $D_{rl}$, and replace all its occurrences with $y$. Before updating $D_{rl}[{a_{T}}_{B}]$, we update the corresponding entries in $D_{rls}$ of the actions $\{a_{0},a_{1},...a_{t},...a_{T}\}$ in the episode by

where $r:=D_{rl}[{a_{T}}_{B}]$, and ${v_{T}}_{\tt{neg}}<0<{v_{T}}_{\tt{pos}}$ are the episode end scores. Next, we decay the scores $D_{rls}[a_{B}]$ of all the rules $a_{H}\leftarrow a_{B}$ over episodes by:

where $\epsilon$ is a small positive decay factor. Thus, the score of a rule that is not accessed over training episodes will decrease.

Finally, when a rule score is less than a negative threshold $\sigma$, the corresponding rule will be omitted from the rule memory. We recursively check the rule memory to remove all the candidate rules that contain the omitted rule’s head ${a_{H}}_{bad}\in N$. Thus, the bad rules will be gradually removed from the rule memory. $B_{\tt{unkn}}$ and $D_{rls}$ will be updated accordingly. Besides, when the invented relations buffer $B_{\tt{unkn}}$ is empty, we will look up the rules scores hash table $D_{rls}$ to find the rule ${a_{H}}_{bad}\leftarrow{a_{B}}_{bad}$ that has the smallest rule score, where ${a_{H}}_{bad}\in N$. Then all the corresponding rules in $D_{rl}$ that contain ${a_{H}}_{bad}$ will be removed, and ${a_{H}}_{bad}$ will be added back to $B_{\tt{unkn}}$.

We first evaluate R5 on the CLUTRR dataset and demonstrate its Systematicity. In other words, we test whether R5 can perform reasoning over graphs with more hops than the training data.

Dataset and Experimental Setup. CLUTRR (Sinha et al., 2019) is a dataset for inductive reasoning over family relations. Paragraphs describing family relationships are encoded as a large set of graphs, where family members are modeled as nodes, and the relations in between are modeled as edges. The goal is to infer the relation between two family members, which is not explicitly mentioned in the paragraph. The train set contains graphs with up to 3 or 4 edges, and the test set contains graphs with up to 10 edges. The train and test splits do not share the same set of entities.

As baselines, we evaluate several embedding-based models (Veličković et al., 2017; Kipf & Welling, 2017; Schuster & Paliwal, 1997; Hochreiter & Schmidhuber, 1997; Chung et al., 2014; Kim, 2014; Kim et al., 2015; Vaswani et al., 2017) and several neuro-symbolic models that consists the rule induction baselines (Minervini et al., 2020a; b). The results of the baselines are taken from Minervini et al. (2020b). Due to limited space, for the baselines other than CTPs and GNTPs, we only present the results fine-tuned on short stories. The comparisons with these baselines fine-tuned on long stories are provided in the Appendix C, and none of them can beat R5.

Results. Table 11, 11 present the prediction accuracy on two datasets pulished by Sinha et al. (2019). We call the two datasets CLUTRR_k∈{2,3} and CLUTRR_k∈{2,3,4}, as they train on graphs with $\{2,3\}$ edges while test on graphs with $\{4,...,10\}$ edges, and train on graphs with $\{2,3,4\}$ edges while test on graphs with $\{5,...,10\}$ edges, respectively. As shown in both tables, R5 either outperforms all the baselines or reach the best results in both metrics. In particular, R5 has better generalization capability than CTPs. For both datasets, the prediction accuracy of CTPs goes down when evaluated on longer stories. In contrast, R5 still predicts successfully on longer stories without significant performance degradation. This demonstrates the strong compositional generalization ability of R5, in terms of systematicity, productivity, substitutivity, and localism (Hupkes et al., 2020).

In Section 3.1, we test model performance with no bad cases in data. A bad case is a logically wrong example. For example, the mother of a sister is a father is a bad case. Apart from the compositional generalization ability, it is also important to assess how effectively a model can discover all underlying rules. The existing relational reasoning works only show that they are able to improve relation prediction with some learned rules, without an evaluation on whether the extracted rules are exhaustive. GraphLog, the dataset we use in this subsection, provides the ground rules used to create each logical world, so we evaluate this ability by the recall rate of ground truth rules.

Dataset and Experimental Setup. Graphlog (Sinha et al., 2020) is a benchmark suite designed to evaluate systemeticity. It contains logical worlds where each world contains graphs that are created using a different set of ground rules. Similar to CLUTRR, the goal is to infer the relation between a queried node pair. However, GraphLog is much more challenging than CLUTRR. Firstly, it contains more bad examples than CLUTRR does. Secondly, the graphs in both train sets and test sets of Graphlog contain more edges; it is normal that there are more than 10 edges in a graph. Thirdly, the underlying compositional rules may not be shared between a train set and its corresponding test set. The above facts make GraphLog a more challenging dataset for rule induction.

In the evaluations on GraphLog, we consider E-GAT (Sinha et al., 2020) and R-GCN (Schlichtkrull et al., 2018) as the embedding-based baselines, and CTPs (Minervini et al., 2020b) as the rule induction baseline. For E-GAT and R-GCN, we use the results in (Sinha et al., 2020). For CTPs, we use the implementation publicly provided by the author with the best configurations. In addition, we use the core generator provided by (Sinha et al., 2020), which is used to produce the GraphLog datasets, to generate three new worlds: Re 0, Re 1, Re 2. The new worlds have less training data and more distinct relations sequences. We aim to test how the rule induction models perform when only a small amount of training samples are given. We divide the 57 worlds in Graphlog into two groups in terms of the average resolution length (ARL), where the resolution length is the number of nodes traversed in the correct path between the queried nodes. Figure 1a,b are instances of resolution lengths of 2, and Figure 1c is an instance of resolution length of 3. We then randomly pick 5 worlds with an ARL less than 4, and 5 other worlds with an ARL greater than 4 to conduct experiments on.

Results. Table 4 shows the results on 10 selected worlds (datasets) published by (Sinha et al., 2020). When the ARL is larger, the dataset becomes noisier and contains more bad cases. In terms of relation prediction accuracy, R5 significantly outperforms both the embedding-based and rule induction baselines over all 10 datasets. In terms of the ground rule recall rate, R5 also consistently outperforms CTP, and even reaches full recall rate on some datasets.

At a closer inspection, the two GNN models perform poorly for ARL $<4$. A dataset with a smaller ARL indicates the graphs are very simple, and models that represent the graph structure based on message-passing have not enough information to learn from. As for datasets with ARL $>4$, the graph structures become more complex, the GNNs are able to aggregate more information to each node from the neighbors. Thus, the performance of GNNs improves on these datasets. However, CTP performs worse than GNNs in this case, due to two reasons. On one hand, the increasing amount of bad cases influences the rule recall rate, making CTP less confident on some correct rules. On the other hand, it is hard for CTP to generalize to complex data.

Table 5 presents the results for the three new datasets created that have fewer training samples. In particular, resolution lengths of graphs in Re 0 can be up to 10, while resolution lengths of graphs in Re 1 can be up to 15. Re 2 requires the models to train on graphs with resolution lengths of $\{3,...,5\}$, and to test on graphs with resolution lengths of $\{2,...,7\}$. As shown in the table, R5 achieves better results than CTP when the amount of training facts is very small. And we can observe from Re 0 and Re 1 that as the resolution length becomes longer, CTP has a sharp performance drop on the prediction accuracy, whereas R5 maintains relatively high accuracy and full ground rule recall rate. In the train set of Re 2, there is no simple data. Simple data refers to a sample whose resolution length is 2, and the ground rules can be naturally collected from such data. However, in Re 2, as we train on samples of lengths $\{3,...,5\}$ and evaluate on samples of lengths $\{2,...,7\}$, the models have to infer the ground rules. As is shown, R5 can still extract most of the ground rules while CTP fails.

Policy Value Network. As stated earlier, the policy value network is trained to sequentially decide which edge pair we should select to do a deduction. Without training this network, only MCTS will help us to make the decisions, whose action distribution is given by the numbers of visits at all possible nodes. From Table 6 we acquire that although we are still able to recall the ground rules with rules extraction, the prediction accuracy drastically decreases since the paths are not deducted in the correct order.

Pre-allocated Invented Relations. We pre-allocate $n$ invented relations to allow a relation pair to deduct to an intermediate predicate when it still is unclear whether the relation pair can give us a correct rule. And the intermediate predicate will be tracked during the training until it is identified as a known relation. Refer to Table 6, when we only allow a single invented relation, the prediction accuracy will drop, which indicates that the extra hidden relations improve the model’s capability of learning complicated graphs. And in Re 2, where the training samples’ resolution lengths are greater than 2, the results catastrophically decrease. We evaluate the prediction accuracy and rules recall each training epoch. As shown in Figure 3, the model loses the induction ability at the early stage with only one intermediate predicate allowed. Once the first ground rule is extracted, more will be gradually learned, and finally the R5 with a single hidden predicate reaches the same rules recall rate as R5 with 50 hidden predicates. This figure proves that a number of pre-allocated invented relations can not only help to learn complicated graphs, but also speed up the convergence.