Seer: Facilitating Structured Reasoning and Explanation via Reinforcement Learning

At reasoning step $t$, we define the state $s_{t}=\{h,P_{t},C_{t}\}$ as a combination of the hypothesis $h$, existing reasoning steps $P_{t}$ and candidate sentences $C_{t}$. $P_{t}$ contains the reasoning steps so far, and $C_{t}$ is the set of sentences that can be selected as premises. Each sentence in $C_{t}$ is either unused facts or intermediate conclusions $I_{t}$ generated by previous steps, i.e., $C_{t}=\{X\cup I_{t}\backslash U_{t}\}$, where $U_{t}$ is the set of used sentences. For the initial state, $s_{1}=\{h,P_{1}=\varnothing,C_{1}=X\}$.

Given the state $s_{t}$, we consider two types of actions $a_{t}\in\mathcal{A}(s_{t})$: (1) "Reason: <premises>": the entailment module is invoked to generate a new intermediate conclusion $i_{t}$ based on the given <premises>. Here, <premises> are selected from $C_{t}$. Then, the state is updated as follows: $P_{t+1}=P_{t}\cup\{\text{\textless premises\textgreater}\rightarrow i_{t}\}$, $U_{t+1}=U_{t}\cup\{\text{\textless premises\textgreater}\}$, and $I_{t+1}=I_{t}\cup\{i_{t}\}$. (2) "End": This action signifies the end of the reasoning process and returns the trajectory $\tau$.

The action type "Reason: <premises>" induces a large action space, since premises can be any combination of sentences from the candidate set $C$. To enumerate the probabilities of all potential actions and then sample an action to execute, previous studies (Liu et al., 2022; Hong et al., 2022) limit combinations to pairwise premises, such that the action space is reduced to $\binom{n}{2}$, where $n$ is the size of the set $C$. However, such a simplification incurs some potential drawbacks. First, as the number of candidate sentences increases, the number of potential actions grows exponentially. This renders them impractical for complex reasoning tasks with limited computational resources. Second, by restricting combinations to pairs only, the interdependencies among multiple premises are ignored, which may limit the effectiveness and richness of the derived conclusions.

To address this issue, we adopt a generative model to represent the policy $\pi$, which can directly sample from the action space $\mathcal{A}(s_{t})$. Using the generative model essentially expands the action space where the combinations of premises can be arbitrary. This enables the policy to extensively explore better actions during RL training, not limited to paired premises. Further, to speed up RL training, we first generate the top-$k$ actions using policy $\pi$:

where the input is a linearized state $s_{t}$ (i.e., the concatenation of $h$, $P_{t}$, and $C_{t}$). Then, we proceed with re-normalization to form an appropriate probability distribution over the top-$k$ actions, and sample from it to select the action $a_{t}$ to be performed in the current reasoning step, that is,

If the action $a_{t}$ is "Reason: <premises>", we invoke the entailment module to derive the intermediate conclusion to obtain the next state. The entailment module is also a generative model with its input being <premises>. Following Hong et al. (2022); Liu et al. (2022), we fine-tune the entailment model in a supervised manner and freeze the parameters during the reinforcement learning process, as shown in Figure 2.

To evaluate the correctness of the entailment tree, Dalvi et al. (2021) proposed an alignment algorithm based on Jaccard similarity to align each intermediate node of the predicted tree $T_{\text{pred}}$ with $T_{\text{gold}}$. However, different from the fully supervised learning methods, we observe that during the RL process, the policy explores different actions to identify the optimal reasoning process, inevitably attempting some redundant steps that do not contribute to reaching the final hypothesis. Existing RL-based work (Liu et al., 2022) simply treats redundant steps with the same penalty as erroneous steps. This simplification may negatively affect the learning process which discourages necessary exploration in the action space. Furthermore, it lacks detailed feedback to guide the policy toward optimal policy, as it fails to differentiate between innocuous actions (redundant steps) and incorrect actions (erroneous steps).

To this end, we propose a fine-grained reward function that assigns different reward values for correct steps, erroneous steps, and redundant steps, as shown in Equation 4. For a trajectory $\tau$, we assume that the last intermediate conclusion is our predicted hypothesis since the policy deems it should End here. Then, we backtrack to construct the predicted entailment tree $T_{\text{pred}}$ (see Appendix C.6 for more details). Note that there might be some steps not participating in $T_{\text{pred}}$, which are regarded as redundant steps. Then, as illustrated in Figure 5, we consider steps that perfectly match via the alignment algorithm (Dalvi et al., 2021) as correct steps and regard others as erroneous steps.

To enhance training stability, we introduce the critic to estimate the state-value function $V(s_{t})$. The input of $V(s_{t})$ is a linearized state, and its output is a scalar representing the return (i.e., cumulative reward) when starting from state $s_{t}$. In the simplest case, the return is the chained sum of the rewards. Accordingly, one-step temporal difference (TD) (Sutton, 1988) is often used to estimate $V(s_{t})$, which is updated by the TD-target:

where $\gamma$ is the discount factor. However, in structured reasoning, reasoning steps typically adhere to inherent tree (Dalvi et al., 2021) or graph (Ribeiro et al., 2023) structures, with the chained structure being merely a special case. Thus, Equation 5 just describes the chained multi-step reasoning, which may not effectively capture the intricate logical dependencies between steps in structured reasoning.

Therefore, we propose the structure-based return, where the TD-target is expressed in a more general formulation:

where $\mathcal{P}(s_{t})$ represents the parent node of state $s_{t}$ in the entailment tree $T_{\text{pred}}$ or reasoning graph. When $s_{t}\notin T_{\text{pred}}$, $\mathcal{P}(s_{t})=s_{t+1}$. It can be seen that our structure-based return (Equation 6) adapts to structured reasoning involving chained, tree-based and graph-based structured scenarios. Especially, entailment tree is a special case of the reasoning graph, in which each state typically has only one parent node, and thus Equation 6 degenerates into $\hat{G}_{t}=r_{t}+\gamma V(\mathcal{P}(s_{t}))$. Furthermore, as shown in Figure 6 (Appendix E), for equivalent trajectories $s_{1}\rightarrow s_{2}\rightarrow s_{3}\rightarrow s_{4}$ and $s_{2}\rightarrow s_{1}\rightarrow s_{3}\rightarrow s_{4}$, previous method (Liu et al., 2022) would assign different returns for state $s_{1}$ and $s_{2}$, even though they represent the same tree in the end. Conversely, our method, by precisely delineating the intricate interdependencies between reasoning steps, consistently allocates the same return to any equivalent trajectories, thereby enhancing both stability and effectiveness.

Incorporating the supervised warm-up strategy before RL offers a relatively stable initial policy, which facilitates faster adaptation to the environment, particularly for complex reasoning tasks (Ramamurthy et al., 2023; Wu et al., 2023). Therefore, we convert the structured reasoning into single-step supervised data to warm up the policy as follows:

where $y$ is the golden action at $s_{t}$.

We conduct experiments on EntailmentBank (Dalvi et al., 2021), the first dataset that supports structured explanation with entailment trees. Following (Hong et al., 2023), we also conduct experiments on EntailmentBankQA (Tafjord et al., 2022), whose objective is to reach the answer based on the entailment tree.

We conduct experiments on the STREET benchmark (Ribeiro et al., 2023) to assess the performance of graph-structured reasoning. Please refer to Appendix B for more details about the dataset statistics.

As shown in Table 2, our Seer outperforms all baseline methods on the most strict metric, "Overall AllCorrect", across all three tasks. Specifically, our method achieves an absolute improvement of 1.7%/1.4%/1.0% in Task 1/2/3 compared to the strongest baseline. The steps dimension, i.e., premises selection, is the core of EntailmentBank³³3A comprehensive error analysis is detailed in Appendix G., contributing to enhancing the accuracy of both leaves and intermediates dimensions, thereby improving the overall AllCorrect metric. (1) Compared to SOTA supervised methods, such as NLProofs and FAME, our method exhibits significant advantages in the steps dimension. This demonstrates that focusing solely on isolated single-step reasoning through supervised learning may yield suboptimal solutions in intricate structured reasoning tasks, even though employing advanced planning algorithms, such as Monte-Carlo planning in FAME. (2) Compared to the SOTA RL-based method, our method outperforms RLET by 5.8%/9.0%/6.0% in Task 1/2/3. Our method employs a generative model as the policy to circumvent the issue of enumerating actions, facilitating the policy’s understanding of structured reasoning tasks (generating potential actions by itself). Moreover, our proposed structure-based return more effectively captures the tree-structured logical dependencies between steps and can assign stable returns for equivalent trajectories, which significantly improves reasoning abilities. Subsequent ablation studies will further demonstrate this. (3) Compared to GPT-4 with CoT, ToT, and ReAct, our method achieves an absolute improvement of 1.9% in Task 3. Although GPT-4 exhibits outstanding reasoning capabilities surpassing many other baselines, its performance relies on a vast number of parameters. Details about the prompts of GPT-4 can be found in Appendix F.

Following Creswell and Shanahan (2022), we introduce the halter module to generate answers based on $T_{\text{pred}}$ and substitute hypothesis with question and option during the reasoning process. As illustrated in Table 3, our method surpasses FAME by an absolute margin of 1.2%/7.4% in Task 1/2. While both FAME and SI are supervised methods, FAME significantly outperforms SI by enhancing the model’s reasoning and exploration capabilities through Monte-Carlo planning. However, our method enhances the structured reasoning capabilities of the policy rather than focusing solely on single-step reasoning, which can significantly improve the quality of the entailment tree to aid in answering, especially in complex reasoning environments.

As shown in Table 4, compared to GPT-4, our method has achieved absolute improvements of 4.8%/3.4%/4.1%/5.2% across various datasets, although the Reasoning Graph Accuracy is a very strict metric (Ribeiro et al., 2023). While GPT-4 excels at answering questions (far surpassing other methods), its parameter is thousands of times greater than other methods. Moreover, during the reasoning process, GPT-4 is prone to hallucinations (Rawte et al., 2023), resulting in poor performance in structured reasoning, particularly evident in the "Reasoning Graph Accuracy" metric. Since SCONE contains sufficient data as well as similar QA and reasoning patterns, we observe that the STREET method would outperform GPT-4 on SCONE. However, by obtaining high-quality reasoning graphs, our method achieves absolute improvements of 2.8%/11.0%/8.9%/5.5% compared to the STREET method, significantly improving answer accuracy and trustworthiness. In reasoning graphs, a state may have multiple parent nodes. Our structure-based return (Equation 6) still precisely describes the cumulative reward for each state, thereby facilitating reasoning performance in graph-structured reasoning.