Answering Numerical Reasoning Questions in Table-Text Hybrid Contents with Graph-based Encoder and Tree-based Decoder

The entire input heterogeneous graph $G=(V,R)$ consists of all types of nodes, that is $V=Q\cup T\cup P$ with the number of nodes $|V|=|Q|+|T|+\sum_{s_{i}\in P}|s_{i}|$, where $|T|$ and $|s_{i}|$ are the number of table cells and paragraph sentences respectively. For relations, $R=R_{I}\cup R_{C}$. $R_{I}$ denotes intra-relation and $R_{C}$ denotes inter-relation. Details of the node types and relation types are described in the Appendix A. Finally, we model the question, table and text as a graph.

The graph input module aims to initialize embeddings for both nodes and edges. For edges, the edge features are directly initialized from a parameter matrix. For nodes, we can obtain their representations from a pre-trained language model (PLM) such as RoBERTa (Liu et al., 2019b). We flatten all question words, table cells and paragraph words into a sequence $\mathrm{[CLS]}q_{1}q_{2}...q_{|Q|}\mathrm{[SEP]}t_{1}t_{2}...t_{|T|}\mathrm{[SEP]}s_{10}s_{11}...\\ s_{|P|0}s_{|P|1}...\mathrm{[SEP]}$. $s_{i0}$ is the special sentence token of the sentence $s_{i}$. Since each word $e$ of the sequence is tokenized into sub-words, we use Multi-granularity type aware pooling to get the node representation $\mathrm{x}$. Details in the Appendix B.

This module aggregates information about the nodes and edges of the heterogeneous graph. It is a stack of $L$ relational graph attention network (RGAT) layers. In each layer $l$, the RGAT (relational graph attention transformers) (Wang et al., 2020) models the graph $G$ and computes the output representation $\mathbf{Z}$ by:

where matrices $\mathrm{W}_{q}$,$\mathrm{W}_{k}$,$\mathrm{W}_{v}$ are trainable parameters in self-attention, and $\mathcal{N}_{i}$ is the receptive field of node $v_{i}$. The output matrices of the final layer $L$ are the desired outputs of the encoder: $\mathrm{Z}=\mathrm{Z}^{L}$.

Inspired by the goal-driven tree structure (GTS) (Xie and Sun, 2019) for solving math word problem, we propose a novel tree-based decoder to construct the calculation expressions for solving text-table numerical reasoning problem. As such, the specialized tree decoder generates an equation following the pre-order traversal ordering. The model takes in question $Q$, table $T$, paragraph $P$ and generates a expression tree $\mathrm{T_{e}}$. Let $V_{num}$ denote numeric values in $T$ and $P$. Generally, $V_{con}$ denotes constant values $V_{con}=\left\{1,\mathrm{AVG}\right\}$, $\mathrm{AVG}$ means to average the sum of the previous numbers. $V_{op}$ denotes mathematical operators $V^{op}=\left\{+,-,\times,{\div}\right\}$.

The tree generation process is designed as a preorder tree traversal (root-left-right). For node $y$ in target $\mathrm{T_{e}}$, $y\in V^{num}\cup V^{con}\cup V^{op}$. We set $V_{num}$ and $V_{con}$ to be the leaf nodes and $V_{op}$ serve as the internal nodes and must have two child nodes.

The tree structured decoder uses the final graph layer representations $\mathrm{z}_{i}$ as input and generates the target expression in $\mathrm{t}$ time steps. At each time step $\mathrm{t}$, let $\mathrm{s_{t}}$ denote the decoding hidden state, $\mathrm{c_{t}}$ denotes the hybrid context state, $\mathrm{g_{t}}$ denotes the generated expressions tree state.

The decoder is a bi-directional GRU (Cho et al., 2014), which updates its states at time step $\mathrm{t}+1$ as follows:

where $\mathrm{E}(y_{t})$ is the embedding of token $\mathrm{y_{t}}$:

$\mathbf{M}_{op}$ and $\mathbf{M}_{con}$ are two trainable embeddings for operators and constants, respectively. For a numeric value in $V^{num}$, its token embedding takes the corresponding hidden state $h_{loc(y_{t},T,P)}^{i}$, where $loc(y_{t},T,P)$ is the index position of $y$ in table $T$ or paragraph $P$ Hong et al. (2021).

Inspired by math word problem solving (Wu et al., 2021), the generated expression tree state $\mathrm{g_{t}}$ is calculated as follows:

where $\sigma$ is a sigmoid function and $\mathrm{W_{g}}$ is a weight matrix. For each generated node, $\mathrm{g_{g,p}}$, $\mathrm{g_{g,l}}$, $\mathrm{g_{g,r}}$ represent the expression tree state of the parent node, left child node, and right child node of the current node, respectively.

The hybrid context state $\mathrm{c}_{\mathrm{t}}$ is computed via attention mechanism as follows:

where $\mathrm{W_{h}}$, $\mathrm{W_{s}}$ are weight matrices. $\alpha_{ti}$ is the attention distribution on the node representations $\mathbf{z_{i}}$.

Lastly, the decoder can generate a word from a given vocabulary $V_{op}\cup V_{con}$. It can also generate a number symbol from $V_{num}$, which is copied a number from the table $T$ or paragraph $P$. The final distribution is the combination of the generated probability and copy probability:

Here, $f(\cdot)$ is a perception layer. $p_{c}$ is the probability that the current word is a number copied from the table or paragraphs.

In addition, there are two separate tasks in the decoding section: operator prediction and scale prediction. For arithmetic questions, a right prediction of a numerical answer should include the right number and the correct scale. The scale in the dataset may be None, Thousand, Million, Billion, and Percent generally. We focus on arithmetic questions for operator prediction, but there are still non-arithmetic questions (span extraction question) in the dataset. So we classify whether the question is arithmetic or not before decoding. For the extraction questions, as shown in Figure 3, we also model them as trees, as described in Appendix C.

To predict the right aggregation operator and scale, two multi-class classifiers are developed. In particular, we take the vector $\mathrm{\left\langle CLS\right\rangle}$ to compute the probability:

where $h_{Q}$,$h_{T}$ and $h_{P}$ are the representations of the question, the table and the paragraphs , respectively, which are obtained by applying an average pooling over the representations of their corresponding tokens. “;” denotes concatenation, and FFN denotes a two-layer feed-forward network with the GELU activation.

To optimize RegHNT, the overall loss is the sum of the loss of the above tasks:

$\mathcal{L}_{tree}$ is the loss function of training the tree-decoder, and we use the cross-entropy loss. $\mathcal{L}_{op}$ and $\mathcal{L}_{scale}$ are the loss functions for operator prediction and scale prediction, respectively, where NLL(·) is the negative log-likelihood loss. $\mathrm{G^{op}}$ comes from the supporting evidence, which is extracted from the annotated answer and derivation. $\mathrm{G^{scale}}$ uses the annotated scale of the answer. We add up the three loss functions as the total loss function.

TAT-QA (Zhu et al., 2021) is a large-scale, hybrid QA dataset which contains numerical reasoning and span extraction questions. And the contents of TAT-QA include both tabular and textual data from real financial reports. It contains a total of 2,757 hybrid contexts and 16,552 corresponding question-answer pairs. The detailed statistics are shown in Appendix D. The original dataset contains four types of questions: Span, Multi-Span, Count, Arithmetic. But in our setup, there are two types of questions: Span Extraction and Arithmetic. And our work mainly focuses on answering the Arithmetic questions.

For evaluation, we adopt the Exact Match (EM) and numeracy-focused F1 score (Dua et al., 2019) to measure the performance of different QA models. All of which are computed using the official evaluation script³³3https://github.com/NExTplusplus/tat-qa. We submit our model to the organizer of the challenge for evaluation. The evaluation detail can be found on the original paper (Zhu et al., 2021).

Implementations. Our model is implemented with PyTorch (Paszke et al., 2019), and the graphs are constructed with the library DGL (Wang et al., 2019b). In the graph input module, we use pre-trained language models (PLMs) RoBERTa (Liu et al., 2019b) to obtain the initial representations. During evaluation, we adopt beam search decoding with beam size 3.
Hyper-parameters. In the encoder, the number of GNN layers $L$ is 8, and the number of heads in multi-head attention is 8. For PLMs, we use learning rate 1e-5 and weight decay rate 0.01. For other model modules, we use a larger learning rate 1e-4, and a weight decay rate 5e-5. In the decoder, The recurrent dropout rate (Gal and Ghahramani, 2016) is 0.2 for GRU. The number of heads in multi-head attention is 8 and the dropout rate of features is set to 0.1 in both the encoder and decoder. Throughout the experiments, we use AdamW (Loshchilov and Hutter, 2018) optimizer with a linear warmup scheduler. The warmup ratio of the total training steps is 0.06. The batch size is 48, and the training epoch is 100. The training process may take around 2 days using a single NVIDIA GeForce RTX 3090.
Baselines. We compare with the standard TAGOP (Zhu et al., 2021), which first applies sequence tagging to extract relevant cells from the table and text spans from the paragraphs. We also compare with other advanced models, which can be found on the TAT-QA challenge leaderboard⁴⁴4https://nextplusplus.github.io/TAT-QA. There are no linked papers to the submissions as yet. We compare our model’s performance on the test split with all of them.

The main results on the test set are provided in Table 1. Our model achieves the state-of-the-art results in the publicly available TAT-QA benchmark and achieves 20$\%$ higher on both EM and $\mathrm{F_{1}}$ compared with the original baseline (TAGOP), which shows that our model can answer more questions with higher accuracy.

The detailed results on the test set are provided in Table 2. For almost all types of questions, the accuracy of RegHNT prediction has been improved. Thanks to the tree decoder for arithmetic questions, the model’s accuracy on this type of questions has been greatly improved, with an overall improvement of almost 30$\%$. For span extraction questions (Span, Spans, Counting), we observe a improvement in the performance of the model as well. From the perspective of answer sources, compared with the original baseline (TAGOP), we focus on "table-text" type questions, both "arithmetic" and "span extraction" type show a performance improvement of about 20$\%$. It demonstrates that our approach works very well in solving table-text hybrid questions.

This paper focuses on the numerical reasoning questions. To verify our model more precisely, we divided the questions into three categories based on the complexity of the derivation, as shown in Table 3. Simple arithmetic has only one operator. Complex arithmetic has multiple operators, usually used to calculate the average and the rate of change. Undefined arithmetic is arithmetic for which no template is defined in TAGOP. The results show that our model significantly improves both simple and complex arithmetic. In particular, our model can solve undefined arithmetic of TAGOP, which offers flexibility and generalizability.

Effect of Tree Decoder. Although our work focuses on arithmetic questions, to unify the whole model into a graph-tree framework, we transform the span extraction type question into a tree generation problem as well, as mentioned in Section 3.5. We conducted two experiments using the sequence tagging method in TAGOP (Zhu et al., 2021) instead of generating expression trees. As shown in the last two rows of Table 4, one is to use sequence tagging for all questions, and the other is to use sequence tagging only for span extraction questions. When we change the decoder only for extraction questions, the $\mathrm{F_{1}}$ drops only 0.7$\%$, but when we change the decoder for all questions, the $\mathrm{F_{1}}$ drops about 11.8$\%$. It shows that the tree decoder is not only tremendously helpful in solving arithmetic questions but also provides a slight improvement in solving span extraction questions.
Effect of Graph Encoder. As shown in the first four rows of Table 4, we show the effectiveness of the proposed graph encoder. Removing the intra-relations reduces the $\mathrm{F_{1}}$ value by 0.9$\%$ and reduces the EM value by 0.8$\%$. Removing the inter-relations reduces $\mathrm{F_{1}}$ value by 1.5$\%$ and reduces the EM value by 1.3$\%$. When we remove all relations (remove graph enhanced module), the $\mathrm{F_{1}}$ decreases by 2.6$\%$, and the EM value decreases by 2.0$\%$. From the results, it can be clearly confirmed that the graph we built plays an essential role in modeling table-text hybrid data, and it captures the semantic association through the message passing between different data types.
Effect of Subword Pooling Layer. In the graph input module, we used a multi-granularity type aware pooling method. The type classification criteria for word granularity are text and number, and for node granularity are question, table, and paragraph. As shown in the fifth row “w/o Multi-granularity type aware pooling” of Table 4, we eliminate this mechanism and unify the pooling approach for all types and granularities. Experimental results of $\mathrm{F_{1}}$ dropped by 0.6$\%$, which shows the effectiveness of this type-aware module.

Scale Study. Scale prediction is a unique challenge over TAT-QA and very pervasive in the context of finance. After obtaining the scale, the numerical or string prediction is multiplied or concatenated with the corresponding scale as the final prediction to compare with the ground-truth answer, respectively. We compare RegHNT with the baseline model for scale prediction results. The experimental results are shown in Table 5. Our model has significantly improved performance on both the dev and test datasets. To explore the impact of the scale on results, we use the gold scale to predict the answer. As shown in the third row of Table 6, model accuracy will slightly increase to 84.2$\%$ when we use the gold scale, which shows that it is necessary to improve the prediction of scale.
Operator Study. For TAT-QA dataset, there are four original answer types: Span, Multi-Span, Count, Arithmetic. As Figure 3 shows, we have adapted it into two categories, where the details of the expression tree for the span extraction questions are in the Appendix C. To investigate whether this category setting causes error propagation, we use the gold operator to predict the answer, and the results are shown in Table 6. When we use the gold operator, the EM and $\mathrm{F_{1}}$ of the model is improved by only 0.1. It suggests, to some extent, that we divide the data into two categories and use tree decoders to generate the answers separately. This approach has no significant impact on performance.

As Figure 4 shows, there are two questions and the prediction results of the models. For the question “What is the percentage change in total net sales…?”, the previous method could not generate complex arithmetic. In contrast, our model can generate the correct prefix expressions, which demonstrates the characteristics and advantages of our tree decoder. For the question “What was the average employee termination…?", the correct expression is derived from table and text (“53.0” from table and “1027” from paragraph). The results in Figure 4 show that the model often fails to answer correctly when a question requires the use of both tables and text. It focuses only on tables, choose “53.0” and “55.5”. This error type is very common in the previous methods. It shows that our graph encoder can better model the association between tables and texts.