Language Independent Neuro-Symbolic Semantic Parsing for Form Understanding

A naive solution that directly parses a document image to an entity-relation graph results in a time complexity quadratic to the number of words in a document in the worst case. First, a model needs to identify if a word belongs to the same group of its adjacent words, followed by predicting relations between any pairs of word groups. In the worst case, if each entity comprises only one word, the time complexity is $O(n^{2})$ because the number of edge predictions is $n\times n-1$, where $n$ is the number of words in a document. In fact, the average word lengths of entities in FUNSD [12] is 3.4, the actual inference time complexity is not far from the worst case. The cost is estimated without considering the one for estimating word groups for each entity.

We map entity-relation graphs to word-relation graphs so that entity-relation graph parsing becomes the task of predicting relations between words in a word-relation graph. To eliminate the task of entity recognition, we introduce an undirected relation same-entity to link two adjacent words in the same word sequence of an entity. For a relation $(v_{a},z,v_{b})$ between two entities in an entity-relation graph, we adapt the relation labels in entity-relation graphs to word-level relations:

• •

  Question-answer, the last word in a question is linked to the first word in the corresponding answer.

• •

  Header-question, the first word in a section is linked to the first word of the corresponding question.

• •

  Proximate_(V), the first word in $v_{a}$ is linked to the first word in $v_{b}$.

• •

  Proximate_(H), the last word in $v_{a}$ is linked to the first word in $v_{b}$.

• •

  Same-entity, the adjacent words within an entity are connected with this relation.

The conversion process is deterministic because i) for a relation between two entities, either the first or the last word of an entity is linked, and ii) same-entity links only adjacent words in an entity. Thus it is straightforward to revert the process to map a word-relation graph and an entity-relation graph by using rules. As a result, an word-relation graph is isomorphic to an entity-relation graph.

In this section, we present LaGNN that parses a document image to a word-relation graph, which is reformulated as predicting relations between words in a document. In contrast to prior studies [6, 26], our model relies on relative distances between the bounding boxes of words, which are robust across languages, rather than linguistic features. Moreover, we adopt Graph Attention Network (GAT) [22] to capture the similarities of relations between the neighbours of a word. However, we slightly modify GAT model to include edge features in addition to node features in message passing. We do this by concatenating our edge features with node features before performing the message passing. The coherence of predicted relations and graph properties of a word-relation graph are ensured by ILP at the inference stage.

Similar to prior works [26, 6], we apply the off-the-shelf OCR model [18] to map document images to bags of words. The OCR outputs are reorganized by sorting lines from top to bottom and arranging words from left to right in their original order by using their bounding-box coordinates. For each word $w$, we define its neighbourhood ${\mathcal{N}}_{w}$ as the set of words, to which it can potentially have relations. The set ${\mathcal{N}}_{w}$ consists of at most $K$ nearest words, including the word to the right of $w$ and the words below it. This definition of neighbourhood is motivated by the fact that the edge score between two words is independent of the orientation of the edge. Hence, during inference, we only compute edge scores between a word and each word in its neighbourhood ${\mathcal{N}}_{w}$ to avoid repeated computations.

We observe that a significant proportion of the neighbours of a word have the same relations. For example, all words in an answer composed of multiple words are associated with the relation same-entity. By viewing a form as a word-relation graph, we take the GAT model as our backbone because it supports both node features and edge features. Herein, each node $w$ is represented by a normalized bounding box coefficient vector ${\bm{h}}_{w}=(x_{left}/W,y_{top}/H,x_{right}/W,y_{bottom}/H)$, where $W$ and $H$ denotes the width and the height of a document image respectively. To support cross-lingual parsing, we represent an edge $(v_{a},v_{b})$ only by a relative spacing feature vector ${\bm{d}}_{ij}$. Such a feature vector ${\bm{d}}_{ij}$ is computed based on both horizontal and vertical relative distances between their normalized bounding boxes. Specifically, given two words $w_{i}$ and $w_{j}$ with the bounding boxes ($x_{i1},y_{i1},x_{i2},y_{i2}$), and ($x_{j1},y_{j1},x_{j2},y_{j2}$) respectively, the spacing between $w_{i}$ and $w_{j}$ is calculated along x and y-axis by

As a result, the spacing feature vector ${\bm{d}}_{ij}=[d_{x1},d_{x2},d_{x3},d_{y1},d_{y2},d_{y3}]$.

In comparison with the prior works [26, 25, 11, 6], which use text, image, and layout features altogether, our model is more parameter efficient by adopting only language-independent and layout-agnostic features. Due to those simple features, our model contains only 7.3K parameters, significantly less than those transformer-based models, which have approximately 200-400M parameters.

We formulate the inference problem as an integer linear program that aims to identify the most likely word-relation graph satisfying the graph properties outlined in Sec. 3.1.

Given an edge embedding ${\bm{e}}_{ij}$, we compute a score $s_{ijz}$ for each possible label $z\in{\mathcal{Z}}^{w}$ by using a linear layer. As a word-relation graph is sparse, we extend ${\mathcal{Z}}^{w}$ to the set ${\mathcal{Z}}^{+}$ by adding a relation no-relation, which indicates there is no relation between two words. Let $u_{jiz}\in\{0,1\}$ be a binary variable indicating if there is an edge with a label $z$ between $v_{i}$ and $v_{j}$, the solution to the following integer linear program is the most likely word-relation graph.

where ${\mathbf{A}}$ denotes the constraint matrix and ${\bm{b}}$ is the corresponding constant vector.

We formulate five constraints specific to documents for efficient joint inference using integer linear programming:

• •

  Connectivity constraint (C1): The generated word-relation graphs are fully connected such that there is a path between any pair of nodes in a graph.

  $$\sum_{v_{j}\in{\mathcal{N}}_{i}}\sum_{z\in{\mathcal{Z}}^{r}}u_{ijz}\geq 1,\forall v_{i}\in{\mathcal{V}}^{w}$$

• •

  QA constraint (C2): If a relation between two words $u_{ijz^{q}}$ is question-answer, where $z^{q}$ denotes question-answer, then the next immediate relation must be either proximate_(H) ($z^{h}$) or same-entity ($z^{w}$). Let the next relation is denoted by $u_{i(j+1)z}$, this constraint is defined as:

  $$u_{i(j+1)z^{h}}+u_{i(j+1)z^{w}}\geq u_{ijz^{q}}$$

• •

  Single label constraint (C3): There is only one relation in ${\mathcal{Z}}^{w}$ between any pair of nodes in a word-relation graph.

  $$\sum_{z\in{\mathcal{Z}}^{+}}u_{jiz}=1,\forall(v_{i},v_{j})\in{\mathcal{V}}^{w}\times{\mathcal{V}}^{w}$$

• •

  At least one semantic relations (C4): A word is part of an entity. Therefore, it is linked to another word via same-entity if the entity contains multiple words. If the word is at the beginning or the end of an entity, it points to another entity via either question-answer or header-question. In other words, a word is involved in at least one relation in ${\mathcal{Z}}^{s}=\{\textit{question-answer},\textit{same-entity},\textit{header-question}\}$. If an entity contains a single word, it should be associated with either question-answer or header-question. Otherwise, if a word is associated only with $proximate_{V}$ or $proximate_{H}$, we cannot determine which entity it belongs to and the type of the corresponding entity.

  $$\sum_{j\in{\mathcal{N}}}\sum_{z\in{\mathcal{Z}}^{s}}u_{jiz}\geq 1,\forall v_{i}$$

• •

  At least one semantic relation in the neighbourhood (C5): The previous constraint can still fail to exclude the cases that a word is only involved in same-entity in ${\mathcal{Z}}^{s}$. In such a case, the word is expected to be at either end of an entity. If we need to infer the type of entity, this word should be linked to another word via either question-answer or header-question. Therefore, if a word is linked with a same-entity relation, it should be linked to a different word with a relation in ${\mathcal{Z}}^{s}$.

  $$u_{i(i+1)z^{h}}+u_{i(i+1)z^{w}}+u_{i(i+1)z^{q}}\geq u_{(i-1)iz^{w}}$$

After parsing a document image into a word-relation graph, we apply the deterministic rules introduced in Sec. 3.1 to convert the graph into an entity-relation graph.

Inspired by [8], we apply the cross-entropy loss to each pair of nodes during training. The construction of word-relation graphs is realized by the ILP-based inference method detailed above.

where $z_{ij}$ and $\hat{z_{ij}}$ denote the ground-truth labels and predicted labels respectively.

FUNSD [12] It is a form understanding dataset consisting of 199 noisy documents which are fully annotated. It has a total of 9,707 semantic entities over 31,485 words. In the official data split, the 199 samples are divided into 149 training samples and 50 testing samples. However, we further split the 149 samples into 139 training and 10 validation samples. Our test split is the same as the official split. Each entity is labelled with one of the four semantic entity labels - ”question”, ”answer”, ”header”, and ”other”. This dataset is widely employed for semantic entity labelling and relation extraction tasks.

To obtain the relation annotations in entity-relation graphs on both datasets, we apply rules to generate the relation annotations based on entity annotations, followed by manually checking all document images for correctness. Due to the isomorphism between word-relation graphs and entity-relation graphs, we map the resulting entity-relation graphs to word-relation graphs by using the deterministic rules.

The entity-relation graphs are constructed using Deep Graph Library (DGL). We use a single-layer GAT with 3 heads and a hidden dimension size of 64 for both node and edge features. We train our model using Adam optimizer for 500 iterations with a learning rate of $1\times e^{-3}$. If the performance on the validation data does not improve after 100 iterations, training stops early. During training, we save the model checkpoint based on its performance on validation data. For inference on the test set, the checkpoint with the best performance across all training epochs is loaded. We train our model on 1 GTX 1080Ti 12 GB GPU.

We first evaluate our model on the word-relation graphs on FUNSD by considering the task as relation extraction between words. More specifically, we run the state-of-the-art models LayoutLM [26], LayoutLMv2 [25], LayoutLMv3 [11], StructuralLM [15], LayoutXLM [6], InfoXLM_BASE [3], and LiLT[InfoXLM]_BASE [23] to predict entities, followed by relation extraction.

For baselines, we perform relation extraction by following the approach in [6]. First, we create all possible entity pairs as relation candidates. Each candidate is represented by the concatenation of the corresponding entity representations. Furthermore, the representation of an entity is first constructed by concatenating the embedding of the first token of each entity and the entity type embedding, followed by feeding them through two position-wise feed-forward networks (FFN) modules. The resulting relation candidate representations are fed into a bi-affine classifier for relation classification. The conversion from entity-relation graphs to word-relation graphs is performed by the same set of deterministic rules introduced in Sec. 3.1.

Table 1 reports the relation extraction results on word-relation graphs in terms of Precision, Recall, and F1. For each metric, we take the micro-average among the relations in ${\mathcal{Z}}^{w}$. The two variations of our model achieve superior performance over the baselines by only using the relative spacing features. In contrast, all baselines use textual features extracted from large language models that require pre-training on large-scale datasets. Some of the baselines, such as variations of LayoutLM, require even vision features. The number of parameters of our models is also significantly smaller than their competitors. Although pre-trained language models are widely used in a number of AI applications, our work raises the basic question for future research “Are language models necessary for form recognition?”.

Apart from relation extraction, we also evaluate the models in terms of entity recognition on converted entity-relation graphs. Table 2 summarizes the results based on an exact match in terms of Precision, Recall, and F1.

We further evaluate model performance by applying the models trained on FUNSD directly to the scanned forms in other languages on XFUND. Herein, we compare our models with the state-of-the-art methods: XLM-RoBERTa_BASE, InfoXLM_BASE, LayoutXLM_BASE [6], and LiLT[InfoXLM]_BASE [23]. The latter two models are pre-trained on large-scale document datasets of size 30M and 11M respectively.

Table 3 reports the corresponding zero-shot multilingual relation extraction results on the word-relation graphs of XFUND dataset. Overall, our best model outperforms the baselines in 7 out of 8 languages in terms of F1. The geometric mean of the F1 is more than 10% better than the strongest baseline. It is noteworthy that performance improvement is achieved without any time-consuming pre-training. We only use the 139 forms from the FUNSD dataset to train our models. Our models based on relative spacing features are more transferrable than those multilingual language models on this task. A further inspection shows that our models benefit from the fact that the space between two words within an entity, as well as the distance between a question word and an answer word, are similar across languages. In this zero-shot multilingual setting, incorporating the constraints into inference does not always help, though it leads to improvements in 6 out of 8 languages.

We evaluate the effectiveness of using GAT, edge features, constraints in ILP, and the size of neighbourhood in our model. To show the usefulness of GAT, we compare it with GraphSAGE [10] and graph convolutional networks (GCNs) [14] by using the same features. For edge features, we run ablation studies by removing them with or without applying ILP. To understand the usefulness of the constraints during inference, we remove the connectivity constraint (C1), QA constraint (C2), and the last two semantic constraints (C4+C5) respectively from the full model.

As shown in Table 4, it is expected that our full model performs the best among all variations. Removing relative spacing features leads to the largest drop in terms of all metrics. GAT demonstrates its strengths over the two alternative neural structures. Removing any of the constraints, the model performance drops slightly. Among them, C2 is clearly the most useful one among them in terms of improving performance. As such, C2 is the qa constraint that is designed to correct the wrongly predicted question-answer relation labels, which are one of the most frequent model prediction errors based on our evaluation. Before applying ILP, we observe that there were 169 out of 51775 word pairs in the test set that violate this constraint. Table 5 illustrates the number of violations in the predictions of LaGNN before applying the constraint-based inference.

The constraints C4 and C5 are designed to ensure isomorphism between entity-relation graphs and word-relation graphs. This constraint assists in preventing any isolated word pairs that are not connected to an entity (question, or header). For instance, if a question has more than one word in the answer, the words within the answer are linked together using the same-entity relation. The last word of the question and the first word of the answer are linked by the question-answer relation. When transforming a word-relation graph to an entity-relation graph, we take into account the rule that any word pairs with a same-entity connection following a question-answer relation are mapped to the entity relation ”answer” in the entity-relation graph. However, if LaGNN predicts a different relation for a word pair within the answer than same-entity, this will result in an inaccurate mapping of word-relation graph to entity-relation graph. Additionally, a word pair can only have a same-entity relation if it is part of a chain of words whose head is linked to either a ”question” or a ”header.” It is impossible to have an isolated word pair with a same-entity relation that isn’t related to either the ”question” or ”header” in our proposed way of word-relation graph construction. However, a few of the LaGNN model’s relation predictions could result in isolated word pairs. In order to prevent this, our uniquely designed semantic constraints correct these mistakes and aid in the accurate mapping of the word-relation graph to the entity-relation graph. There are 134 total isolated word pairings before semantic constraints are applied. However, with the application of the constraints, there were no violations, and the performance benefit from using semantic constraints—which are made particularly to achieve isomorphism between word-relation graph and entity-relation graph —is very little.

Where to put edge features To demonstrate the significance of using spacing between pairs of nodes as its edge feature, we conduct four experiments with and without edge features. The results for these experiments are available in Table 6. Our key objective is to determine whether it is more effective to concatenate edge features with node features when computing the node attention rather than doing so only at the model’s final classifier layer. It is evident that employing edge features solely while computing node attention (row 2) results in much lower performance than using the model’s final (classifier) layer (row 3). Nevertheless, it is still marginally preferable to not use edge features at all (row 4). The optimal performance can be obtained by employing edge features while computing node attention as well as at the last layer of the model (row 1).