Semantic Enhanced Text-to-SQL Parsing via Iteratively Learning Schema Linking Graph

We first formulate the Text-to-SQL task as follows. Given a natural language question $Q=(q_{1},q_{2},\cdots,q_{|Q|})$ and database schema $S$, which contains multiple tables $T=\left\{t_{1},t_{2},\cdots,t_{|T|}\right\}$ and multi columns $C=\left\{c_{1},\ldots,c_{|C|}\right\}$, the goal of Text-to-SQL system is to generate a SQL query $y$.

As a graph encoder is included in our framework, we give a definition to the question-schema graph. The entire graph can be represented as $G=\left(\mathrm{V},E\right)$, which contains all three types of mentioned nodes above. The node sets $V=Q\cup T\cup C$ , in which the number of nodes $\left|V\right|=|Q|+|T|+|C|$. Here, we denote $S=T\cup C$ to represent all the schema nodes. The edge set $E=E_{Q}\cup E_{S}\cup E_{Q\leftrightarrow S}$ contains three kinds of edges: $E_{Q}$, $E_{S}$, and $E_{Q\leftrightarrow S}$. $E_{Q}$ contains the edges inside questions tokens, which are defined by the sequence adjacency relationship, $E_{S}$ includes the edges inside schema items, which are defined by the pre-existing database relations (e.g., primary key relation). Meanwhile, $E_{Q\leftrightarrow S}$ contains the edges between question nodes and schema nodes, which are generated by the schema linking component in Text-to-SQL models. Our work mainly focuses on the construction of $E_{Q\leftrightarrow S}$. The total relation matrix of the graph can be defined as follows:

where $|S|=|C|+|T|$ is the length of the schema items. Note that the graph is heterogeneous and the value in $E$ represents the edge type (e.g., 1 for semantic linking relation, 2 for primary key relation).

The initial Graph Probing module aims to obtain a semantic schema linking graph, which is achieved by masking one question token per time and observing how the PLM (e.g., BERT (Devlin et al., 2019)) embeddings of the schema items are affected. Different from the exact matching based method, the generated graph is not dependent on the surface form of the question tokens. Specifically, in this section, we aim to generate an adjacency matrix for an edge type $r^{sem}$ in heterogeneous graph $E_{Q\leftrightarrow S}$ (Eq. 1).

We first concatenate the question tokens and schema items as $X$:

The PLM maps the input $X$ into hidden representations. The question and schema embeddings can be denoted as $(\mathbf{q}_{1},\mathbf{q}_{2},\ldots,\mathbf{q}_{|Q|})$ and $(\mathbf{s}_{1},\mathbf{s}_{2},\ldots,\mathbf{s}_{|T|+|C|})$ respectively.

Our goal is to induce a function $f\left(q_{i},s_{j}\right)$ to capture the impact a question word $q_{i}$ has on the representation of schema item $s_{j}$. Inspired by the previous probing method (Wu et al., 2020), we utilize a two-stage approach to achieve this goal in an unsurpervised manner. First, we replace a question token $q_{i}$ with spacial token [MASK] and feed the new sequence $X\backslash\left\{q_{i}\right\}$ into the PLM again. Then, the representation of schema item $s_{j}$ is changed from $\mathbf{s}_{j}$ to $\mathbf{s}_{j\backslash q_{i}}$, which is affected by the [MASK] of $q_{i}$. We define the $f\left(q_{i},s_{j}\right)$ by measuring the distance between $\mathbf{s}_{j}$ and $\mathbf{s}_{j\backslash q_{i}}$ as follows:

where $d(\mathbf{x},\mathbf{y})$ is the Euclidean distance between two vectors.

To obtain a sparse initial schema linking matrix and avoid the noisy edges with low confidence level, we introduce a pre-defined threshold $\tau$ to filter the probing result by:

where $\mathbf{A}^{(init)}\in\mathbb{R}^{|Q|\times(|T|+|C|)}$.

To repair the noisy and incomplete initial schema linking graph obtained before training, the Implicit Graph Learning module aims to learn an adaptive graph structure and iteratively refine it during model training.

The goal of the implicit graph learning module is to learn an implicit matrix $\mathbf{A}^{(t)}\in\mathbb{R}^{|Q|\times(|T|+|C|)}$ between question tokens and schema items, where $t$ is the epoch number in training. Specifically, we first calculate the similarity matrix $\mathbf{A}^{(t)}$ by a cosine similarity based function as follows:

where $ReLU(x)$ is the rectified linear unit activation function (Glorot et al., 2011) adopted to retain the positive part of its argument. $\mathbf{W}_{1}$ and $\mathbf{W}_{2}$ are two learnable weight vectors that transform the question/schema embeddings for similarity calculation and $<\mathbf{x},\mathbf{y}>$ denotes vector inner product operation.

We could get an weighted similarity matrix from the function in Eq.5. In practice, most schema items only link to one question node, so we propose a graph sparsification component which only keeps the max similarity score for each schema. Our graph sparsification component is defined as follows:

where $A^{t}_{j}=\left[A^{t}_{0,j},\ldots,A^{t}_{|Q-1|,j}\right]$ is the similarity vector of each schema based in Eq.5.

Given the initial schema semantic linking matrix $\mathbf{A}^{init}$ and the implicit semantic schema-linking matrix $\mathbf{A}^{(t)}$, the graph encoder module tries to learn a better question token and schema item embedding with the enhanced schema linking graph, which could help Text-to-SQL decoder to identify the correct schema item.

The schema linking graph construction procedure at epoch $t$ can be represented as:

where $\lambda$ is a hyperparameter used to adjust the weights of the two graphs. As shown in Eq.1, the input graph of the graph network is a heterogeneous network which contains different edge types (e.g., semantic linking relation between the question and schema and primary key relation inside schema items). $\widetilde{\mathbf{A}}^{(t)}$ can be seen as a weighed matrix of a specific relation $r^{sem}$. Given $\widetilde{\mathbf{A}}^{(t)}$, we update the graph $E_{Q\leftrightarrow S}$ in Eq.1 as:

In epoch t, the whole graph $\mathbf{E}^{(t)}$ is then generated by replacing $\mathbf{E}_{Q\leftrightarrow S}$ in Eq.1 by $\mathbf{E}_{Q\leftrightarrow S}^{(t)}$. Then we expand the weighted matrix $\widetilde{\mathbf{A}}^{(t)}$ to the whole graph, which can be represented as:

As shown in Eq.9, we only calculate the weight matrix between question and schema. The weight of other graphs can be directly set to 1 because there is no uncertainty in these edges.

To process the input heterogeneous graph $E$, we introduce a relational graph attention network (RGAT) (Wang et al., 2020a) as our graph encoder. The RGAT network utilizes a learnable relational embedding to control the different impacts among edge types. To simplify, we use $\mathbf{X}^{l}\in\mathbb{R}^{(\left|Q\right|+\left|C\right|+\left|T\right|)\times d}$, to denote the entire node embedding matrix in layer $l$ where d is the GNN embedding size. Similar to previous works (Wang et al., 2020b; Cao et al., 2021), we utilize the multi-head scaled dot-product for attention weights calculation. To optimize the weighted matrix $M^{(t)}$ in each epoch, different from previous works, we introduce $M^{(t)}$ to the graph attention calculation process.

Specifically, given the current node representations $\mathbf{X}^{l}$, graph $E^{(t)}$ and weighted matrix $M^{(t)}$, the attention weight $\alpha_{ji}^{h}$ is calculated by following equations:

where matrices $\mathbf{W}_{q}^{h},\mathbf{W}_{k}^{h},\mathbf{W}_{v}^{h}\in\mathbb{R}^{d\times d/H}$ are trainable transformations and H is the number of heads. Function $\psi\left(\right)$ is a learnable matrix which transforms the relation type $E_{ji}^{(t)}$ into a d-dim feature vector. $\mathbf{M}^{(t)}_{ji}$ is used to control the influence of relation embedding by a simple multiplication operation. Operator $[\cdot]_{h}^{H}$ first evenly splits the vector into $H$ parts and returns the $h$-th partition.

Then the output representation of current layer $\mathbf{x}_{i}^{l+1}$ is generated by calculating the relation embedding in a similar way:

where matrices $\mathbf{W}_{o}\in\mathbb{R}^{d\times d}$ is the output transformation, $\|$ represents vector concatenation operation and FFN $(\cdot)$ denotes one feedforward neural network layer.

Since the weighted matrix $M^{(t)}$ is optimized in each epoch, our schema linking graph could be updated iteratively during model training.

The final output of the graph encoder $\mathbf{x^{e}}$ can be split into question representation $(\mathbf{q}_{1}^{e},\mathbf{q}_{2}^{e},\ldots,\mathbf{q}_{|Q|}^{e})$ and schema representation $(\mathbf{s}_{1}^{e},\mathbf{s}_{2}^{e},\ldots,\mathbf{s}_{|T|+|C|}^{e})$ for the computation of Text-to-SQL decoder.

Given the question representation $(\mathbf{q}_{1}^{e},\mathbf{q}_{2}^{e},\ldots,\mathbf{q}_{|Q|}^{e})$ and schema representation $(\mathbf{s}_{1}^{e},\mathbf{s}_{2}^{e},\ldots,\mathbf{s}_{|T|+|C|}^{e})$ generated by graph encoder module, Text-to-SQL decoder module aims to generate the corresponding SQL statements.

The architecture of our Text-to-SQL decoder is similar to previous work (Yin and Neubig, 2017), which generates the abstract syntax tree (AST) of the target SQL $y$ in depth-first traversal order. The action generated by decoder in each timestep is either: (1) An APPLYRULE action that expands the current non-terminal node based on SQL grammar in the partially generated AST. (2) A SELECTTABLE or SELECTCOLUMN action that chooses one table or column item from the encoded schema representation $(\mathbf{s}_{1}^{e},\mathbf{s}_{2}^{e},\ldots,\mathbf{s}_{|T|+|C|}^{e})$ ²²2More implementation details are provided in appendix.

Although the combined graph of the initial graph $\mathbf{A}^{init}$ and implicit graph $\mathbf{A}^{(t)}$ could approach the optimized graph iteratively, the quality of the implicit graph is not guaranteed. Due to that the schema linking module is to find the mentioned schema in question, it is more important for the implicit graph to find the correct schema items than question tokens. In the following, we introduce the auxiliary graph regularization loss to further optimize the schema linking graph

For each SQL query $y$, we denote the set of column and table names appearing in $y$ as $S_{SQL}$, which consists of columns and table mentions, i.e., $S_{SQL}$ = $S_{COL}\cup S_{TBL}$. During the training process, we can directly utilize $S_{SQL}$ as the supervision of our graph regularization loss. Specifically, we make the sum of $\mathbf{A}^{(t)}$ in the question dimension approximate the $S_{SQL}$ by binary cross-entropy loss.

Note that in Eq.6, only the largest value item of question dimension in $\mathbf{A}^{(t)}$ is retrained. Therefore, the graph regularization loss actually only optimizes the most relevant question of mentioned schema.

This graph regularization auxiliary loss is combined with the main Text-to-SQL loss as follows:

where $\mu$ is a hyperparameter that balances the absolute value of the two losses.

Table 1 shows the exact match accuracy on three benchmarks with the exact match average accuracy of 3 runs. LGESQL is the previous state-of-the-art model in all three embedding configurations (without PLM, with BERT-large and with model adaptive PLM). In general, ISESL-SQL could outperform previous baselines in all configurations. More specifically, with GloVe word vectors, our model could surpass the previous best model by an average of $2.5\%$ over all benchmarks. Similarly, ISESL-SQL could achieve an average performance boost of $1.3\%$ with a unified pre-training model BERT-large over the three benchmarks. Furthermore, our ISESL-SQL achieves state-of-the-art performance with ELECTRA-large (Clark et al., 2020) on the model adaptive PLM setting.

We observe that ISESL-SQL could obtain greater improvement on Spider-SYN and Spider-DK benchmarks than standard spider benchmark, which proves the robustness of ISESL-SQL in challenging datasets. Also, since the ability for GloVe word vectors to capture semantic linking is inferior to PLM embeddings, ISESL-SQL achieves more performance boost with GloVe word vectors setting.

We conduct ablation studies to show the effectiveness of different modules of ISESL-SQL to the overall improved performance. ISESL-SQL w/o initial graph pruning is the proposed model without the initial probed graph and only keeps the learned graph during training. ISESL-SQL w/o implicit graph learning only adopts the initial probed graph with no more graph optimization process. ISESL-SQL w/o schema linking replaces all the edges in $\mathbf{E}_{Q\leftrightarrow S}$ (Eq.1) to none-relation edge type $r^{none}$. ISESL-SQL w/o graph regularization removes the graph regularization loss and only keeps the Text-to-SQL loss during training. ISESL-SQL w exact match schema linking replaces our semantic schema linking edge with the the exact match schema linking edges.

A general conclusion from ablation results in Table 2 is that all modules contribute positively to the improved performance. More specifically, ISESL-SQL w/o initial graph probing gives us $1.4\%$ less performance averaged on all datasets. Similarly, implicit graph learning gives $1.3\%$ performance boost in average over all benchmarks. Removing the graph regularization loss leads to an average of $0.9\%$ performance drop. Furthermore, when totally removing the schema linking edges, ISESL-SQL w/o schema linking brings an average performance drop of $1.4\%$. Compared with the exact match schema linking method, our ISESL-SQL gives an average improvement of $2.4\%$.

We can discover that the initial graph probing module contributes more in spider-SYN and spider-DK than spider benchmark, which proves the importance of semantic linking edges in challenging settings. Also, the exact match schema linking method is even worse than no schema linking settings, which shows it vulnerability in challenging settings.

To better demonstrate the quality of our constructed schema linking graph, we compare the schema information (tables and columns) produced by ISESL-SQL with human annotations.

Following the previous works (Lei et al., 2020; Liu et al., 2021), we report the micro average precision, recall and F1-score for both columns ($Col_{P}$ , $Col_{R}$, $Col_{F}$) and tables ($Tab_{P}$ ,$Tab_{R}$, $Tab_{F}$). The metric focus on whether the correct schema item is identified.

We consider five strong baselines for comparison. (1) N-gram Matching links all n-gram ($n\leq 5$) phrases in a natural language question to schema items by fuzzy string matching. (2) SIM computes the cosine similarity between question tokens and schema items using PLM embeddings without task specific fine-tuning. (3) CONTRAST learns by comparing the aggregated embedding scores of mentioned schemas with unmentioned ones in a contrastive learning style, as done in Liu et al. (2020). (4) SLSQL_L (Lei et al., 2020) is trained with full schema linking supervision by a learnable schema linking module. (5) ETA (Liu et al., 2021) trains the schema linking module using pseudo alignment as supervision generated from PLMs.

Table 3 shows the experimental results on the schema linking accuracy. Our proposed ISESL-SQL model could surpass all weakly supervised and unsupervised methods by a large margin. Specifically, our method brings an improvement of $3.7\%$ on $Col_{F}$ and $5.1\%$ on $Tab_{F}$ over the previous best baseline ETA (Liu et al., 2021). SIM is a similar method to our initial graph probing component which constructs schema linking graph in an unsupervised way. We discover that our method outperforms SIM by a huge margin, which indicates the effectiveness of our iterative graph learning process. Surprisingly, our ISESL-SQL method could outperform SLSQL_L which is trained in a fully supervised manner. This indicates that the weak supervision from schema information could achieve a similar result with fully supervision.

To further investigate how our model learns the schema linking graph iteratively during the training process, we visualize the changing trend of schema match F1 score during the training process in the spider development set. As shown in Figure 3, our method could outperform the previous best baseline ETA during the entire training process. And after epoch 45, our ISESL-SQL surpasses the N-gram match based method. Our model could achieve the best schema linking result before epoch 100 in both column matching and table matching. Also, it can be seen that our ISESL-SQL model has an initial F1 score before training, which is the result of the initial graph probing. These results prove our model could iteratively refine the schema linking graph during the training process.

To better analyze the reasons for the performance improvements achieved by our ISESL-SQL, we perform a detailed analysis of the matching accuracy of different components of the SQL statements on the Spider-SYN benchmark. As shown in Table 4, the performance improvement mainly comes from the components including schema items, such as SELECT, SELECT (no AGG), WHERE and WHERE (op). Specifically, on the WHERE (op) component, our ISESL-SQL model outperforms the previous best baseline by $2.2\%$. On other components where schema items are not included, the performance remains unchanged or slightly drops on some components, such as IEU and AND/OR. The observation shows a high-quality schema linking graph could help the Text-to-SQL model find the correct schema items.

One natural question is how much improvement will be made when the column and table mentioned information (oracle schema information) is provided, As shown in Table 5, we report the exact match accuracy of our ISESL-SQL model on Spider and Spider-SYN benchmarks with oracle schema information provided. Specifically, we directly change the implicit graph matrix $\widetilde{\mathbf{A}}^{(t)}$ in Eq. 7 based on the oracle information.

From Table 5, we could discover that: 1) Oracle schema linking information could bring a huge improvement (7.7% and 17.9% in Spider and Spider-SYN benchmarks respectively) to the final Text-to-SQL accuracy, which proves the importance of schema linking component. 2) When oracle schema linking information is provided, the results on the Spider-SYN benchmark (80.1%) will be close to those on the Spider(83.1%) , which demonstrates that the original performance drops in the Spider-SYN benchmark mainly come from the corrupted schema linking graph. 3) Only providing oracle schema information (which schema item appears) could achieve a similar improvement compared to the oracle schema linking, which could demonstrate our assumption that schema information is more important in the schema linking graph. 4) Oracle column information contributes more than the oracle table information, which is because the column information is more difficult to capture.

Based on the above results, we can conclude that there is a huge potential for improvement by designing a better schema linking component, which could be an important direction for future work.

To intuitively show the effectiveness of our model, we select four cases from Spider-SYN and Spider-DK benchmarks to compare the generated SQL statements of our ISESL-SQL model and LGESQL. The first two cases are from the Spider-SYN benchmark and the last two cases are from the Spider-DK benchmark. As shown in Table 6, we can observe that our model could generate correct SQL even in synonym substitution scenario. As in the first case, when the schema-related token dogs is replaced with puppies in the question, LGESQL fails to identify table name dogs while our ISESL-SQL model could successfully recognize the correct table. In the third case, the token highest in question could be matched to the column highest via the exact match based method, which leads to the error in LGESQL. Since our method uses a semantic-based approach to do matching, the above problem could be avoided in most cases. However in the more complicated cases (e.g. the forth case in Table 6), ISESL-SQL may fail to recognize the complex structure information.

Figure 4 shows the alignment matrix between question and schema during the training process. We take the sentence Show the ages for all French Musicians and the schema in the database concert_singer as an example. The first subfigure shows the alignment matrix generated by the initial graph probing component before training. Although the linking of column ‘age’ and table ‘singer’ is identified, the initial graph probing component still fails to capture the linking between French and country. As we can see, during the training process, the graph is iteratively optimized. The correct linking is emphasized in epoch 50. Finally, in epoch 100, our method could almost exclusively focus on the correct linking. The whole process demonstrates that our ISESL-SQL method could generate better schema linking information iteratively during training.

Text-to-SQL Parsing is an essential sub-field of Natural Language Processing and could benefit many other tasks such as Question Answering and Information Extraction (Chen et al., 2017; Lin et al., 2022; Hu et al., 2021a; Liu et al., 2022; Hu et al., 2021b; Li et al., 2022a; Hu et al., 2020). Schema linking is an indispensable module in recent Text-to-SQL models, which establishes a link between question tokens and schema items (Gan et al., 2020). Many previous works treat schema linking as a minor pre-processing procedure (Guo et al., 2019; Wang et al., 2020b; Xu et al., 2021a; Cao et al., 2021; Li et al., 2022b), which use surface form match based method to find the occurrences of the schema names in the question. However, these methods may fail in synonym and typo scenarios. Other works implement schema-linking by learning a similarity score between a word and a schema item (Bogin et al., 2019a; Krishnamurthy et al., 2017), but still suffer from the limitation of the static word embedding. Guo et al. (2019) and Wang et al. (2020b) conduct an ablation study on schema linking, and the results show that removing the extra schema linking causes the biggest performance decline compared to removing other removable modules. To better investigate the influence of schema linking, Lei et al. (2020) and Taniguchi et al. (2021) invest human resources to annotate schema linking information in dataset and employ the full supervision to train Text-to-SQL models. The result shows that with gold schema linking information as training data, current Text-to-SQL models can exceed the baseline by a very large gap. Dong et al. (2019) utilize implicit linking supervision to train their schema linking model with reinforcement learning method. Liu et al. (2021) train the schema linking module using pseudo alignment as supervision from PLMs, but the pseudo alignment could still be noisy. Our ISESL-SQL iteratively refines the schema linking graph using both supervision from SQL generation task and the schema mention information in SQL statements.

Graph neural networks (GNNs) are neural models that capture the dependence of graphs(Zhou et al., 2020). The structure of the graph is normally labeled by experts (Sen et al., 2008) , pre-processed with existing relation parsing tools (Wu et al., 2021) or precomputed by the k-nearest neighbor algorithm (Anastasiu and Karypis, 2015). However, the graph structure generated by these methods may contain missing or noisy edges which may not be optimal for downstream learning tasks. To alleviate this problem, some works propose a robust graph learning schema by removing noise and errors in the raw data adaptively (Kang et al., 2019). Similar to these works, graph attention network is proposed to use self-attention mechanism to reweight edge importance (Velickovic et al., 2018). As these robust learning approaches still cannot handle the missing edges, adaptive graph structure learning methods are proposed to facilitate downstream graph-based tasks (Jiang et al., 2019). These adaptive graph construction approaches typically utilize a parameterized graph similarity metric learning function to learn an adjacency matrix by considering pair-wise node similarity in the embedding space (Wu et al., 2021). These models only perform graph structure learning for one time which is not enough. To better optimize graph structure and downstream tasks jointly, Chen et al. (2020) proposed an iterative deep graph learning model to let graph learning models and downstream tasks optimize together. In the scenarios when the input graph is not available, LDS (Franceschi et al., 2019) is proposed to learn the graph structure by solving a bilevel program that learns the discrete probability over the graph egdes. However, these models cannot be applied to Text-to-SQL directly because the graph in Text-to-SQL model is heterogeneous which contains different types of nodes and edges. In this paper, we propose a framework to introduce the graph structure learning network into a modified version of Relation-aware graph attention network (Wang et al., 2020a) to enhance Text-to-SQL models.