Automatically Generating Counterfactuals for Relation Classification

We introduce the structure casual model (SCM) (Judea, 2000) to investigate the casual relationship between data and the RC model, where random variables are vertices and an edge denotes the direct causation between two variables. Before start, we first propose our casual questions to guide the generation of SCM. (1) What would happen if the important syntactic structure SY around the entity is changed? (2) What would happen if the semantic path SE between two entities in the text changes?

Based on the casual questions, we consider two important factors $SY$ and $SE$ as variables in SCM to capture the casual relation in the text. As shown in Figure 1(a), there is a confounding variable $G$ that influences the generation of $SY$ and $SE$, $Y$ is the label variable, and $U_{*}$ represents the unmeasured variable. $X$ $\to$ $Y$ means there exits a direct effect from $X$ to $Y$. Furthermore, the path $SY$/$SE$ $\to$ $X$ $\to$ $Y$ denotes $SY$/$SE$ has an indirect effect on $Y$ via a mediator $X$. $Y$ can be calculated from the values of its ancestor nodes, which is formulated as: $Y_{sy/se,x}=f(SY=sy/SE=se,X=x)$, where $f(.)$ is the value function of $Y$.

In order to estimate the casual effects of the variable $SY$/$SE$ on the target variable $Y$, we need to block the influence of the confounding variable $G$ on $SY$/$SE$, and see how $Y$ is changed by a unit intervention when fixing the value of $sy$/$se$ as $sy$^∗/$se$^∗, as shown in Figure 1(b).

Since a counterfactual should flip the label of the original sample $X_{\text{ori}}$, the candidate substitute $X_{\text{can}}$ which will be used for identifying causal features is randomly chosen from the training samples with different class labels ¹¹1In our implementation, we randomly select three candidate substitutes for the original sample. If none of generated candidates meets the requirements, they will all be discarded. If there are multiple candidates qualified as counterfactuals, one of them is chosen at random.. Moreover, since entities are of the most importance in RC tasks, the entities in the substitute should be most similar with those in $X_{\text{ori}}$. In view of this, we first identify the syntactically similar entity nodes in SynDG using the centrality metric, which is proposed to account for the importance of nodes in a graph (network). We employ three types of centralities including betweenness centrality (BC), closeness centrality (CC), and degree centrality (DC) for this purpose. After that, we generate the contextual counterfactual for these syntactically similar entities to meet the condition of invariant entities, i.e., instead of changing entities, we intervene on their contexts. We term this proposed method SynCo as the substitution of casual features is based on the syntactic graph SynDG.

We take two samples for illustration: an original sample $X_{\text{ori}}$ “They drank wine produced by wineries.” with “Product-Producer” relation, and a candidate sample $X_{\text{can}}$ “They bought the grapes from farms” with “Entity-Origin”. Their SynDGs and the intervention procedure are shown in Figure 2. The main procedure for SynCo is summarized below.

1. We first calculate the average score for three centrality metrics of the entity and denote it as avgC(entity), e.g., avgC(wine_o) and avgC(grapes_c) is 1.24 and 1.17.

2. We then calculate the topological distance (TD) of two entities by minusing their avgC values. If TD is smaller than a predefined topological distance threshold (TDT), they form a candidate entity pair for substitute structures. For example, given TD(avgC(wine_o), avgC(grapes_c)) = 0.07 $<$ TDT and TD(avgC(wine_o), avgC(farms_c)) = 0.68 $>$ TDT, the entity ‘grapes’ in $X_{\text{can}}$ and the entity ‘wine’ in $X_{\text{ori}}$ form a candidate entity pair.

3. We now generate the candidate syntactic counterfactual. We calculate the cosine similarity of the syntactic features (POS and dependency relation embedding) for entities in the candidate entity pair. We denote it as FS. If it is greater than a predefined feature similarity threshold (FST), we substitute the first-order neighbors around the entity in $X_{\text{ori}}$ with those of the candidate entity in $X_{\text{can}}$. These neighbors should have the same type of POS tags to ensure the correctness of the syntax. Moreover, if there are several words around entities and they all have the same type, we will replace the one with the smallest TD. For example, given FS(wine_o, grapes_c) = 0.877 $>$ FST, we can substitute the verb-type neighbors ‘drank_o’ and ‘produced_o’ of ‘wine’ with the same type neighbor ‘bought_c’ of ‘grapes’. Moreover, since TD(avgC(drank_o), avgC(bought_c)) = 0.57 and TD(avgC(produced_o), avgC(bought_c)) = 0.50, we replace ‘produced_o’ with ‘bought_c’ and retain ‘drank_o’ unchanged.

4. After substituting the contexts of the candidate entity pair and adding the label of $X_{\text{can}}$, a candidate counterfactual is produced. In order to ensure that the sample we generate is a real counterfactual, we put it into the trained backbone model and re-predict its label. If the label is indeed changed, we treat it as the counterfactual of $X_{\text{ori}}$. For example, “They drank wine bought from wineries.” with “Entity-Origin” relation is a counterfactual sample after SynCo.

This section presents our semantically intervened counterfactual generator SemCo. We prefer to replace the contexts between entities in $X_{\text{ori}}$ with words of the most similar semantics and get a different label. To this end, we exploit the shortest path in SemDG for identifying semantically similar contexts between entities.

As shown in Figure 3, we first obtain the SemDG of an original sample “Wine is in the bottle” with “Content-Container” label and that of a candidate sample “Letter is from the city” with “Entity-Origin” label. We then extract the SDP between entities in $X_{\text{ori}}$ and $X_{\text{can}}$. We propose to calculate the cosine similarity of averaged word embeddings between two paths. If the similarity score is larger than the semantic similarity threshold (SST), we replace the semantic path in $X_{\text{ori}}$ with that in $X_{\text{can}}$. Similar to SynCo, we put the generated semantic counterfactual into the trained backbone model and re-predict its label. We consider the new sample as a counterfactual of $X_{\text{ori}}$ if the label is changed.

Finally, the generated semantic counterfactuals, as well as the syntactic counterfactuals, are used to augment the original data to train a robust classifier.

We briefly introduce four datasets in our experiments.

We evaluate our model with a dedicated protocol. We first adopt three typical RC methods as the backbone, which will be used as the encoder for the input and for training the base classifier for the prediction of the candidate counterfactual. Moreover, the backbone is used for training the final classifier on the original data and CAD. The backbone RC methods include PA-LSTM (Zhang et al., 2017), AGGCN (Guo et al., 2019), and R-BERT (Wu and He, 2019). They are chosen as the representative of sequence based models, graph-based models, and the PLMs methods, respectively. R-BERT is the best among the state-of-the-art BERT${}_{\text{base}}$ methods. Note the backbone methods will be combined with all counterfactual generation methods including ours and the baselines.

We further compare our model with automatical counterfactual generation baselines. These methods are either general text generator or designed for other tasks. For the general methods (Madaan et al., 2021; Fern and Pope, 2021; Robeer et al., 2021), we directly generate counterfactuals on RC datasets using their generator, and we avoid masking entities for a fair comparison with our model. Among the specific methods, COSY (Yu et al., 2021) is for the cross-lingual understanding task, and we follow it to build a counterfactual dependency graph by maintaining the graph structure, and replacing each type of relation with a randomized one and randomly selecting a POS tag. RM/REP-CT (Yang et al., 2021) is proposed for sentiment classification tasks. We employ its self-supervised context decomposition and select the human-like counterfactuals using MoverScore.

For models with static word embedding, we initialize word vectors with 300-dimension GloVe embeddings provided by Pennington et al. (2014). For models with contextualized word embedding, we adopt BERT${}_{\text{base}}$ as the PLM. The embeddings for POS and dependency relation are initialized randomly and their dimension is set to 30. We adopt the Stanford CoreNLP  (Manning et al., 2014) as the syntactic parser and use semantic dependency parsing  (Dozat and Manning, 2018) to generate semantic dependency graph. We use SGD as the optimizer with a 0.9 decay rate. The L2-regularization coefficient $\lambda$ is 0.002 ²²2Please refer to Appendix for detailed information for datasets and baselines, as well as the hyper-parameter setting for three thresholds..

The comparison results for cross-sentence $n$-ary and sentence-level tasks are shown in Table 1 and Table 2, respectively. From these results, we make the following observations.

(1) Our CoCo model significantly improves the performance of all the backbones across three datasets. Specifically, CoCo outperforms two backbone approaches PA-LSTM and AGGCN by around 2-3 absolute percentage points in terms of accuracy on PubMed. It also improves F1 scores by 1-2 absolute percentage points on TACRED and SemEval. More importantly, our model boosts the performance of R-BERT on all three datasets especially on PubMed, which is impressive because it is a very strong backbone, and also because most baselines damage the performance of R-BERT.

(2) Our model achieves the state-of-the-art performance in terms of a counterfactual generator for RC. For example, CoCo outperforms the syntax-based model COSY and the best semantics-based model GYC by 1-3 absolute percentage points. COSY randomly replaces syntactic features which cannot flip the label and makes little sense for RC. CounterfactualGAN and GYC get a slight increase on PubMed with PA-LSTM and AGGCN, but their other results are unsatisfactory. The results of RM/REP-CT on three datasets fluctuate. On PubMed, some of them keep the same as the original ones and some of them decline, while on SemEval and TACRED they all decline. The poor results of the baseline counterfactual generators can be due to the lack of grammar constraint and the lack of entity-centric viewpoint. We will illustrate this in our case study.

Our model has two unique characteristics. Firstly, it utilizes both syntactic and sematic information. Secondly, it takes advantage of the graph topological property. We hence design two types of ablation study. One is performing one separate component only, i.e., SynCo or SemCo. The other is replacing our graph topology based intervention methods with other syntactic/semantic ones, i.e., tree edit distance (TED) (Zhang and Shasha, 1989) which measures the syntactic closeness between the candidate and the original text and BERT-attack (BA) (Li et al., 2020) which generates substitutes for the vulnerable words in a semantic-preserving way. We present the ablation results on three datasets in Table 4.

We find that a single SynCo or SemCo is already good enough to enhance the performance of the backbone. In addition, we observe that our single SynCo/SemCo outperforms the baselines in Table 1 and Table 2. Moreover, SynCo and SemCo have almost the same effects on the model. For example, the backbone PA-LSTM has an accuracy score 77.0 on PubMed_B. After SynCo/SemCo, its accuracy rises up to 79.4/79.7, showing a similar 2.4/2.7 absolute increase. These results demonstrate that both SynCo and SemCo contribute to our model, and their combination CoCo is more powerful.

The replacement of SynCo with TED and SemCo with BA hurts the performance. For example, on SemEval with AGGCN as the backbone, TED results in a 0.5 F1 decrease (SynCo 86.3 vs. Syn-TED 85.8). On TACRED with R-BERT as the backbone, BA brings about a 0.3 (SemCo 70.3 vs. Syn-TED 70.0) absolute decrease. The reason might be that TED only considers the causal words from the whole syntactic tree while our SynCo exploits the topological structure and syntactic feature of the words. Meanwhile, BA is unable to capture casual associations when generating adversarial samples since it only replaces the words with similar ones generated by BERT.

To evaluate the model robustness, we perform the generalization test on ACE2005 dataset by using the training and test data from different domains. Following the settings for fine-grained RC tasks (Yu et al., 2015), we use the union of the news domains (nw and bn) for training, and hold out half of the bc domain as development data, and finally evaluate on the remainder of bc, cts, and wl domains. We choose two best baselines GYC and COSY for comparison.

As can be seen in Table 5, our proposed CoCo model enhances the performance of all backbones on the combined CAD and the original (Ori.) data, i.e., PA-LSTM (+4.1%, +3.7%, +2.3% ), AGGCN (+1.9%, +2.2%, +1.8%), and R-BERT (+0.5%, +1.0%, +0.7%) on three different target domains. It is worthy of noting that our improvements over three backbones (including R-BERT) are statistically significant on all generalization tests. In contrast, two baselines GYC and COSY only get very small improvements or even decrease the performance of backbones in some cases. All these clearly prove that our method can enhance the robustness of backbone methods with the generated counterfactuals.

To have a close look, we select two samples from SemEval for case study and present results by different models in Table 3. RM/REP-CT deletes the important preposition ‘from’ to change the label and results in an incomplete sample. CounterfactualGAN employs an adversarial attack method. Its generated ‘sleep’ is ungrammatical and the sample is low-quality as it is inconsistent with the flipped label. GYC and CLOSS generate the counterfactual with a specified target label and the substitution tends to be common words in the specified class like ‘community’, which makes the sentence unreadable. Moreover, the word ‘almost’ identified by GYC is out of the scope of two entities and is not a causal feature.

Overall, the results by other methods are very uncontrollable, and it is easy for them to generate sentences with semantic or syntactic errors. In contrast, our model not only identifies the correct causal features, but also produces human-like counterfactuals, which can force the classifier to better distinguish between causation and confounding.