An Empirical Study of Causal Relation Extraction Transfer: Design and Data

Causality is essential for understanding the relationships between variables, enabling us to identify actual cause-and-effect dynamics rather than mere correlations. This understanding is crucial in fields such as medicine (Bollegala et al., 2018), where identifying causal factors can lead to effective treatments, and finance (Mariko et al., 2021), where it informs decisions with wide-reaching impacts. In technology and AI, causality improves the reliability and interpretability of predictive models. Establishing causality leads to better outcomes across various domains, often requiring the extraction of causal relationships from free-text sources like scholarly articles and news reports, which is considered a key task in the field of Natural Language Processing (NLP).

Although the extraction of causal relationships may have direct applications to many downstream tasks (Zhou et al., 2024), a major goal of causal relation extraction is to construct knowledge graphs that can facilitate downstream uses. ConceptNet (Speer et al., 2017) and CauseNet (Heindorf et al., 2020) are examples of this; however, the rule-based extraction methods used for these datasets limit the type of causal relations present in the knowledge graph. CauseNet, for example, identified causal sentences in Wikipedia and ClueWeb12 with syntactic templates then used a neural sequence tagger to label causes and effects in the identified sentences. Though this method was able to create the largest available causal knowledge base, it could be missing many implicit causal relations. In this sentence from CauseNet, “Insomnia is often caused by fear, stress, and anxiety.” the relationship is made explicitly clear by the highlighted marker “caused”. Other markers (e.g., “leads to”, “associates”, “trigger”, etc.) comprise the source sentences underlying CauseNet. By contrast, take “The clock struck twelve with a loud chime that made me jump.” from (UzZaman et al., 2013). Causal semantics are present, but the lack of a clear marker means such sentences would have been excluded from CauseNet. This under-representation of implicit sentences could bias knowledge graphs and knowledge bases when rule-based extraction is used on a large scale.

We have yet to see a large-scale causal knowledge graph extracted using both implicit and explicit source sentences. This paper addresses the root cause of that problem; causal relation extraction models are trained on disparate benchmark datasets that vary significantly in lexical composition and annotation style. Recent efforts have employed increasingly complicated cause/effect/relation annotation schemes (Li et al., 2021), which produce over-specified models that are not well suited for open-domain extraction. To address this, our analysis examines transfer across six unique causal relation datasets that span varying domains, annotation styles, and implicit/explicit causality makeups. An empirical analysis of architectural components including choice of embedding, recurrent unit, and usage of conditional random fields (CRF) is conducted. In particular, we see transformer-based contextual word embeddings (Devlin et al., 2018) with specialized pre-training datasets, such as BioBERT (Lee et al., 2020) and RoBERTa (Liu et al., 2019), work best across all six target datasets. Experiments further reveal that across domains, common architectures for relation extraction, BiLSTM, and Conditional Random Field (CRF) (Li et al., 2021) are perhaps unnecessary for an open domain causal relation extraction model as these components barely improve performance at the expense of speed and generalizability over a simpler BiGRU. Our current focus is on extracting intra-sentence causal relations, as they are easier to identify and evaluate using existing benchmarks, providing a foundation for more complex tasks. While newer models like Mistral 7b, Llama, and GPT may offer better performance, this work is limited to BERT and Flair embeddings due to their balance of computational efficiency and robust performance in domain-specific tasks. These embeddings are also well-studied and integrated into existing NLP pipelines, making them a practical choice for understanding causality.

The introduction of our $F1_{phrase}$ metric allows for a fair comparison of transfer performance by recognizing the target of causal relation extraction as the cause/effect phrase subtrees of the dependency tree. The application of this metric reveals that many target datasets can benefit from data augmentation, regardless of domain and annotation styles. Using $F1_{phrase}$ and the best transfer models, we further see that improved transfer performance relates to implicit/explicit sentence makeup of a dataset and noun phrase coverage training size, rather than training data size. Causal relation extraction from the web sources is a challenging task, due to noise in the open domain and lack of consistency across viable training datasets. Through the lens of phrasal evaluation, our results imply that models like BioBERT-BiGRU are particularly well suited to causal relation extraction in this setting since they can best incorporate disparate data domains and annotation styles. Large causal knowledge bases have relied on rule-based methods for localizing causal relations (Heindorf et al., 2020), creating a bias for sentences with explicit expressions of causality. This work shows that BioBERT-BiGRU, trained on a collection of disparate data and evaluated with $F1_{phrase}$ offers a path towards richer extraction of causal knowledge from noisy web-based sources. Our work contributes:

1. (1)

   An empirical evaluation of causal relation extraction models across a combination of datasets showed the high performance of specially pre-trained BERT embeddings in the transfer setting, as well as the unnecessariness of Bi-LSTM and CRF layers.

2. (2)

   The $F1_{phrase}$ metric which facilitates the empirical evaluation analysis.

3. (3)

   Furthermore, we conduct additional qualitative analysis to determine what data characteristics contribute to transfer performance. We show that the proportion of implicit causal sentences in the data improves transfer, more so than merely increasing dataset size. Our code files and supplementary materials will be made available after the review process.

An early example of a machine learning application for causal extraction is Blanco et al. (2008), modified in (Zhang et al., 2023) where Bagging and Decision Trees were used to classify cause-and-effect information. A study by Egami et al. (2018) provides a conceptual framework for text-based causal inferences, generalizing a codebook function to fit several models and linking higher-dimensional text to lower-dimensional representations for making inferences and estimating treatment effects. While machine learning methods are tried-and-true, they cannot learn incrementally in real time and require supervision, limiting their usefulness in real-world applications.

Deep learning applications have been used in recent advances in causal extraction. For instance, (Roemmele and Gordon, 2018) used the Choice of Plausible Alternatives (COPA) framework to predict causally related events in stories. CausalTriad a minimally supervised method for determining causality that relies on discourse connectives and focused distributional similarity techniques was created by (Zhao et al., 2018). Transferred embedding has been employed in more recent research, such as the BiLSTM-CRF model developed by (Li et al., 2021). Tests revealed that models with transferred embeddings performed better than those without. These models included Flair, ELMo, and BERT. As demonstrated by the BERT architecture utilized by (Khetan et al., 2022), transformer-based architectures have also excelled in causal extraction. Training on data other than the target data provided results that were equivalent to or better than training and testing on the target data. We distinguish our work by using transferred embeddings across many architectures and testing a larger number of datasets. Our study also evaluates general-purpose architectures without assuming domain-specific information.

Not much work has investigated transferable causal relation extraction models specifically. Moghimifar et al. (2020) combined a graph convolutional network for encoding dependency trees (GCE) with an adversarial domain adaptive module (ACE) for cross-domain causality classification and relation extraction. This approach is a step towards general-purpose, large-scale causal relation extraction, showing increased transfer performance across domains. However, it requires the model to explicitly learn different domains, which might not always be available in open-domain causal relation extraction.

We focus on model design and data augmentation for sentence-level causal relation extraction since most recent efforts in mining causal knowledge from the web sentence level (Moghimifar et al., 2020; Heindorf et al., 2020). Unlike previous work, however, we distinguish sentences that express causality implicitly or explicitly and make sure to analyze both. The results from sentence-level explicit causal relation extraction could inform future work on multi-sentence causal relation and even document-level causal relation extraction.

Causal-TimeBank (Mirza et al., 2014) consists of causal annotations of the TempEval-3 corpus (UzZaman et al., 2013), which consists of news articles from the web. For uniformity across other datasets, we only consider sentence-level relations, although Causal-TimeBank also contains document-level relations.

CauseNet (Heindorf et al., 2020) is a large graph of explicit causal relations and supporting sentences from ClueWeb12 and Wikipedia, with a high-precision subset that we utilized. For our purposes, we subsampled a collection of 5,000 sentences containing explicit markers such as ”cause,” ”caused,” and ”causing” (CauseNet-cause). Another subsample from CauseNet includes 5,000 sentences without variants of the ”cause” marker (CauseNet-noncause) but with other explicit causal markers like ”leads to” and ”due to.”

The FinCausal2020 dataset (Mariko et al., 2021) is a benchmark for the detection and extraction of causal relations in financial text. It was mined from financial news articles from various economics and finance websites. For our purposes, FinCausal was limited to relations contained in single sentences.

MedCaus (Moghimifar et al., 2020) is a dataset consisting of causal sentences mined from ”medical articles” in Wikipedia that matched certain seed patterns. While we found that many sentences in this dataset are medical or biological, some general sentences (E.g., “The eastern water is saltier because of its proximity to Mediterranean Water”) seem to be captured as well, so we have labeled it as a ”General” domain dataset.

SemEval 2010 Task 8 (Hendrickx et al., 2019) is a multi-way classification dataset. It has widely been used as a general domain benchmark for evaluating relation extraction tasks. Causal relation extraction literature has particularly focused on the Cause-Effect relations in this data which represent 12.4% of the entire dataset. We use only the Cause-Effect relations for our analysis.

A first step towards open-domain, large-scale causal relation extraction is finding a model that works well across different datasets. We explore this through experiments on embeddings, recurrent units, and conditional random fields (CRFs), analyzing components used by high-performing models like SCITE (Li et al., 2021). Since most models are developed on a single dataset, comparing them in the transfer setting can be misleading. For example, SCITE’s (Li et al., 2021) annotation scheme is specific to SemEval, so some unifying simplifications should be made across datasets for fair comparison. Yang et al. (2022) analyze recent model performance outside the transfer setting. Our experiments cover these models using combinations of tested components (e.g., SCITE is Flair-BiLSTM-CRF).

We are focused on extracting causal linkages on a sentence level, emphasizing intra-sentence relationships. ”Single Sentence-level” described in this paper involves locating and identifying causal linkages in one sentence. Our reason is that intra-sentences are label consistent across various datasets and our attempt to evaluate our model on multiple datasets justifies this focus. Furthermore, this allows us to take advantage of robust assessment measures and available annotated datasets to systematically improve our models before further expanding our scope to more complex inter-sentence causal relations, which involves identifying causal connections spanning multiple sentences and possibly entire documents.

Sentences for all datasets were tokenized at the word level (Akbik et al., 2019), dependency parsed (Honnibal and Montani, 2017), and each token was labeled as C (cause), E (effect), or O (other). Each dataset was split into 70%-30% train-test subsets. All models used an embedding layer, followed by a recurrent NN (RNN) layer, and either a CRF or linear layer for final prediction. Cross-entropy was used as the loss function. The RNN hidden size was fixed at 256 (empirically determined). Early stopping was used in training with a minimum of ten epochs. For sections 5 and 6, twelve training epochs were used as this was found to be consistently sufficient. An ADAM optimizer with a learning rate of 0.001 and a minibatch size of 16 was used.

The larger size of the MedCaus dataset, which contains 8,682 sentences, maybe a contributing factor to its effective transfer. To explore this hypothesis, training was conducted using progressively larger subsets of MedCaus, starting with 10% and increasing incrementally by 10% up to the full dataset. The outcomes of these trials suggested that expanding the training subset beyond 40% of the MedCaus dataset does not significantly enhance the $F1_{phrase}$ score for the majority of datasets, with the notable exception of the CausalTime Bank dataset. Additionally, there appears to be no negative impact from using larger amounts of data; however, it seems to be a superfluous and potentially inefficient use of computational resources, considering the plateau in average performance observed when training with 40% to 60% of the data. This indicates that simply increasing the size of the dataset used for transfer does not necessarily correlate with better model performance.

Another consideration for causal transfer is the composition of implicitly and explicitly causal sentences in the training data. (Chen, 2023) While explicit relations like those in CauseNet are more readily available, a relation extraction model trained on explicit relations alone might merely learn the syntactic templates that generated CauseNet, and therefore fail to generalize to implicit relations. To test this, the BioBERT-BiGRU model was trained on implicit and explicit subsets of MedCaus, at accumulating larger sizes, then tested on SemEval, Causal-TimeBank, and CauseNet. Training with only explicit sentences from MedCaus leads to better performance on the CauseNet datasets since these are all based on explicit syntactic templates. For SemEval, equal-sized subsets of explicit and implicit sentences seemed to provide equal transfer performance. For CausalTimeBank, training with implicit sentences was better. Since CausalTimeBank is over half implicit (54.7% over SemEval’s 34%). Thus, the implicit/explicit composition of the training data should reflect the composition of the target data in the transfer setting.

Our experiments show that a BioBERT-BiGRU sequence tagger is a strong candidate for large-scale causal relation extraction in the open domain. BioBERT’s pretraining with dynamic masked language modeling seems to enhance model performance, suggesting that dynamic masking for causal phrases is worth investigating. Causality-specific pretraining/fine-tuning, as demonstrated by Khetan et al. (2022) and Li et al. (2021), is further evidenced by our results. Future research could explore pretraining for open-domain relation extraction.

While our current study focuses on intra-sentence causal relation extraction, this approach allows us to fine-tune our models and metrics in a more manageable context. Addressing intra-sentence relations first ensures that our methods are well-grounded and effective within simpler boundaries. This step-by-step approach is crucial for developing the sophisticated techniques required for inter-sentence causal relation extraction, which will also be the focus of future research.

The $F1_{phrase}$ metric reveals that combining disparate data from varying annotation schemes and domains can significantly improve transfer performance in causal relation extraction. No single dataset is likely to be the key to large-scale causal relation extraction, but combining diverse data has proven beneficial. Future work could benefit from combining even more causal datasets than we used here. Our results from section 6 show that data size matters up to a point, but increasing training data beyond a few thousand sentences does not always improve transfer performance. Instead, attention should be given to the composition of implicit and explicit sentences in the dataset.

This study did not consider advanced large language models like GPT, Mistral-7B, and Llama. Although these recent models may have the potential to offer enhanced performance, our study is confined to using BERT and Flair embedding, because we considered the optimal balance between computational efficiency and robust performance in domain-specific tasks. Moreover, BERT and Flair’s embedding are well-documented and widely integrated into existing NLP pipelines, making them a practical and reliable choice for investigating causality in this context.

Also, this study was limited to sentence-level causal relation extraction, when inter-sentence and document-level relations offer a rich source of causal information for open-domain applications. While some of the datasets used, such as FinCausal and CausalTimeBank, do contain inter-sentence relations, we limited our analysis to sentence-level relations for compatibility with other datasets.

Although this study includes a range of datasets with varying domains and annotation styles, the selection may still not fully capture the diversity of causal relations present in real-world texts. Additional datasets from different domains and with different annotation styles could further validate the findings. The models and methods were primarily tested on specific benchmark datasets. While efforts were made to ensure transferability, the generalizability to other, non-tested datasets and domains cannot be assumed and therefore, remains uncertain.

While our goal was to use a simple model like BioBERT-BiGRU which showed promising results, this study did not explore more advanced architectures that might offer improved performance or interpretability. It is possible in the future to explore the trade-offs between model complexity and performance in more detail.

We conducted an empirical study of model design and data augmentation strategies for building causal relation extraction models that transfer effectively. We showed BioBERT embeddings paired with a relatively simple BiGRU, is a strong performing base causal relation extraction model across varying datasets. To better facilitate data transfer experiments, we introduce $F1_{phrase}$, a variant of the F1 measure that prioritizes noun phrase localization. This revealed that longer annotations transfer well, cross-domain data augmentation is consistently beneficial, and implicit/explicit dataset composition trumps dataset size in terms of transfer performance.

1. (1)

   All data used in this experiment is open-source and most are licensed by Creative Commons. The others had their individual licensing

2. (2)

   Non-sensitive data was used during this research and as such, no privacy violations were made.

3. (3)

   There was clear documentation of methodologies and algorithms to enable transparency, reproducibility, and accountability.

4. (4)

   For each of the datasets, we followed their individual terms of usage

1. (1)

   The Causal-TimeBank corpus is provided under the Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license. Click here for more details

2. (2)

   CauseNet is available for non-commercial academic use under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license. Click here for detailed terms

3. (3)

   The FinCausal2020 dataset is provided under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license. Click here for detailed terms

4. (4)

   The MedCaus dataset, derived from Wikipedia and medical articles, is available under the Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

5. (5)

   The SemEval 2010 Task 8 dataset is available for research purposes under the Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.