ChronoFact: Timeline-based Temporal Fact Verification

The spread of false information has reached alarming levels, undermining social trust significantly. One prominent category of false information involves temporal claims, which are statements that include time-specific elements, either explicitly (e.g., ”in 1953”) or implicitly (e.g., ”before [another event]”). The complexity of verifying these claims escalates with the number of events mentioned. Effective verification must assess not only the veracity of each event within its temporal context but also understand the relationships between these events, especially their chronological order.

Existing research has largely focused on verifying the veracity of individual events within claims, and overlook the chronological order of these events Barik et al. (2024); Qudus et al. (2023). This oversight can undermine the effectiveness of fact-checking systems, particularly when dealing with complex narratives where the sequence of events is crucial for determining the truth. Understanding the chronological order of events is hindered by the absence of the explicit temporal cues. The complexity increases when events overlap or recur, complicating the task of establishing a clear and accurate timeline for the verification process.

To overcome the limitations in existing temporal fact verification methods, we introduce ChronoFact, a framework that systematically identify events from both claim and evidence to construct a coherent timeline for temporal fact verification. ChronoFact examines the relationships between claim and evidence events at three levels: event-level, token-level, and time-level and learns a model to predict the veracity of each claim event and their chronological accuracy.

We develop a new dataset called ChronoClaims for timeline-based fact verification. This dataset encompasses complex claims involving multiple, related events that unfold over time with both implicit and explicit temporal expressions. These features are lacking in current datasets like T-FEVER, which mainly consist of single-event claims, and T-FEVEROUS, which, despite its more complex claims, often has events that are not chronologically related.

There has been a stream of research on evidence-based claim verification to assess whether the evidence sentences support or refute the claim Stammbach and Neumann (2019); Soleimani et al. (2020). Techniques like GEAR Zhou et al. (2019), KGAT Liu et al. (2020), and DREAM Zhong et al. (2020) model claim and evidence into a graph using Graph Attention Network to facilitate information propagation of information between them. CGAT Barik et al. (2022) enhances this by incorporating commonsense knowledge from ConceptNet to enrich the contextual representation. These works do not consider temporal information.

Only a few studies have incorporated temporal information in the claim verification process. NMT Mori et al. (2022) verifies economic claims against time series data in tabular format by translating the claims into Datalog rules to query the tabular evidence. TemporalFC Qudus et al. (2023) focuses on verifying claims represented as tuples against a knowledge graph. It utilizes temporal graph embeddings to determine the validity timing of underlying triples. However, these works are limited to structured data, hindering its applicability to natural language claims and evidence.

Allein et al. (2021) capture the temporal relevance of evidence through a re-ranking process based on the proximity of publication dates between the evidence and the claim. ITR Allein et al. (2023) extends this by assigning publication dates of claims and evidence to fixed-size time buckets for temporal reasoning. However, it ignores implicit temporal cues or chronological order in the claim and evidence.

TACV Barik et al. (2024) decomposes claim and evidence sentences into events and employ temporal-aware representation encoder to retrieve evidence that are both semantically and temporally related to the claim. It utilizes GPT for temporal reasoning to verify individual claim events. Unlike TACV, our approach also evaluates the chronological timeline accuracy between the claim and evidence, providing a more thorough verification of complex claims.

A related area of research, question answering (QA), has also begun to incorporate temporal information. FAITH Jia et al. (2024) focuses on implicit temporal reasoning for the temporal question answering task by generating intermediate questions with explicit temporal information. This method uses heterogeneous sources to improve the completeness of evidence retrieval and employs a general-purpose QA system to respond to the question. While it is common to utilize QA in the claim verification process Pan et al. (2023), they do not consider the chronological order of events. In contrast, our work focuses on verifying complex claims that require timeline-based reasoning.

ChronoFact follows the typical automated claim verification which involves collecting relevant evidence from credible sources and assessing the claim’s veracity based on the evidence. Figure 2 shows the key modules in the framework.

$\bullet$ token-level attention score $\alpha_{ij}$: is the average of cosine similarities between all pairs of tokens in $c_{i}$ and $e_{j}$.

$\bullet$ event-level attention score $\beta_{ij}$: is derived from the cosine similarity between the $H_{CLS}$ representations of claim $c_{i}$ and evidence $e_{j}$.

$\bullet$ time-level attention score $\gamma_{ij}$: is computed from the cosine similarity between the mean pooled date representations in $c_{i}$ and $e_{j}$.

The final multi-level attention score between $c_{i}$ and $e_{j}$, denoted as $\omega_{ij}$, is the average of the event-level, token-level, and time-level attention scores. These computed attention scores are then employed to predict the label of each event in the claim, assess the accuracy of the claim’s chronological order based on the evidence timeline, and evaluate the claim’s overall veracity. Note that we considered dynamically learn the weights of each attention level. However, our experiment indicated that this approach did not improve performance, leading us to adopt the average weighting method instead.

We use $\tanh$ for its bounded output range of $[-1,1]$, which aligns with our interpretation of relevance as a continuous spectrum ranging from negative to positive. We sort the evidence events based on their relevance to the claim and obtain the top-$k$ events with the highest score $r^{e_{j}}$. These top-$k$ evidence events, along with the claim events, are then input into GPT to be reordered according to their chronological sequence. Given the limited number $k$ of events, this reordering process requires minimal time and resources, making it practical for real-world applications.

The reordered sequence of claim events, denoted as $seqC=H^{c_{1}}_{CLS}\oplus\cdots\oplus H^{c_{n}}_{CLS}$ is passed to a Bi-LSTM to capture and embed the chronological order into the model. Similarly, the reordered sequence of evidence events, weighted by their relevance scores is given by

This sequence is passed to a second Bi-LSTM. The outputs from the two Bi-LSTM, denoted as $o^{C}$ for claim events and $o^{E}$ for evidence events, are then fed into two fully connected layers followed by softmax to obtain the distribution $z^{o}$

where $z^{o}[0]$ is the probability that the chronological order of the claim events is supported by that of the evidence events, and $z^{o}[1]$ is the probability that the chronological order refutes the claim’s timeline. The output of the chronological order classifier $y^{o}$ is the label with the highest probability.

We introduce a new benchmark dataset called ChronoClaims that is designed for enhancing the accuracy and complexity of timeline-based fact verification. Utilizing the November 2022 Wikidata snapshot Vrandečić and Krötzsch (2014), we preprocess and extract facts in the format <subject, relation, object, time_start, time_end>. To construct an evidence timeline, we organize all the facts that have the same subject in a chronological order. Then we randomly select $N$ facts from this timeline and use GPT to transform these facts into coherent sentences.

For generating sentences with implicit temporal information, we omit the time_start and time_end from the fact, and prompt GPT to craft sentences that subtly embed the temporal context. Then we use a cloze-style template¹¹1<sentence₁> and then <sentence₂> and then <sentence₃> to synthesize the sentences into a claim that preserves the chronological order. To generate claims that contradict the evidence timeline, we rearrange the order of these sentences. Finally, we use GPT to refine and rephrase the synthetic claim to make it more natural and fluent. Each claim is labeled as either SUP or REF, depending on whether it aligns or conflicts with the timeline.

We evaluate the quality of the ChronoClaims dataset by sampling 500 claims, ensuring equal distribution across different temporal expressions and event complexities (16% each type), and temporal categories (50% each category) to achieve representative coverage. Two human annotators are tasked with determining the labels of the generated claims based on the ground truth evidence timeline. The agreement rates are high, with 96.8% and 97% of the labels assigned by the annotators matching the labels of the generated claims.

We analyze the claims where the generated labels are different from the annotators, and discover that most of the discrepancies are due to errors in rephrasing For example, a claim ”Nasrallah Peter Sfeir worked as a Catholic priest, and then Nasrallah Peter Sfeir was educated at Saint Joseph University,” was rephrased to ”Nasrallah Peter Sfeir pursued his education at Saint Joseph University before becoming a Catholic priest,” which altered the original chronological sequence. To rectify this, we perform a second verification step using GPT to ensure that the semantic meanings of the original and rephrased claims remain consistent. Any claims where the meaning has been altered are discarded.

In total, we generated 40,249, 3,544 and 3,735 claims for the training, validation and test sets. Table 1 shows the detailed statistics of the ChronoClaims dataset.

We train the model using Adafactor with a batch size of 8 and a learning rate of 5e-5 for 5 epochs on each dataset. ChronoFact is trained to predict REF and SUP labels on ChronoClaims, as the dataset does not have NEI labels. For T-FEVER and T-FEVEROUS, the model is trained to predict SUP, REF, and NEI. We report the macro F1 score of the best performing model on the test set.

Finally, we present case studies to illustrate the importance of considering the chronological order in verifying temporal claims. Table 9 shows a T-FEVEROUS claim with three events: ”Yossi Yona studying for a PhD”, ”Yossi Yona become a professor at Ben-Gurion University”, and ”Yossi Yona join the left camp of Israel party”. The temporal context provided by the word ”before” is lost between the events ”Yossi Yona became a Professor” and ”Yossi Yona joined the Left Camp of Israel Party”, resulting in each individual claim event being supported by the retrieved evidence. However, the chronological order of the claim events is incorrect because evidence shows that ”Yossi Yona joined the left camp of Israel party” before ”became a professor”. ChronoFact addresses this issue by inferring the missing temporal relationships from the timeline of events, and correctly predicts the label as REF. In contrast, TACV evaluates the claim events independently and mistakenly predicts the claim as SUP.

Table 10 shows a claim from ChronoClaim dataset. The claim has five events, where the first event ”Davie was a member of Dundee United F.C” overlaps with the second event ”Davie joined Arbroath F.C.”. TACV does not accommodate the possibility of overlapping events and assumes that ”join” implies a full transfer or departure, and conclude that the evidence refutes the claim event c2. In contrast, ChronoFact can handle overlapping events and correctly conclude that c2 is supported by the evidence.

Table 11 shows another claim which involves a recurring event where Georgi Andonov was a member of Botev Polediv from 2002 to 2006, left, and rejoined in 2009. TACV fails to ecognize this recurrence, mistakenly using evidence of Andonov’s initial membership period (2002-2006) to refute the claim of his 2009 return. In contrast, ChronoFact organizes the evidence chronologically, recognizing Georgi Andonov’s return to Botev Plovdiv in 2009 after a hiatus, yielding a correct prediction.

We have introduced a framework for temporal claim verification that incorporates temporal reasoning based on the chronological order of events. By utilizing a multi-level attention encoder, ChronoFact effectively captures the relevance of evidence events to claim events. This enables ChronoFact to accurately predict to claim event labels and verify the chronological order consistency between claim and evidence events before determining the overall claim label. We have curated a new benchmark dataset that involves complex claims with multiple events that may overlap or recur. Extensive experiments on multiple datasets have shown that ChronoFact outperforms state-of-the-art models.