The Veracity Problem: Detecting False Information and its Propagation on Online Social Media Networks

The creation and spread of false information (aka fake news) is rapidly increasing, with online social media networks (OSMN) such as X¹¹1Formerly known as Twitter, Facebook, and Weibo contributing to its rise. False information can, often unbeknownst to them, manipulate how individuals responds to topics such as health, politics, and social life. One such example is a tweet in 2013 that claimed an explosion injured the US president, resulting in a loss of $130 billion in stock values (ElBoghdardy, 2013). Further, some research suggests false information is connected to election outcomes e.g. the 2016 US presidential elections (WTOE, 2016) and 2020 UK Brexit vote (Dennis et al., 2020).

OSMN enables anyone to access the latest information in a variety of formats (i.e. news articles, blogs) and sources (i.e. news outlets, public figures). The expansion of new formats and sources has decentralised the distribution of information, removing centralised control over its authenticity. This unregulated creation and spread of information places the onus of validating the truthfulness (or falsehood) of the information on the individual. However, an individual’s ability to identify falsehoods objectively is influenced by factors such as confirmation bias, which makes one trust and accept information that confirms their preexisting beliefs (Nickerson, 1998), and selective exposure, which is when one prefers to consume information that aligns with their beliefs (Freedman and Sears, 1965). External factors such as the bandwagon effect, which motivates one to perform an action because others are doing it (Leibenstein, 1950), and the validity effect, where one believes information after repeated exposure (Boehm, 1994) play a critical role.

A recent narrative which exemplified coordinated behaviors and the bandwagon effect is the Sydney Bondi Junction Stabbings (Nguyen and Workman, 2024). An image of the attacker circulated on social media, resulting in widespread misidentification of his heritage, religion, and identity. This false information initiated from small accounts, was amplified by verified accounts, and reported on by mainstream news. The coordinated behaviour exhibited both malicious and benign propagation patterns.

Detecting false information is crucial in alleviating the burden placed on individuals to verify the truthfulness of information, and the resulting consequences of this burden. There are three main streams of research key to this work which aims to address the problem of false information. These relate to (1) false information detection methods; (2) detection of coordinated behaviours; (3) cross-platform analysis of false information propagation.

False Information Detection Methods: To overcome the limitations of human-driven false information detection (e.g. scalability and bias (Tchakounté et al., 2020; Wallace et al., 2022)), AI-based decision support techniques have been developed. Most techniques fall into two types: content-based methods and context-based methods. Content-based methods are built using the information contained within the document, such as words and images (Zhang et al., 2024; Ghanem et al., 2021; Shu et al., 2019; Wang et al., 2024; Jin et al., 2022). While these methods can preemptively forestall the dissemination of false information, they are prone to adversarial manipulation of linguistic and stylistic features to evade detection (Jin et al., 2022). On the other hand, context-based methods use the information on the document’s propagation over OSMNs and the users who engage with it (Soga et al., 2024; Chowdhury et al., 2020; Bing et al., 2022; Min et al., 2022; Ruchansky et al., 2017). These methods are independent of linguistic and stylistic features and knowledge bases, but rely on information generated when a document is propagated on an OSMN, meaning false information is already spread, along with its related negative consequences. Although some hybrid models such as dEFEND (Shu et al., 2019) and CSI (Ruchansky et al., 2017) have been proposed, they use limited sources of information, making them prone to loss of reliability and effectiveness.

Detecting Coordination: There are two approaches to detect coordination, the first considers the similarity of actors (Dennis et al., 2020; Nizzoli et al., 2021). For example, (Nizzoli et al., 2021) proposed a network-based framework using iterative community detection to estimate the extent of coordination among actors. The second method considers the temporal synchronisation of actors. For example, (Zhang et al., 2021) incorporated a neural point process to identify synchronised behaviours. Moreover, most detection works treat the threshold of coordination as a distinct boundary within detection methods. However, it is a spectrum and varies for each community of actors (Giglietto et al., 2020). Methods that attempt to distinguish intent have found an inability to differentiate (Pacheco et al., 2021). While (Fazil and Abulaish, 2020) assumed all coordinated behaviours indicated inauthenticity, no other works were found that identify the intent of an actors when engaging in coordinated behaviours.

Existing Datasets: The available datasets for developing detection methods vary largely in terms of features for each document. For example, FakeNewsNet (Shu et al., 2020) provides both context and content data, while LIAR provides only content data. This means models developed on one dataset are often not transferable to other datasets. Additionally, most datasets include context data from one OSMN, making cross-platform analysis challenging. However (Ma et al., 2016) does include context data from both X and Weibo. The labelled datasets use manual labelling of documents and often require the manual identification of items corresponding to the document. Consequently, these datasets are static, and often out of date, leading to domain drift and diminished model performance in new domains.

The aim of this research is to develop methods for the effective detection of false information on social media, and the analysis thereof, of its propagation.

The first research gap identified pertains to the detection of false information propagated on OSMNs. The existing methods consider the context and content features in isolation which provides a uni-dimensional view on a multi-dimensional concept. For example, in one dimension (e.g. content) an document may be labelled as true while in another (r.g. context), that same document may be labelled false. Further, the available data for a document and performance of a model can be impacted by the temporal and dynamic aspect of real world data. For example, consider a document observed at two times ($d_{1}.t_{1}$, $d_{1}.t_{100}$) where $d_{1}.t_{1}$ has 3 engagements and $d_{1}.t_{100}$ has 500 engagements. A model based on context data (e.g. number of user engagements) may assign a falsehood probability to $d_{1}.t_{1}$ based on significantly fewer data points compared to $d_{1}.t_{100}$, thus affecting the quality of the prediction. This motivates the research question: RQ1 How can a model agnostic framework be developed to improve explainability and accuracy of existing false information detection techniques without incurring excessive computational overheads?

The second research gap identified is the detection of coordinated behaviours. The few works that detect coordination campaigns or actors are built on static datasets covering a limited range of topics and campaigns. However, no works look at identifying coordination on streaming data or without predefined campaigns to search for. Furthermore, no work has distinguished the intent of coordinated behaviours. This motivates the research question: RQ2 How can we identify actors working in coordination to maliciously amplify and manipulate the creation and propagation of information?

The third research gap is a lack of consideration of cross-platform interactions in detecting false information. There are no works to date which merge the context data from multiple OSMNs with the content data for each document to enhance the data available for analysis. Further, few works analyse whether the existing detection models are suitable for the different OSMNs. This motivates the research question: RQ3 How might the propagation of false information change over different online social media networks, and can existing detection methods account for the differences?

The overarching aim of this work is addressed through the proposed methodology summarised in Figure 1, which outlines the approaches to solve the three research questions. The first research question RQ1 - Detection framework), will be solved through a three-part framework. This framework takes a document and its associated features as input, producing a probability of falsehood and an explanation as the output. (1) Base modeller: This component employees various false information detection base models (BMs) to generate probabilities of falsehood $(p)$. (2) Aggregator: This component combines the various probabilities $(p)$ into a final falsehood probability $(Prob)$. A novel dynamic aggregation method will be developed which accounts for the reliability of features the prediction is based on. For each instance of a document, a reliability weight $(r)$ will be assigned the the probability $(p)$. (3) Explainer: This component provides a tiered explanation of the contributions to the final probability $(Prob)$. Each tier offers greater detail of the BMs and the reliability factors.

The second research question RQ2 - Coordinated, solution is a two-part detection method which will first identify actors working in coordination and then determines the intent behind their behaviours. (1) Coordinated detector: This component identify the actors contributing to the a coordination campaign using isolated user characteristics (e.g. URLs, user mentions, hashtags) rather then explicit network structures (e.g. follower network, retweet information) which are often incomplete or unavailable. The pre-training and fine-tuning of models to reduce the required training dataset will be incorporated. (2) Intent detector: This component employs contrastive learning to identify an actors intent. That is, whether the actors harmful intent with malicious behaviours, or benign where an actor is engaging in coordinated activities without maliciousness for example as part of the bandwagon effect.

The third research question RQ3 - Dataset creation, solution is a three-part process to incorporate a new document into a multi-OSMN dataset. The dataset once created can enable the analysis of cross platform interactions. (1) Labeller: This segment uses an API to obtain a label for each document from a variety of manual factchecking sites. (2) Data Pull: This segment uses an API which collects all related context data from various OSMNs for each document. This novel method will enable the identification of items on a OSMN without prior manual identification. (3) Dataset: This segment will merge multiple context features and content features for one comprehensive data package for each document. Further, this segment employs entity linking to merge a user’s behaviour from multiple OMSNs.

Preliminary experimentation for the development of the Aggregator (RQ1) have returned favourable results for the use of a dynamic and adaptive weighting scheme. A brief outline of the experimental setup and results are presented below.

In this work, we identify three critical research gaps and propose solutions in the detection and mitigation of false information on social media. The proposed framework, once implemented, will improve false information detection by leveraging multiple base models in a dynamic manner. This will include developing novel methods for dynamic data aggregation based on reliability and providing hierarchical tiered explanation. The second proposed method involves a novel algorithm to identify both coordinated actors and their intent. Finally, the proposed dataset aims to provide a comprehensive view of a document and enable interrogation of cross-platform interactions.

Future research directions for the first research question include exploring additional reliability factors across a broader range of base models to determine whether there is a peak or a plateau in performance as reliability factors change. Another direction is to test the proposed framework on different online social media networks to assess its versatility and determine whether it is truly model-agnostic.

I would like to thank my supervisors Dr Michael Bewong, Dr Selasi Kwashie, Professor Md Zahidul Islam, and Associate Professor Irfan Altas for their support and guidance. This work was supported by Charles Sturt University.