Decoding News Bias: Multi Bias Detection in News Articles

Over the last decade, rapid technological advancements and increased internet accessibility have led to a significant shift in how information is distributed, with a growing preference for online news articles over traditional print media (Krieger et al., 2022; Gonçalves et al., 2021). Today, news articles serve as a vital source of information for millions of people globally (Hamborg et al., 2019). They play a crucial role in spreading awareness, educating the public, and providing real-time updates on various events. As such, they significantly influence public opinion, shaping how people perceive societal, political, and global matters (Hamborg et al., 2019). This power to inform and influence makes the integrity and impartiality of news articles critically important.

However, news articles are often subject to various forms of media bias. These biases can skew the information presented, leading to misleading narratives and polarizing views. Biases in news articles may manifest in several forms, including political bias, where certain political parties or ideologies are unreasonably favored; gender bias, which portrays or criticizes individuals based on gender-related attributes; entity bias, favoring specific organizations or people; racial bias, where content favors or marginalizes certain racial or cultural groups and other kinds of biases. The presence of such biases can significantly distort the objectivity of news, misguiding readers and perpetuating misinformation.

Detecting biases in news articles is essential for promoting transparency and fostering critical thinking. Identifying these biases allows readers to better evaluate the information they consume, making informed decisions rather than being influenced by skewed content. Moreover, news article providers can benefit by improving their content’s credibility and gaining the trust of a wider audience. By highlighting biases, they can maintain higher ethical standards and work towards providing more balanced and objective reporting.

In recent years, Large Language Models (LLMs) have demonstrated impressive capabilities in understanding and generating natural language (Naveed et al., 2023). These advancements have opened new possibilities in analyzing and processing text at an unprecedented scale and accuracy. However, leveraging LLMs specifically for detecting biases in news reporting remains relatively underexplored.

This study make following key contributions:

• •

  Extending the Scope: We broaden the scope of the bias detection problem by identifying and analyzing different types of biases that may occur in news articles.

• •

  Dataset Annotation Method: We introduce a preliminary method for dataset annotation using LLMs, recognizing that further research is required to thoroughly assess the reliability and consistency of LLM-driven annotations.

• •

  Experimental Results: We experimented with different transformer based models on the curated dataset to assess their effectiveness in identifying news biases, providing a comparative analysis of existing methods.

The rest of this paper is structured as follows: Section 2 presents a detailed review of the related works in bias detection and the use of LLMs for natural language tasks. Section 3 discusses our methodology for bias detection, including how we build and annotate the dataset. Section 4 details the experiments conducted and the corresponding results derived from the techniques applied to the dataset. Section 5 offers the concluding remarks of the study and lastly, we address the study’s limitations and explores potential future research directions.

Recent studies have predominantly focused on detecting political bias and media bias in news articles. Media bias, particularly bias by word choice and framing, has been a central topic of research. For instance, Krieger et al. introduced DA-RoBERTa, a domain-adaptive model that detects media bias, particularly bias by word choice, achieving state-of-the-art performance in detecting biased language through a domain-adaptive pre-training approach. Similarly, Gaussian Mixture Models (GMM) have been employed to assess article-level bias by analyzing sentence-level features, such as bias frequency and bias sequence, effectively capturing the probabilistic nature of bias (Chen et al., 2020).

Further research has explored linguistic and context-oriented features to identify subtle word-level bias in media leveraging demographic insights of annotators to enhance bias detection (Spinde et al., 2021). Additionally, Aggarwal et al. have gone beyond mere detection and proposed systems to assess the short-term impact of media bias on public opinion, as seen in research analyzing biased reporting in the context of Twitter posts by Indian media outlets.

Political bias detection research has largely aimed at uncovering partisan leanings in news content. For example, a framework has been used to analyze the framing and structure of news across sources, showing improved performance over traditional models (Nadeem and Raza, 2021). Another method incorporated headline attention mechanisms to detect bias, enhancing accuracy with neural networks (Gangula et al., 2019). Additionally, adversarial media adaptation has focused on emphasizing content over source bias, improving the prediction of political ideologies in news articles (Baly et al., 2020).

In recent years, there has been a growing interest in incorporating Large Language Models (LLMs) into bias detection systems to enhance their capabilities. One prominent example is BiasScanner, an application that leverages a pre-trained LLM to detect biased sentences in news articles and provides explanations for its decisions (Menzner and Leidner, 2024).

One study explored GPT-4’s ability to classify political bias from web domains, showing strong correlation with human-verified sources like Media Bias/Fact Check (MBFC). While promising for scalable bias detection, GPT-4 abstained from classifying less popular sources, highlighting the need for combining LLMs with human oversight (Hernandes, 2024). Another study used GPT-3.5 for annotating politically biased news articles, blending LLMs with expert-driven rules for identifying bias indicators (Raza et al., 2024).

Moreover, there are a few studies which have incorporated LLMs in detecting fake news (Boissonneault and Hensen, 2024; Teo et al., 2024; Jiang et al., 2024). While fake news detection and bias detection seem similar, they address distinct issues. Fake news detection focuses on identifying completely false or misleading content designed to deceive, using fact-checking or external validation. In contrast, bias detection uncovers partiality or slant in how factual information is framed.

However, while these effective in specific bias detection tasks, there is still a gap in exploring diverse types of biases. Most studies focus on singular forms of bias, such as political or word-choice bias, without addressing a more comprehensive range of biases that may exist in news articles. Additionally, none of these approaches have leveraged LLMs to build a multi-label dataset capable of detecting various types of biases in a single framework. Given the limitations of prior work, our research proposes a new approach that explores multiple forms of bias using LLMs, thus enabling a broader, multi-label bias detection framework that has yet to be fully explored in the literature.

In this section, we will first outline the various types of biases along with their definitions to ensure a clear understanding of each bias. Afterward, we will delve into the process of dataset creation, discuss how the labels were extracted, and detail the techniques applied to the collected and filtered data.

In this study, we evaluated several transformer-based models to classify biases in news articles, specifically focusing on BERT (Devlin, 2018), RoBERTa (Liu, 2019), ALBERT (Lan, 2019), DistilBERT (Sanh, 2019), and XLNet (Yang, 2019). One major challenge was the significant class imbalance present in the dataset, as illustrated in Figure 9, where the number of samples for the positive class is substantially lower than that for the negative class across most biases. To address this issue, we implemented an inverse frequency weighting method, which assigned higher weights to the positive class for each label during training, helping to mitigate the effects of class imbalance.

Experiments were conducted on a T4 GPU using a batch size of 8, leveraging the AdamW optimizer (Loshchilov, 2017), a commonly used variant of the Adam optimizer, designed to better handle overfitting through weight decay. The linear learning rate scheduler was configured with a starting learning rate of $2\times 10^{-5}$, which decays over time to final learning rate to 0, ensuring smooth convergence. Each model was trained for 6 epochs, balancing between model performance and training time efficiency.

In multilabel classification, it is particularly important to retain the distribution of all labels across folds, as each sample can be associated with multiple labels simultaneously. A crucial aspect of this study was the dataset splitting strategy. Rather than randomly splitting the dataset into training, validation, and test sets, which could lead to skewed splits and unrepresentative distributions of the various labels, we used Multilabel Stratified KFold splitting. This method ensures that the label distribution is proportionally maintained across the training, validation, and test sets, preventing the model from learning from biased splits.

Following our experiments with various transformer-based models—BERT, RoBERTa, ALBERT, DistilBERT, and XLNet—we evaluated each model’s capacity to classify different types of biases present in news articles. Table 1 provides a summary of the models’ performance across the biases, measured through precision, recall, and F1-score. Our analysis highlights both the models’ effectiveness in detecting explicit bias types and the challenges posed by class imbalance and nuanced biases.

• •

  Model Performance Comparison - BERT consistently outperformed other models across most bias categories, particularly achieving an F1-score of 0.89 in Political Bias detection, which may be attributed to BERT’s robust contextual embedding capabilities. RoBERTa showed a competitive performance, closely following BERT in several categories. However, models like DistilBERT and ALBERT, with fewer parameters, generally exhibited lower F1-scores, particularly in Gender and Sensational Bias detection.

• •

  Class Imbalance Impact and Mitigation - Class imbalance posed a considerable challenge in our dataset, especially in Racial and Region Bias detection, which had notably fewer positive samples. Despite our use of inverse frequency weighting to emphasize the minority classes, models like DistilBERT and XLNet struggled with these biases, achieving F1-scores as low as 0.38 and 0.51 in Racial Bias detection, respectively. This imbalance suggests that additional data or more sophisticated augmentation techniques may be necessary to further alleviate these performance gaps.

• •

  LLM Annotations Reliability - While GPT-4omini provided fairly accurate annotations, we identified several errors in its assessments. In some cases, it incorrectly flagged articles as biased when they did not contain bias particularly in articles with multiple biases. Appendix B includes examples of these misannotations. Improving the accuracy of LLM annotations remains an area of future exploration.

In this study, we expanded the framework for bias detection in news articles by identifying various types of biases rather than simply categorizing them as biased or unbiased. Our unique dataset, labeled using large language models (LLMs), represents the first attempt to classify news articles according to seven distinct bias categories prevalent in the media landscape. We applied transformer-based models to this dataset and conducted a comparative analysis of their performance, concluding that BERT emerged as the most effective model overall. This research provides valuable insights into the complexities of bias detection in news media, paving the way for more nuanced approaches in future studies.

We utilized an LLM, specifically GPT-4omini, to label the dataset, which significantly reduced the time required for annotation. However, relying solely on the LLM for labeling can introduce some discrepancies and inconsistencies. In future work, we aim to explore methods to balance annotation reliability while maximizing the LLM’s capabilities.

Another limitation lies in the dataset itself, particularly the imbalance of samples across bias categories. Although techniques were employed to mitigate the impact of class imbalance, exploring advanced methods to expand and diversify the dataset—especially for underrepresented biases—could lead to more accurate model performance.

Additionally, the dataset splitting method could be further refined. While Multilabel Stratified KFold was used to maintain label distribution, future work could investigate other splitting strategies to prevent potential skewness and better reflect real-world distributions. This, combined with further experimentation on alternative data augmentation techniques, could improve the generalization capabilities of the models. Addressing these limitations would strengthen the reliability of the system and contribute to more accurate and comprehensive bias detection in news articles.