Investigating Subtler Biases in LLMs:
Ageism, Beauty, Institutional, and Nationality Bias in Generative Models

Alongside the impressive new capabilities of recent language generation models such as ChatGPT Brown et al. (2020), GPT-4 OpenAI (2023), and Llama-2 Touvron et al. (2023), these systems are increasingly involved in consequential decisions made in the real world. This includes job hiring and performance reviews, with tips for hiring managers appearing across the internet. Even before these recent advancements, AI has been used in the criminal justice system leading to the amplification of social inequities Moy (2019). In order to manage these biases prior research has investigated the most salient dimensions of bias in word embeddings and LLMs, such as gender and ethnicity Bolukbasi et al. (2016); Caliskan et al. (2017); Kurita et al. (2019). Prior work also focuses particularly on whether these AI systems produce specific stereotypes of underrepresented minorities, such as associating Middle Eastern people with perfumes or Jewish people with greed Nangia et al. (2020); Nadeem et al. (2021).

Social scientists found that human biases extend beyond simple stereotypes and can lead to general associations of positive attributes to members holding (or perceived to hold) certain key characteristics. For example, Dion et al. (1972) found that people are more likely to infer a plethora of other desirable characteristics to people that are judged more attractive—a result that has been confirmed and elaborated upon for the present day by more recent studies Commisso and Finkelstein (2012); Peng et al. (2019); Maurer-Fazio and Lei (2015); Weber et al. (2013).

In this paper, we extend the evaluation of bias in LLMs in the following key ways: we investigate whether LLMs make general associations between stereotyped categories and unrelated positive, negative, and neutral attributes rather than specific stereotype inferences. In addition, we investigate dimensions of bias that have been largely overlooked: age, beauty, academic institution, and nationality. Although understudied, these dimensions of bias follow people as much as gender and ethnicity do. Furthermore, the few existing studies regarding these biases in LLMs all study them in terms of specific stereotypes. Figure 1 shows an example of how we formulate the completion task for LLMs in the bias domain of academic institutions.

The contributions of this paper are the following.

1. 1.

   We formulate a task to investigate biases along generalized positive-negative sentiment rather than specific stereotypes and present a procedure for semi-automatically collecting a large dataset for this task.

2. 2.

   We investigate both directions of biased association: generating unrelated attributes given a bias-triggering description and generating biased triggering descriptions from unrelated attributes.

3. 3.

   We find that current LLMs show a pattern of bias in the domains we considered save for a few specific model-domain combinations.

Many studies demonstrate the pervasive impact of social biases in various spheres of life. Dion et al. (1972) found that people who are considered attractive are more likely to be believed to hold socially desirable traits and higher occupational status. This attractiveness bias also occurs inversely, as shown by Gross and Crofton (1977), where known desirable qualities influence our perceptions of an individual’s beauty. These two papers inspired us to set up our experimental design. More recent psychological research has confirm using modern scientific standards that such biases extend in specific real-world settings like employee termination Commisso and Finkelstein (2012), hiring practices Peng et al. (2019), interview callbacks Maurer-Fazio and Lei (2015), and even in victim blaming scenarios Weber et al. (2013).

An opposite bias effect for beauty exists for people in particular contexts, in which a highly attractive person is dispreferred Agthe et al. (2010). These effects are subject to interactions with sexual motive, social competition, culture, and the nature of the interaction making them more difficult to study Wan and Wyer (2015); Li et al. (2020). Generally, this opposing beauty bias effect appears when a preference for beauty would introduce a social threat (e.g., hiring a same-sex person, an embarrassing purchase from an opposite-sex person, etc.). In this paper we focus on the most general beauty bias where beauty is unduly associated with positive attributes leaving the opposite bias effect for future work.

Age-related stereotypes similarly impact societal perceptions and actions, with studies by Perdue and Gurtman (1990), Marques et al. (2020), and Donizzetti (2019) indicating a general propensity to infer negative traits based on age. This aligns with the foundational work on ageism by Butler (1969) and has been found to have implications in professional settings Ng and Feldman (2012).

Organizations and institutions have been found to be associated with specific personality traits, both within the US Slaughter et al. (2004) and internationally Anderson et al. (2010). Rutter et al. (2017) found that universities leverage specific personality traits when branding and Humburg (2017) student personality traits play an important role in student choice of university and alternatives (e.g., vocational education). Such bidirectional effects of personality traits result in hiring biases that prioritize students from academic institutions with particular reputations Morley and Aynsley (2007); Mavridopoulou and O’Mahoney (2020).

Nationality bias has found to affect student interactions in multicultural online learning environments Morales-Martinez et al. (2020), consumer perception towards products Insch and McBride (2004), academic philosophy Seabra et al. (2023), and peer evaluation Tavoletti et al. (2022). Tavoletti et al. (2022) identified the economic development of a person’s country of origin as an important factor within this bias, overshadowing individual qualities when one evaluates their peers.

As LLMs are trained on data created by humans, we hypothesize that they are prone to similar biases to those identified in people. If present, we must measure the degree to which such biases are present so that we can appropriately account for them when LLMs are used for consequential decisions, such as recruiting and hiring. In order to do this, we measure our bias in terms of “representational harms” (using the terminology from Blodgett et al. (2020)), that is, an LLM is biased if it makes general associations between stereotyped categories and unrelated positive, negative, and neutral attributes. Following the guidance of Blodgett et al. (2020), our objective is to define and measure these biases, particularly how they manifest and potentially perpetuate existing social hierarchies.

Moving on to related work in NLP specifically, bias in models have been studied for word embeddings using cosine similarity (e.g., the Word Embedding Association Test) and sentence embeddings using templates such as “This will [target]” (e.g., the Sentence Encoder Association Test.) Bolukbasi et al. (2016); Caliskan et al. (2017); May et al. (2019). Nangia et al. (2020) created Crowdsourced Stereotype Pairs (CrowS-Pairs), a dataset that studied nine different types of social biases (e.g., age, nationality, physical appearance, etc.) in masked language models. This work is the closest to our own in its study of ageism, beauty, and nationality and the use of intrasentence biases. However, it differs in both the model type (autoregressive vs. masked) and in the generality of the associations that are studied. Nadeem et al. (2020) introduced a dataset called StereoSet to measure stereotypical bias in both masked and autoregressive pretrained language models. Nationality bias has been studied before in GPT-2 by Venkit et al. (2023). They generated stories and analyzed how other factors (e.g., number of internet users, economic condition, etc.) affect nationality bias.

Czarnowska et al. (2021) explored seven social biases in three RoBERTa-based models, including age, gender, and nationality. They used GDP to categorize countries for nationality bias. Sun et al.’s (2022) systematic study on pretrained language model-based evaluation metrics revealed that they exhibit greater bias than traditional metrics on attributes such race and gender. Zhang et al. (2021) designed cloze-style example sentences to explore performance differences across demographic groups (age, race, etc.). Smith et al. (2022) created HOLISTICBIAS, a dataset for measuring bias across various demographic identities in language models.

Unlike previous studies which have focused on identifying bias in a single direction, we take a more general measurement by considering two directions of bias. For example, Nadeem et al. (2021) measured stereotypes of LLMs by giving the race description (e.g., Hispanic, Ghanaian etc.) or a profession (e.g., physicist, tailor, etc.) and asked LLMs to choose between stereotypicallty associated attributes (e.g., poor, creative, etc.), but they did not measure the bias in reverse context (e.g., by giving attributes and asking LLMs to choose the associated race or profession). In our study, we study both directions, inspired by Dion et al.’s (1972) and Gross and Crofton’s (1977) work showing that people demonstrate beauty bias in both directions. In our experiment, we use fill-in-the-blank style sentences which evoke the biases in the provided text using a description of a person based on the bias category we are studying. We will refer to this description as simply the stimulus. For example, in Figure 1 (a), the stimulus is “Hinds Community College” for institutional bias.

We consider four domains of bias in creating our dataset: age, beauty, academic institutions, and nations. We refer to academic institutions as institutions throughout our writing.²²2We focused on educational institutions, where rankings and quality classifications are readily available, but institution quality lacks a uniform metric across other types.

We use GPT-4, PaLM-2, Llama-2-13B, and Mistral-7B in our experiments.¹⁶¹⁶16We use 4-bit quantized versions for Llama-2 and Mistral 7B due to our resource constraints.^,171717We also have experimental results for GPT-3.5, which follow the same broad trends as the other models discussed here. Results for GPT-3.5 are only in the appendices due to space constraints. For model details, see Appendix B. Exact experimental details are in our supplementary materials, including scripts for replication. We examine how LLMs respond to positive and negative attributes and stimuli. Specifically, we calculate the conditional likelihood of a model selecting positive, negative, and neutral attributes in response to stereotypically positive and negative stimuli. We will refer to these as [stimulus]-to-[attribute] likelihoods. For example, we call the likelihood of the model to select positive attributes (e.g., friendly, motivated, creative, etc.) in response to stereotypically negative stimuli (e.g., unattractive, Hinds Community College, South Sudan, 65 years old, etc.) the negative-to-positive likelihood (NPL). Our shorthand uses P for positive, N for negative, and Nu for neutral. We consider a system to be biased if the conditional likelihood of positive, negative, or neutral completions change when the polarity of the prompting text changes. That is, an unbiased system will have $\Delta$PL, $\Delta$NL, and $\Delta$NuL values (defined in Figure 3) of 0. For example, in Figure 3, we find that $\Delta$NL is systematically negative, so the negative predictions of the models are biased in the opposite direction of the prompt text changes.

We further report correlations and statistical significance using Kendall’s $\tau$ test Kendall (1938).¹⁸¹⁸18We selected the Kendall’s $\tau$ test instead of the $\chi^{2}$ test because there is a natural order to negative, neutral, and positive categorical values. In the SAI direction, we calculate Kendall’s $\tau$ statistic between the binary positive and negative stimulus variable and the ternary positive, negative, and neutral attribute variable.

We reverse everything in the ASA direction, e.g., calculating the likelihood of selecting positive, negative, and neutral stimuli in response to positive and negative attributes.

This correlation between the stimuli and attributes for LLMs is statistically significant. Table 1 shows the results of the Kendall’s $\tau$ test for each model and in each direction. The null hypothesis is rejected in all eight settings. This serves as a clear indication of a pattern of bias in modern LLMs.

We next break down the results by bias category, where the $\tau$-test results are presented in Table 2. Here we focus on the broad trends of our results. Complete results and additional discussions are available in Appendix C.

We examine the behavior of four common LLMs across several less-studied domains of bias, looking at general positive and negative polarity associations rather than precise stereotypes. Our findings indicate that all four models exhibit statistically significant biases. Our dataset and experimental design draw upon prior literature on both social science and computer science. As the use of LLMs continues to grow and these models are increasingly employed in various tools, it becomes crucial to be vigilant about even subtle forms of bias. While much work has been done to investigate overt biases such as those related to race, gender, and religion, less attention has been paid to subtler biases such as ageism, beauty, and institution. Through the introduction of our dataset, we encourage the consideration of these overlooked biases when using LLMs. We hope that this dataset will help to further research and mitigate these types of biases. Future research is needed to extend our findings to other models and further biases that have been identified by social scientists.

There are a few factors in our experiments that may limit the generalizability of our results and conclusions. While we selected four of the most common and powerful LLMs currently available, our experiments were far from exhaustive. Many LLM variants exist today and will be developed in the future. In constructing our dataset, we limited many of our stimuli and attributes to a relatively small number of options. For instance, in the case of institutional bias, we consider only 100 universities and 100 community colleges. It is possible that our specific set of prompts and template structures also affected the results that we saw.

In grouping our stimuli and attributes we used several proxies to allow us to collect data efficiently. For example, economic conditions were used as proxies for grouping both professions and nationalities. We also used college rankings as a proxy for the sentiment polarity of academic institutions. All of these proxies are noisy approximations of the biases that we seek to measure and could have affected our results.

Our experiment was also only conducted in English. This means that the behavior of the LLMs, which are capable of working in multiple languages, may change in other languages. This could be due to technical reasons, such as the relatively smaller training data and development investment in other languages. Or this could be due to cultural or sociolinguistic reasons. That is, the prevalence, degree, and specific stimuli of each of the bias categories are culture-dependent and the way that the bias is realized in language is dependent on the sociolinguistic context—the linguistic and social norms of the community within which the utterance was generated.

This project was fully supported by the University of South Florida. We thank the reviewers for their valuable feedback especially their suggestions to add different types of confounding variables to our experiments.