Men Are Elected, Women Are Married: Events Gender Bias on Wikipedia

Our collected corpus contains demographics information and description sections of celebrities from Wikipedia. Table 2 shows the statistics of the number of celebrities with Career or Personal Life sections in our corpora, together with all celebrities we collected. In this work, we only explored celebrities with Career or Personal Life sections, but there are more sections (e.g., Politics and Background and Family) in our collected corpus. We encourage interested researchers to further utilize our collected corpus and conduct studies from other perspectives. In each experiment, we select the same number of female and male celebrities from one occupation for a fair comparison.

There are two definitions of events: one defines an event as the trigger word (usually a verb) Pustejovsky et al. (2003b), the other defines an event as a complex structure including a trigger, arguments, time, and location Ahn (2006). The corpus following the former definition usually has much broader coverage, while the latter can provide richer information. For broader coverage, we choose a state-of-the-art event detection model that focuses on detecting event trigger words by Han et al. (2019).³³3We use the code at https://github.com/rujunhan/EMNLP-2019 and reproduce the model trained on the TB-Dense dataset. We use the model trained on the TB-Dense dataset Pustejovsky et al. (2003a) for two reasons: 1) the model performs better on the TB-Dense dataset; 2) the annotation of the TB-Dense dataset is from the news articles, and it is also where the most content of Wikipedia comes from.⁴⁴4According to Fetahu et al. (2015), more than 20% of the references are news articles on Wikipedia. We extract and lemmatize events $e$ from the corpora and count their frequencies $|e|$. Then, we separately construct dictionaries $\mathcal{E}^{m}=\{e_{1}^{m}:|e_{1}^{m}|,...,e_{M}^{m}:|e_{M}^{m}|\}$ and $\mathcal{E}^{f}=\{e_{1}^{f}:|e_{1}^{f}|,...,e_{F}^{f}:|e_{F}^{f}|\}$ mapping events to their frequency for male and female respectively.

To check the model performance on our corpora, we manually annotated events in 10,508 sentences (female: 5,543, male: 4,965) from the Wikipedia corpus. Table 3 shows that the model performs comparably on our corpora as on the TB-Dense test set.

After applying the event detection model, we get two dictionaries $\mathcal{E}^{m}$ and $\mathcal{E}^{f}$ that have events as keys and their corresponding occurrence frequencies as values. Among all events, we focus on those with distinct occurrences in males and females descriptions (e.g., work often occurs at a similar frequency for both females and males in Career sections, and we thus neglect it from our analysis). We use the Odds Ratio (OR) Szumilas (2010) to find the events with large frequency differences for females and males, which indicates that they might potentially manifest gender biases. For an event $e_{n}$, we calculate its odds ratio as the odds of having it in the male event list divided by the odds of having it in the female event list:

The larger the OR is, the more likely an event will occur in male than female sections by Equation 1. After obtaining a list of events and their corresponding OR, we sort the events by OR in descending order. The top $k$ events are more likely to appear for males and the last $k$ events for females.

The difference of event frequencies might come from the model bias, as shown in other tasks (e.g., gender bias in coreference resolution model Zhao et al. (2018)). To offset the potential confounding that could be brought by the event extraction model and estimate the actual event frequency, we propose a calibration strategy by 1) generating data that contains target events; 2) testing the model performance for females and males separately in the generated data, 3) and using the model performance to estimate real event occurrence frequencies.

We aim to calibrate the top 50 most skewed events in females’ and males’ Career and Personal Life descriptions after using the OR separately. First, we follow two steps to generate a synthetic dataset:

1. 1.

   For each target event, we select all sentences where the model successfully detected the target event. For each sentence, we manually verify the correctness of the extracted event and discard the incorrect ones. For the rest, we use the verified sentences to create more ground truth; we call them template sentences.

2. 2.

   For each template sentence, we find the celebrity’s first name and mark it as a Name Placeholder, then we replace it with 50 female names and 50 male names that are sampled from the name list by Ribeiro et al. (2020). If the gender changes during the name replacement (e.g., Mike to Emily), we replace the corresponding pronouns (e.g., he to she) and gender attributes Zhao et al. (2018) (e.g., Mr to Miss) in the template sentences. As a result, we get 100 data points for each template sentence with automatic annotations. If there is no first name in the sentence, we replace the pronouns and gender attributes.

After getting the synthetic data, we run the event extraction model again. We use the detection recall among the generated instances to calibrate the frequency $|e|$ for each target event and estimate the actual frequency $|e|^{*}$, following:

Then, we replace $|e|$ with $|e|^{*}$ in Equation 1, and get $k$ female and $k$ male events by sorting OR as before. Note that we observe the model performances are mostly unbiased, and we have only calibrated events that have different performances for females and males over a threshold (i.e., 0.05).⁶⁶6 Calibration details and quantitative result in App. A.2.

We further check if the extracted events are associated with gender attributes (e.g., she and her for females, and he and him for males) in popular neural word embeddings like Glove Pennington et al. (2014). We quantify this with the Word Embedding Association Test (WEAT)  Caliskan et al. (2017), a popular method for measuring biases in text. Intuitively, WEAT takes a list of tokens that represent a concept (in our case, extracted events) and verifies whether these tokens have a shorter distance towards female attributes or male attributes. A positive value of WEAT score indicates that female events are closer to female attributes, and male events are closer to male attributes in the word embedding, while a negative value indicates that female events are closer to male attributes and vice versa.⁷⁷7Details of WEAT score experiment in App. A.4.

To show the effectiveness of using events as a lens for gender bias analysis, we compute WEAT scores on the raw texts and detected events separately. For the former, we take all tokens excluding stop words.⁸⁸8We use spaCy (https://spacy.io/) to tokenize the corpus and remove stop words. Together with gender attributes from  Caliskan et al. (2017), we calculate and show the WEAT scores under two settings as “WEAT^∗” for the raw texts and “WEAT” for the detected events.

Table 5 and Table 5 show the associations of both raw texts and the extracted events in Career and Personal Life sections for females and males across occupations after the calibration. The values in WEAT^∗ columns in both tables indicate that there was only a weak association of words in raw texts with gender. In contrast, the extracted events are associated with gender for most occupations. It shows the effectiveness of the event extraction model and our analysis method.

There is a possibility that our analysis, although it picks out distinct events for different genders, identifies the events that are infrequent for all genders and that the frequent events have similar distributions across genders. To verify, we sort all detected events from our corpus by frequencies in descending order. Then, we calculate the percentile of extracted events in the sorted list. The smaller the percentile is, the more frequent the event appears in the text. Figure 1 shows that we are not picking the events that rarely occur, which shows the significance of our result.¹⁰¹⁰10See plots for all occupations in Appendix A.5. For example, Figure 1a and Figure 1b show the percentile of frequencies for selected male and female events among all events frequencies in the descending order for male and female writers, respectively. We can see that for the corresponding gender, event frequencies are among the top 10%. These events occur less frequently for the opposite gender but still among the top 40%.

We find that there are more Life events for females than males in both Career and Personal Life sections. On the other hand, for males, there are events like “awards” even in their Personal Life section. The mixture of personal life with females’ professional career events and career achievements with males’ personal life events carries implicit gender bias and reinforces the gender stereotype. It potentially leads to career, marital, and parental status discrimination towards genders and jeopardizes gender equality in society. We recommend: 1) Wikipedia editors to restructure pages to ensure that personal life-related events (e.g., marriage and divorce) are written in the Personal Life section, and professional events (e.g., award) are written in Career sections regardless of gender; 2) future contributors should also be cautious and not intermingle Personal Life and Career when creating the Wikipedia pages from the start.