Highly Generalizable Models for Multilingual Hate Speech Detection

We retrieved our datasets from hatespeechdata.com, a website published by Poletto et al that compiles hate speech data from 11 different languages. Data statistics for the combined datasets for each language is summarized in Table 1. The collection source and taxonomy used for each dataset is described below.

1. (1)

   English: The Ousidhoum et al. (Ousidhoum et al., 2019) dataset labels tweets with five attributes (directness, hostility, target, group, and annotator.) The Davidson et al. (Davidson et al., 2017) classifies collected tweets as hate speech or not hate speech using a lexicon directory for increased labeling accuracy. Gilbert et al. (de Gibert et al., 2018) uses a text extraction tool to randomly sample posts on Stormfront, a White Supremacist forum, and label them as hate speech or not. The Basile et al. (Basile, [n.d.]) dataset categorizes tweets as hate or not hate. Waseem et al. (Waseem and Hovy, 2016) uses the Twitter API to obtain tweets, then classifies them as hate speech or not hate speech. The Founta et al. (Founta et al., 2018) dataset uses a crowd-sourcing project to compile tweets that are then labeled as hate speech or not hate speech.

2. (2)

   Arabic: The Arabic Leventine Hate Speech and Abusive Language Dataset (ara, [n.d.]) consists of Syrian/Lebanese political tweets labeled as normal, abusive or hate. Ousidhoum et al. (Ousidhoum et al., 2019) dataset of tweets are labeled based on 5 attributes (directness, hostility, target, group, and annotator.)

3. (3)

   German: Ross et al. (Ross et al., 2016) compiles a small dataset of tweets regarding refugees in Germany that are classified as hate or normal. Bretschneider et al. (Bretschneider, [n.d.]) crawls 3 German Facebook pages (including comments published) and compiles data annotated as hate or not hate.

4. (4)

   Indonesian: Ibrohim et al. (et.al, [n.d.]e) collects tweets from Indonesian Twitter for multi-label classification where labels are hate or no-hate, abuse or no-abuse, and attributes of the people targeted including race, gender, religion, etc. The Alfina et al. (et.al, [n.d.]a) tweet dataset classifies them as hate speech or not hate speech. The aforementioned datasets were both obtained using a Python Web Scraper and the Twitter API.

5. (5)

   Italian: Sanguinetti et al. (Sanguinetti et al., 2018) dataset of tweets regarding certain minority groups in Italy that are labeled as hate or not hate. Bosco et al. (et.al, [n.d.]b) compiles Facebook posts and tweets about minority groups and immigrants in Italy using the Facebook and Twitter APIs.

6. (6)

   Portuguese: Fortuna et al. (Fortuna and Nunes, 2019) compiles hate speech tweets in Portuguese that are labeled at two levels: a simple binary label of hate vs no hate followed by a fine-grained hierarchical multiple label scheme with 81 hate speech categories in total.

7. (7)

   Spanish: Basile et al. (Basile, [n.d.]) includes tweets classified as hate or not hate. Pereira et al. (Pereira, [n.d.]) compiles tweets that are labeled as hate or not hate.

8. (8)

   French: Ousidhoum et al. (Ousidhoum et al., 2019) compiles a dataset of tweets that are labeled based on five attributes (directness, hostility, target, group, and annotator.)

9. (9)

   Turkish: Coltekin et al. (Çağrı Çöltekin, [n.d.]) randomly samples Turkish blog posts from Twitter and reviews and annotates them as offensive language or not offensive language. Tweets were sampled over a period of 18 months and are regarding various issues currently prominent in Turkish culture.

10. (10)

    Danish: Sigurbergsson et al. (Gudbjartur Ingi Sigurbergsson, [n.d.]) introduces dkhate, a dataset of user-generated comments from various social media platforms that are annotated as offensive or not offensive speech.

11. (11)

    Hindi: Kumar et al. (Ritesh Kumar, [n.d.]) develops a dataset from Twitter and Facebook that is annotated using binary labels depending on whether or not the example exudes aggression.

Our primary task is binary hate speech classification in a single modality, i.e., for only text data. Let $S$ be the raw input text sequence that we want to classify, $Y\in[0,1]$ be the target variable with $Y=0$ when $S$ is not hate speech and $Y=1$ when $S$ is hate speech. We generate a vectorized representation $X=E(S)$ where $E$ is an encoding technique (for example, an embedding generated by LASER (Research, [n.d.]a) or the output of the encoder in BERT models.) Let $M$ be the model architecture that we train to perform binary hate speech classification. We state the classification problem as follows:

Given $S$, we want our model to output $\hat{Y}=M(X)$ where the predicted label $\hat{Y}$ is ideally the same as $Y$.

We chose to use the following evaluation metric to measure model performance:
Weighted F1 Score: The F1-score is the harmonic mean of the precision and recall of a classifier. In our case, a lot of the datasets are class-imbalanced, which makes the precision metric extremely important. Additionally, the cost of a false negative (predicting something that is hate speech as benign) is high which makes the recall equally important. The F1 score is able to capture both the precision and recall under a single, balanced evaluation metric. We also ensure that the computation of the F1-score is class-weighted, i.e., each class is weighted by the fraction of the dataset that it represents.

The system configuration we used for experiments was as follows:

1. (1)

   CPU: 2 x 2.2GHz vCPU

2. (2)

   RAM: 13 GB DDR4

3. (3)

   GPU: 1 x Nvidia Tesla P100

We use the LASER + LR model architecture introduced by Aluru et al. (Aluru et al., 2021) as our baseline model. The model takes in the raw text sequence as input, generates a 1024 dimensional vector representation of the text from LASER (Research, [n.d.]a), and passes the resultant embedding into a Logistic Regression model. We utilize this model in 2 scenarios for our baseline results:

1. (1)

   Monolingual-Train Monolingual-Test: We train the model on a particular language and test it on the same language.

2. (2)

   Multilingual-Train Monolingual-Test: We train the model on all the languages and test it on each language separately.

The intuition behind this model is to use LASER as a method of capturing semantic and syntactic relationships of each language and use the representation it outputs to tune the parameters of the logistic regression model.

So far, research in multilingual hate speech detection, suffers from some overarching issues: (1) The lack of generalizable models due to hyper-focused set of languages or insufficient training examples in certain languages, and (2) The lack of creativity in leveraging certain properties of each language to their own advantage (for instance, the similarity of languages within a certain family.) We attempt to overcome these shortcomings by:

1. (1)

   Combining the benefits of an enriched multilingual dataset, cross-lingual representation methods, and/or complex model architectures to learn parameters using higher-quality representations.

2. (2)

   Training complex model architectures on different language families to leverage each family’s semantic tendencies and achieve relevant parameter learning.

Formally, we attempt to introduce two novel scenarios as part of our experiments:

1. (1)

   Multilingual-Train Monolingual-Test: We train the model on all available languages and test model performance on each language.

2. (2)

   Language Family Train-Test: We train the model on a particular language family and test model performance on each language in that family.

We train the following model architectures for each scenario in our experiment:

1. (1)

   LASER + LR: We generate sentence-level LASER embeddings and pass them as input to a Logistic Regression model. The model is expected to outperform the SoTA as its trained on our enriched dataset.

2. (2)

   MUSE + CNN-GRU: We generate word-level MUSE embeddings for each input word and pass them as input to the CNN-GRU architecture.

3. (3)

   mBERT: Multilingual-BERT is a variation of BERT pretrained on 104 languages without their respective language IDs. It can generalize to most of the langauges we consider. We use the raw sentences from each language to fine-tune the mBERT model.

All 3 of the candidate models use a batch size of 16 and are optimized using the AdamW optimizer and Cross Entropy loss. The model-specific hyperparameters are tabulated below:

Idea: Given $n$ languages, we train models $m_{1},m_{2},\dots,m_{n}$ for each langauge and test the performance of model $m_{i}$ on language $l_{i}$ for $i\in[1,n]$.

MUSE + CNN-GRU significantly outperforms LASER+LR and mBERT for 8/11 languages. mBERT in particular performs worse than both the other models in languages with heavy class imbalances such as Indonesian and Italian. Since mBERT requires a significant amount of data to fine-tune a specific task, it appears from the Monolingual-Train-Tests results that there is not enough data available for each language to properly fine-tune the model. The Arabic dataset has the highest absolute difference in performance, where the LASER+LR F1-score is approximately 60% lower than the other two models. It is curious that LASER+LR does no better than random guessing for this language while MUSE+CNN-GRU and mBERT are able to achieve a much higher F1 score with only 3300 training examples.

Idea: Given $n$ languages, we train a single model $m$ on all $n$ languages and test the model’s performance on each langauge $l_{i}$ for $i\in[1,n]$.

mBERT and MUSE+CNN-GRU perform equally well, with mBERT narrowly beating out the latter in 6/11 languages. This is likely because mBERT requires more data to achieve good performance, and with the multilingual dataset, it has access to a richer dataset consisting of 11 languages.

Compared to the Monolingual-Train, Monolingual-Test scenario, the F1 score on the Monolingual-test scenario with the Multilingual-Train model is better for almost all languages because the model is able to incorporate the semantics of multiple languages to enhance its performance. The F1 score goes down for a few languages which might be due to an undue influence from other languages that are dissimilar and have a large number of examples.

Idea: Given $n$ languages, we take a subset of the languages that form a language family $L_{F}$. We train a model $m_{F}$ on all languages in $L_{F}$ and test the model’s performance on each language $l\in L_{F}$.

From Table 6 and 7, it is evident that for all three model architectures, the Multilingual or Language-Family Train scenarios outperform the Monolingual-Train scenario in most cases. This demonstrates that the richer dataset across all languages allows for better classification performance due to a better generalized idea of hate speech and language semantics. What is interesting to note is: although the Multilingual train scenario tends to outperform the Language Family scenario in most cases, the F1 scores for the Monolingual Test cases are very similar. This indicates that despite having less data, the Language Family model is able to capture more relevant semantic information within a family of languages to achieve a performance close to that of the model trained on all languages. When compared to Germanic languages, the Romance Languages seem to achieve better performance in this scenario. Possible explanations for this could be that the Romance language group has more generalizability between languages compared to the Germanic group. However, we suspect that the lower performance in the Germanic language group can be attributed to the heavy bias of the model towards English with around 80% of the dataset consisting of English examples. In the future, one could investigate using a sub-sample of the English dataset to identify if this improves performance of the Germanic-language-family-trained model on other languages in the family.

Although we outperform the current SoTA for multilingual hate speech detection models, our work has the following limitations:

1. (1)

   Severe Imbalances in the Dataset: The datasets we use for each language suffers from two major types of imbalances that bias model performance: (1) textbfInter-Language Imbalance: There is a significant difference in the number of examples for each language in our dataset, which biases models towards the semantic tendencies of languages with lots of examples, and (2) Class Imbalance: A number of languages’ dataset consists mostly of examples that are not hate speech, which may strongly bias neural models.

2. (2)

   Limited Explainability in Model Performance: While we have performed a significant number of experiments using various models, our work fails to provide concrete explainability on the model’s performance and why one architecture outperforms another for a certain language or scenario.

Multilingual Hate speech detection is an extremely important area of research with lots of potential for improvement. We believe the following to be the most effective extensions to our work:

1. (1)

   Data Augmentation to Low-Resource/Imbalanced Languages: Future work could utilize techniques such as back-translation introduced in Markus et al. (Bayer et al., 2021) to augment examples and improve model performance in low-resource/imbalanced languages.

2. (2)

   Novel Candidate Architectures: Future work could test different architectures such as LASER + CNN-GRU and XLM-RoBERTa to compare performances across models.

3. (3)

   Zero-Shot/All-but-one Hate Speech Detection: Another useful extension would be to train models in a multilingual (all-but-one or language families) to detect how well they can generalize to unseen languages.