CRAB: Class Representation Attentive BERT for Hate Speech Identification in Social Media

The purpose of representation learning layer is to generate a contextualized fixed-sized embedding vectors for the tweets. We utilize BERT encoder to vectorize the input data. BERT can provide a more sophisticated text representation by learning from left and right of a token’s context in all layers. CRAB takes all the BERT embedding generated from final block of the transformer. More precisely, this layer encodes the input tweets into matrix $B=(e_{0},...,e_{N-1}),B\in R^{|k|\times N}$ where $e_{i\in\{0,...,N-1\}}$ is the embedding vector for the i-th token, N is input text length, and $|k|$ is embedding dimension.

This layer is devised to allow the neural model to learn how the encoded classes should attend to every single token in the input. To this end, we introduce a multi-head token-wise class representation network A. Each head in this block learns the interactions between the encoded input tokens and classes independently. This layer takes $B^{\prime}=(e_{1},...,e_{N-1})$ as an input; $B^{\prime}$ is concatenation of all final layer’s hidden states except the hidden state corresponds to the first token (special token [CLS]). Similar to previous studies Du et al. (2019), this layer calculates the matching scores between classes and input data using dot product operation:

Where $A=(A_{0},...,A_{m})$, $A_{i}\in R^{c\times|k|}$ and $T=(T_{0},...,T_{m})$,$T_{i}\in R^{c\times(N-1)}$; number of classes is shown as c, and m is number of class representation heads.

Sentence-wise trainable class representation matrix is depicted as $\textbf{S}\in R^{c\times|k|}$ in Figure 1. Given the sentence embedding, $\textbf{{e}}\in R^{|k|\times 1}$ as an input to this layer, S is tuned to learn sentence-level class representation during the training process. Similar to sub-section 2.2, here, we also apply dot product to compute matching scores (shown as w) between the sentence embedding e, and sentence-wise class representation S:

This layer aims to fuse information learned from previous layers. Equations 4-7 describe how we aggregate the token-level interaction signals and compute the matching scores, z:

where $\sigma$ is LeakyReLu activation function Xu et al. (2015) and $\oplus$ indicates concatenation operator. Also, multi-dimensional array $(T^{\prime}_{0},...,T^{\prime}_{m})$ is shown as $T^{\prime}$ in Figure 1. In above equations, $W_{fc1},W_{fc2}$ and $W_{lin}$ are trainable weights. Finally, as denoted in equation 8, token-level and sentence-level matching scores are combined and passed to a softmax layer to predict the class labels:

We trained our model with tweets collected by Founta et al.¹¹1https://sites.google.com/view/icwsm2020datachallenge The corpus is comprised of 4 classes: Normal, Abusive, hateful and Spam. Class distribution is shown in Table 1; it is clear from the table that classes are heavily imbalanced. We apply stratified sampling and create train, validation and test sets with ratios of 80%,10%,10% respectively.

Tweets are full of emojis, emoticons, hashtags, and website links. To further clean tweets and yet preserve the useful information as much as possible, we develop a pipeline which maps emotionally similar emojis and emojicons into same special tokens. Likewise, the website links are also replaced with a special token. Generally, tweets are short and people do not follow grammatical rules in them. Therefore, we do not apply stop words removal, stemming or lemmatization as these techniques are often imperfect and could lead to information loss.

BERT was initialized with the pre-trained weights and we fine-tuned them for our downstream task during the training process. We chose batch size of 32 and number of neurons in the transformations $W_{fc1}$ and $W_{fc2}$ were 64 and 128 respectively. Model was implemented by Pytorch Paszke et al. (2017) with a single NVIDIA P100 GPU.

As listed in Table 2, we compare our approach with several baselines to evaluate effectiveness of our proposed model. The input to Naive Bayes classifier is TF-IDF Zhang et al. (2011) feature vectors. As part of our neural model baselines we employ CNN, RNN as well as EXAM Du et al. (2019); the word embedding size of all aforementioned neural models are set to 200. We also include classification performances of feeding BERT’s special token [CLS] embedding (BERT-CLS) and average pooled BERT’s embedding vectors (BERT-Avg-P) to a linear classifier. In both cases the outputs of last hidden layer would be sent to the classifiers. Given that our class distribution is imbalanced, we consider Macro-Averaged F1, Precision, Recall as well as Accuracy to quantify the prediction performance of the models.

Table 2 shows performances of different variations of our model, CRAB-n, as well as the baselines. Bold numbers denote the best performance and underlined numbers are the best performance among the baselines only. Letter n in CRAB-n indicates number of heads in token-wise class representation layer. During our experiment we tried various $n\in\{1,2,4,8,16\}$. To evaluate effectiveness of sentence-wise class representation layer, we also report performance of CRAB-4 with this layer removed (depicted as CRAB-4 w/o SA in Table 2). As shown in Table 2, CRAB-4 consistently outperformed all baselines and other CRAB variations. Our model, CRAB-4, achieved 1.89% relative improved Macro-averaged F1 score and 0.9% relative improved accuracy compared to BERT-Avg-P and BERT-CLS respectively. In terms of Macro-averaged precision and recall, there is relative improvement of 1.81% and 1.57% over BERT-Avg-P, respectively. To further analyze performance of CRAB-4 and understand the way it handles imbalanced classes, we conducted error analysis on each class. Compared to our baseline BERT models, we noticed CRAB-4 obtained 1% Macro-F1 boost in the first two major classes and for the two minor classes it gained 2% increase. We hypothesize that, minor classes benefited the most from this architecture. It is worth mentioning our model can be extended to multi-label classification by simply replacing softmax with sigmoid layer. We emphasize that in our proposed architecture, input representation layer is not just limited to BERT and any other form of transformer-based encoder can be used instead.