End-to-End Annotator Bias Approximation
on Crowdsourced Single-Label Sentiment Analysis

Problem definition.  The need for data in the growing research areas of machine learning has given rise to the generalized use of crowdsourcing. This method of data collection increases the amount of data, saves time and money but comes at the potential cost of data quality. One of the key metrics of data quality is annotator reliability, which can be affected by various factors. For instance, the lack of rater accountability can entail spamming. Spammers are annotators that assign labels randomly and significantly reduce the quality of the data. Raykar and Yu (2012) and Hovy et al. (2013) addressed this issue by detecting spammers based on rater trustworthiness and the SpEM algorithm. However, spammers are not the only source of label inconsistencies. The varied personal backgrounds of crowd workers often lead to annotator biases that affect the overall accuracy of the models. Several works have previously ranked crowd workers Hovy et al. (2013); Whitehill et al. (2009); Yan et al. (2010), clustered annotators Peldszus and Stede (2013), captured sources of bias Wauthier and Jordan (2011) or modeled the varying difficulty of the annotation tasks Carpenter (2008); Whitehill et al. (2009); Welinder et al. (2010) allowing for the elimination of unreliable labels and the improvement of the model predictions.

Ground truth estimation.  One common challenge in crowdsourced datasets is the ground truth estimation. When an instance has been annotated multiple times, a simple yet effective technique is to implement majority voting or an extension thereof TIAN and Zhu (2015); Yan et al. (2010). More sophisticated methods focus on modeling label uncertainty Spiegelhalter and Stovin (1983) or implementing bias correction Snow et al. (2008); Camilleri and Williams (2020). These techniques are commonly used for NLP applications or computer vision tasks Smyth et al. (1995); Camilleri and Williams (2020). Most of these methods for inferring the ground truth labels use variations of the EM algorithm by Dawid and Skene (1979), which estimates annotator biases and latent labels in turns. We use its recent extension called the Fast Dawid-Skene algorithm Sinha et al. (2018).

End-to-end approaches.  The Dawid-Skene algorithm models the raters’ abilities as respective bias matrices. Similar examples include GLAD Whitehill et al. (2009) or MACE Hovy et al. (2013), which infer true labels as well as labeler expertise and sample difficulty. These approaches infer the ground truth only from the labels and do not consider the input features. End-to-end approaches learn a latent truth, annotator information, and feature distribution jointly during actual model training Zeng et al. (2018); Khetan et al. (2017); Rodrigues and Pereira (2018). Some works use the EM algorithm Raykar et al. (2009), e.g., to learn sample difficulties, annotator representations and ground truth estimates Platanios et al. (2020). However, the EM algorithm has drawbacks, namely that it can be unstable and more expensive to train Chu et al. (2020). LTNet models imperfect annotations derived from various image datasets using a single latent truth neural network and dataset-specific bias matrices Zeng et al. (2018). A similar approach is used for crowdsourcing, representing annotator bias by confusion matrix estimates Rodrigues and Pereira (2018). Both approaches show a mismatch between the bias and how it is modeled, see Zeng et al. (2018, figure 5) and Rodrigues and Pereira (2018, figure 3). We adapt the LTNet architecture (see section 3), as it can be used to model crowd annotators on singly labeled sentiment analysis, which, to our knowledge, is not done yet in the context of annotator bias modeling. Recent works about noisy labeling in sentiment analysis do not consider annotator bias Wang et al. (2019).

Sentiment and Emotion.  Many works use the terms sentiment and emotion interchangeably Demszky et al. (2020); Kossaifi et al. (2021), whereas sentiment is directed towards an entity Munezero et al. (2014) but emotion not necessarily. Both can be mapped to valence, which is the affective quality of goodness (high) or badness (low). Since emotion recognition often lacks annotated data, crowdsourced sentiment annotations can be beneficial Snow et al. (2008).

Multi-Labeled Crowdsourced Datasets.  Crowdsourced datasets, such as, Google GoEmotion Demszky et al. (2020) and the SEWA database Kossaifi et al. (2021), usually contain multiple labels per sample and require their aggregation using ground truth estimation. Multi-labeled datasets are preferable to singly labeled ones on limited data. Snow et al. (2008) proved that many non-expert annotators give a better performance than a few expert annotators and are cheaper in comparison.

Singly Labeled Crowdsourced Datasets.  Singly labeled datasets are an option given a fixed budget and unlimited data. Khetan et al. (2017) showed that it is possible to model worker quality with single labels even when the annotations are made by non-experts. Thus, multiple annotations can not only be redundant but come at the expense of fewer labeled samples. For singly labeled data, it can be distinguished between reviewer annotators and external annotators. Reviewer annotators rate samples they created themselves. It is common in forums for product and opinion reviews where a review is accompanied by a rating. As an example of this, we utilized the TripAdvisor dataset Thelwall (2018). Further candidates are the Amazon review dataset Ni et al. (2019), the Large Movie Review Dataset Maas et al. (2011), and many more comprising sentiment. External annotators annotate samples they have not created. Experts are needed for complex annotation tasks requiring domain knowledge. These are not crowdsourced, since the number of annotators is small and fixed. More common are external non-experts. Snow et al. (2008) showed that multi-labeled datasets annotated by non-expert improve performance. Khetan et al. (2017) showed that it also performs well in the singly labeled case. Thus, datasets made of singly labeled non-expert annotations can be cheaper, faster, and obtain performances comparable to those comprised of different types of annotations. Our organic dataset is annotated accordingly, see section 4.3.

The model choice is determined by the fact that some of our datasets are small. Thus, the model should have only few trainable parameters to avoid overfitting. We utilize a simple attention mechanism, as it is comon for NLP applications. The input words $w_{j}$ are mapped to their word embeddings $e_{w_{j}}\in\mathds{R}^{D}$ with $j=1,...,S$, and $S$ being the input sequence length and $D$ the dimensionality of the input word vectors. These are GloVe embeddings of 50 dimensions pre-trained on 6B English tokens of the ”Wikipedia 2014 + Gigaword 5” dataset Pennington et al. (2014). Then, it computes the attention $a_{i}$ of each word using the trainable attention vector $e\in\mathds{R}^{D}$ via $a_{j}=e\cdot e_{w_{j}}$. It takes the accordingly weighted average $z_{n}=\sum_{i=1}^{S}a_{i}\cdot e_{w_{i}}$ of the word vectors with $n$ denoting the $n$-th sample or input text.

Finally, the classification head is the sigmoid of a simple linear layer $p_{n}=\text{softmax}(W\cdot z_{n}+b)$, with $W\in\mathds{R}^{L\times D}$ and $b\in\mathds{R}$ as the weights of the model. We refer to this last layer and to $p_{n}$ as latent truth layer or latent truth.

On top of the basic modeling architecture, the biases of the annotators are modeled as seen in figure 1. The theory is explained by Zeng et al. (2018) as follows:

“The labeling preference bias of different annotators cause inconsistent annotations. Each annotator has a coder-specific bias in assigning the samples to some categories. Mathematically speaking, let $\mathcal{X}=\{x_{1},\dots,x_{N}\}$ denote the data, $y^{c}=[y^{c}_{1},\dots,y^{c}_{N}]$ the regarding annotations by coder $c$. Inconsistent annotations assume that $P(y^{c}_{n}|x_{n})\neq P(y^{\hat{c}}_{n}|x_{n}),\forall x_{n}\in\mathcal{X},c\neq\hat{c}$, where $P(y^{i}_{n}|x_{n})$ denotes the probability distribution that coder $c$ annotates sample $x_{n}$. LTNet assumes that each sample $x_{n}$ has a latent truth $y_{n}$. Without the loss of generality, let us suppose that LTNet classifies $x_{n}$ into the category $i$ with probability $P(y_{n}=i|x_{n};\Theta)$, where $\Theta$ denotes the network parameters. If $x_{n}$ has a ground truth of $i$, coder $c$ has an opportunity of $\tau^{c}_{ij}=P(y^{c}_{n}=j|y_{n}=i)$ to annotate $x_{n}$ as $j$, where $y^{c}_{n}$ is the annotation of sample $x_{n}$ by coder $c$. Then, the sample $x_{n}$ is annotated as label $j$ by coder $c$ with a probability of $P(y^{c}_{n}=j|x_{n};\Theta)=\sum^{L}_{i=1}P(y^{c}_{n}=j|y_{n}=i)P(y_{n}=i|x_{n};\Theta)$, where $L$ is the number of categories and
$\sum^{L}_{j=1}P(y^{c}_{n}=j|y_{n}=i)=\sum^{L}_{j=1}\tau^{c}_{ij}=1$. $T^{c}=[\tau^{c}_{ij}]_{L\times L}$ denotes the transition matrix (also referred to as annotator bias) with rows summed to $1$ while $\left[p_{n}\right]_{i}=P(y_{n}=i|x_{n};\Theta)$ is modeled by the base network Zeng et al. (2018). We define $\left[p_{n}^{c}\right]_{j}=P(y^{c}_{n}=j|x_{n};\Theta)$. Given the annotations from $C$ different coders on the data, LTNet aims to maximize the log-likelihood of the observed annotations. Therefore, parameters in LTNet are learned by minimizing the cross entropy loss of the predicted and observed annotations for each coder $c$. ”

We represent the annotations and predictions as vectors of dimensionality $L$ such that $y^{c}_{n}$ is one-hot encoded and $p^{c}_{n}$ contains the probabilities for all class predictions of sample $n$. The cross entropy loss function is then defined as $-\sum_{n=1}^{C}\sum_{n=1}^{N}\log({p_{n}^{c}}^{\intercal}\cdot y_{n}^{c})$.

The logarithm in the cross entropy formula leads to an exponential increase in the loss for false negative predictions, i.e., when the predicted probability $\left[p^{c}_{n}\right]_{i}$ for a ground truth class $i$ is close to $0$ and $\left[y^{c}_{n}\right]_{i}$ is $1$. This increase can be helpful in conditions with numerical underflow, but at the same time this introduces a disproportionate high loss of the other class due to constantly misclassified items. This happens in crowdsourcing, for example, when one annotator is a spammer assigning a high degree of random annotations, which in turn leads to a disproportionally higher loss caused by that annotator’s many indistinguishable false negative annotations. Consequentially, the bias matrix of that annotator would be biased towards the false classes. Moreover, this annotator would cause overall more loss than other annotators, which can harm the model training for layers which are shared among all annotators, e.g., the latent truth layer when it is actually trained.

By omitting the log function, these effects are removed and all annotators and datapoints contribute with the same weight to the overall gradient and to the trainable annotator bias matrices, independent of the annotator and his respective annotation behavior. As a consequence, the annotator matrices are capable of modeling the real annotator bias, which is the mismatch between an annotation $y^{c}_{n}$ of coder $c$ and the latent truth prediction $p_{n}$. If $p_{n}$ is one-hot encoded, this results to the according classification ratios of samples and is equal to the confusion matrix, without an algorithmically encoded bias towards a certain group of items. This is shown mathematically in the following, where it is assumed that the base network is fixed, i.e., backpropagation is performed through the bias matrices and stops at the latent truth layer.

We define $N=\sum_{k=1}^{L}N_{k}$ as the number of all samples and $N_{k}$ of class $k=1,...,L$. $L$ is the number of classes, $T^{c}=[\tau^{c}_{ij}]_{L\times L}$ the bias matrix of coder $c$, $p_{n}$ the latent truth vector of sample $n=1,...,N$, and $p_{n}^{c}$ the annotator prediction. $p_{km}$ is the latent truth of the $m$-th sample of class $k$ with $m=1,...,N_{k}$, same for $x_{km}$ and $y_{km}^{c}$. The loss without logarithm is

Apparently, the derivation step between the second and third line would not work if there would be the logarithm from the standard cross entropy. Now, let the learning rate be $\alpha$, the number of epochs $E$ and the starting values of the initialized bias matrix $\left(\tau^{c}_{lh}\right)_{0}$. The bias parameters $\tau^{c}_{lh}$ of the bias matrix $T^{c}$ are updated according to

For sufficiently large $E$ the starting values $\left(\tau^{c}_{hk}\right)_{0}$ become infinitesimally small in comparison to the second additive term and thus negligible. As we are normalizing the rows of $(T^{c})_{E}$ after training so that the bias fulfills our probability constraint defined in section 3.2, the linear factor $-\alpha E$ is canceled out, too. Thus, the bias matrix $T^{c}$ results in the row normalized version of $[Z_{hk}]_{L\times L}$. $Z_{hk}$ is the sum of the latent truth probabilities for class $h$ on all samples of a ground truth class $k$. If we assume that the latent truth is one hot encoded, $[Z_{hk}]_{L\times L}$ equals to the confusion matrix, of which the $k$-th column sums up to the number of samples in class $k$: $\sum_{h=1}^{L}Z_{hk}=\sum_{h=1}^{L}\sum_{m=1}^{N_{k}}\left[p_{km}\right]_{h}=\sum_{m=1}^{N_{k}}1=N_{k}$.

The following experiment compares how training with and without the logarithm in the cross entropy loss affects the LTNet bias matrices empirically. The mathematical explanations in section 3.3 suggest that the logarithm removal from cross entropy leads to an annotator bias matrix identical to the confusion matrix, which would not be the case for the normal cross entropy.

Experiment Description.  For the data, we use the TripAdvisor dataset from Thelwall et al. consisting of $11,900$ English consumer reviews about hotels from male and female reviewers plus their self-assigned sentiment ratings Thelwall (2018). We use the gender information to split the data into two annotator groups, male and female, from which we model each one with a corresponding bias matrix. We exclude neutral ratings and binarize the rest to be either positive or negative. As the dataset is by default completely balanced regarding gender and sentiment at each rating level, it is a natural candidate for correct bias approximation. Throughout our experiments, we use 70% of the obtained data as training, 20% as validation and the 10% remaining as test sets.

Similar to the explanation in 3.3, the base model with its latent truth predictions is pre-trained on all samples and then frozen when the bias matrices are trained. The stochastic gradient descent method is used to optimize the parameters, as other wide-spread optimizers, such as Adam and AdaGrad (the latter introduced that feature first), introduce an – in our case undesired – bias towards certain directions in the gradient space, namely by using the previous learning steps to increase or decrease the weights along dimensions with larger or smaller gradients Kingma and Ba (2014). For all four sub-experiments, we train the base models with varying hyperparameters and pick the best based on accuracy. We train the transition matrices 50 times with different learning rates from the interval $[1\mathrm{e}{-6},1\mathrm{e}{-3}]$. The batch size is 64. In addition to a normal training setting, we add random annotations to 80% of the instances annotated by male subjects, such that  40% from them are wrongly annotated. This results in four models: with and without logarithm in the cross entropy, with and without random male annotations, each time respectively with two annotator group matrices, male and female – see figure 2.

Results.  The bias matrices of the models with the best accuracy are picked and presented in figure 2 in the top row. The corresponding confusion matrices depict the mismatch between latent truth predictions and annotator-group labels in the bottom row. The bias matrices trained without logarithm in the cross entropy are almost identical to the confusion matrices in all cases, which never holds for the normal cross entropy. This confirms our mathematically justified hypothesis given in section 3.3 that the logarithm removal from cross entropy leads to a correctly end-to-end-trained bias. In this context, it is relevant that the related work shows the same mismatch between bias and confusion matrix when applying cross entropy loss without explaining nor tackling this difference, see Zeng et al. (2018, figure 5) and Rodrigues and Pereira (2018, figure 3).

It is worth mentioning for the 80% random male annotations that these are correctly modeled without cross entropy, too, as opposed to normal cross entropy. If the goal is to model the annotator bias correctly in an end-to-end manner, this might be considered as particularly useful to analyze annotator behavior, e.g., spammer detection, later on.

Finally, we report how much variation the bias matrices show during training for cross entropy with and without logarithm. As mentioned in the experiment description, we trained each model $50$ times. The elements of the resulting bias matrices with standard cross entropy have on average $7.7\%$ standard deviation compared to $2.8\%$ without logarithm. It can be concluded that the bias produced by standard cross entropy is less stable during training, which raises questions about the overall reliability of its outcome.

In summary, the observations confirm our assumptions that cross entropy without logarithm captures annotator bias correctly in contrast to standard cross entropy. This carries the potential to detect spammer annotators and leads to an overall more stable training.

In the following paragraphs, we demonstrate how to estimate the ground truth based on the latent truth from LTNet. This is then compared to two other kinds of ground truth estimates. All of them can be applied in a single label crowdsourcing setting.

The Dawid-Skene algorithm Sinha et al. (2018) is a common approach to calculate a ground truth in crowdsourcing settings where there are multiple annotations given on each sample. This method is, for instance, comparable to majority voting, which tends to give similar results for ground truth estimation. However, in single label crowdsourcing settings, these approaches are not feasible. Under single label conditions, the Dawid-Skene ground truth estimates equal to the single label annotations.

This is given by Sinha et al. (2018, formula 1) in the expectation step, where the probability for a class $k\in{1,2,...,L}$ given the annotations is defined as

Here, $n$ is the sample to be estimated, $C$ the number of annotators for that sample, ${n_{1},n_{2},...,n_{C}}$ the set of annotators who labeled this sample, ${k_{n_{1}},k_{n_{2}},...,k_{n_{C}}}$ the set of annotation choices chosen by these $C$ participants for sample n, and $Y_{n}$ the correct (or aggregated) label to be estimated for the sample $n$ Sinha et al. (2018).

In the single label case $C$ equals to 1, which reduces the formula to $P(Y_{n}=k|k_{n_{1}},k_{n_{2}},...,k_{n_{C}})=P(Y_{n}=k|k_{n_{1}})$. This in turn equals to $1$ if $k$ is the assigned class label to sample $n$ by annotator $n_{1}$, or $0$ otherwise. In other words, if there is only one annotation per sample, this annotation defines the ground truth. Since different annotators do not assign labels on the same samples, there is also no way to model mutual dependencies of each other.

LTNet, however, provides estimates for all variables from this formula. $P(Y_{n}=k)$ is the prior and is approximated by the latent truth probability for class $k$ of sample $n$. $P(k_{n_{c}}|Y_{n}=k)$ is the probability that, assuming $k$ would be the given class, sample $n$ is labeled as $k_{n_{c}}$ by annotator $n_{c}$. This equals to $\tau^{c}_{k_{n_{c}},k}$, i.e., the entries of the LTNet bias matrix $T^{c}$ of annotator $c$.

Eventually, the LTNet ground truth can be derived by choosing $k$ such that the probability $P(Y_{n}=k|k_{n_{1}},...)$ is maximized:

We will leverage this formula to derive and evaluate the ground truth generated by LTNet.

We conduct classification comparing LTNet in different configurations on three datasets with crowdsourced sentiment annotations to discuss the potential related benefits and drawbacks of our proposed loss modification.

Emotion Dataset.  The emotion dataset consists of 100 headlines and their ratings for valence by multiple paid Amazon Mechanical Turk annotators Snow et al. (2008). Each headline is annotated by 10 annotators, and each annotated several but not all headlines. We split the interval-based valence annotations to positive, neutral, or negative. Throughout our experiments, we used 70% of the obtained data as training, 20% as validation and 10% as test sets.

Organic Food Dataset.  With this paper, we publish our dataset containing social media texts discussing organic food related topics.

Experiment Description.  The experiment is conducted on the TripAdvisor, organic, and emotion datasets introduced in section 4.3. We compare the classification of the base network with three different LTNet configurations. Two of them are trained using cross entropy with and without logarithm. For the emotion dataset, we compute the bias matrices and the ground truth for the base model using the fast Dawid-Skene algorithm Sinha et al. (2018). This is possible for the emotion dataset, since each sample is annotated by several annotators.

We apply pre-training for each dataset by training several base models with different hyperparameters and pick the best based on accuracy. Eventually, we train the LTNet model on the crowdsourcing annotation targets by fine-tuning the best base model together with the bias matrices for the respective annotators. The bias matrices are initialized as row normalized identity matrices plus uniform noise around $0.1$. The models are trained 50 times with varying learning rates sampled from between $[1\mathrm{e}{-6},1\mathrm{e}{-3}]$. A batch size of 64 is used.

Results.  The classification results of the models are presented in table 2 with their macro F1 score and accuracy as derived via predictions on the test sets. LTNet generally shows a significant classification advantage over the base model. On all three databases, LTNet approaches performed better on the test datasets. The LTNet improvement has a big delta of $11\%+/-1\%$ when there is a low annotation reliability (organic and emotion datasets) and a small delta $<1\%$ with high reliability (TripAdvisor) ³³3Unreliable means that the provided annotations have a low Cohen’s kappa inter-rater reliability on the organic $51.09\%$ and emotion ($27.47\%$) dataset. On the organic dataset we prepared a separate data partition of 300 sentences annotated by all annotators for that purpose. For the TripAdvisor dataset, it is apparent that the correspondence of annotations between the two annotator groups (male and female) is high as can be seen in figure 2 for cross entropy without logarithm.. Apparently, model each annotator separately gives significant advantages.

Regarding the comparison between cross entropy (CE) loss with and without logarithm on LTNet, the removed logarithm shows better classification results on organic ($+3\%$) and TripAdvisor data ($+0.3\%$) and worse on the emotion dataset ($-3\%$). This means that on both of the singly labeled crowdsourcing datasets, the removal of the logarithm from the loss function leads to better predictions than the standard CE loss. On the multi-labeled emotion dataset, however, this does not appear to be beneficial. As this data has only a very small test set of 100 samples, it is not clear if this result is an artifact or not. Concluding, the log removal appears to be beneficial on large datasets, where the bias is correctly represented in the training and test data splits, such that it can be modeled correctly by the denoted approach. It shall be noted, that it is not clear if that observation would hold generally. We advice to run the same experiments multiple times on many more datasets to substantiate this finding.