Who is Killed by Police: Introducing Supervised Attention for Hierarchical LSTMs

In this component, each token $t_{i,j,k}$ of a sentence $s_{i,j}$ in $c_{i}$ is transformed into an embedding vector $x_{i,j,k}=[q_{i,j,k},p_{i,j,k}]$, which is the concatenation of the word embedding vector $q_{i,j,k}$ and the position embedding vector $p_{i,j,k}$ [Nguyen and Grishman, 2015a]. These vectors are obtained as follows:

Word embedding vector: $q_{i,j,k}$ is obtained by taking the column vector corresponding to $t_{i,j,k}$ in the pre-trained word embedding matrix $W_{e}$ (i.e, word2vec in our case): $q_{i,j,k}=W_{e}[t_{i,j,k}]$.

Position embedding vector: $p_{i,j,k}$ captures the relative distance $k-k_{i,j}$ from the token $t_{i,j,k}$ to the entity name token $t_{i,j,k_{i,j}}$ in the sentence: $p_{i,j,k}=W_{d}[k-k_{i,j}]$, the $(k-k_{i,j})-th$ column in the position embedding matrix $W_{d}$ ($W_{d}$ is randomly initialized in this work) [Nguyen and Grishman, 2018].

Once each token $t_{i,j,k}$ has been transformed into a vector, the corresponding input sentence $s_{i,j}$ would become a sequence of vector $(x_{i,j,1},x_{i,j,2},\ldots,x_{i,j,K})$. This allows us to view the container $c_{i}$ as an ordered list of vector sequences for its sentences $(s_{i,1},s_{i,2},\ldots,s_{i,L})$.

The attention component processes the list of vector sequences produced in the previous step for $c_{i}$ at two levels (i.e, the word level and the sentence level) to produce a single representation vector for $c_{i}$.

The word level component consumes each vector sequence of $c_{i}$ once at a time to compute the representation vector for the corresponding sentence. For a vector sequence $(x_{i,j,1},x_{i,j,2},\ldots,x_{i,j,K})$ of $s_{i,j}$, the model architecture consists of two LSTMs [Hochreiter and Schmidhuber, 1997] (i.e, the forward LSTM and the backward LSTM) that operate over two different directions of the vector sequence (bidirectional LSTMs) [Cho et al., 2014, Nguyen et al., 2016a]. The resulting hidden vector sequences of the forward and backward LSTMs are concatenated at each position, generating the hidden vector sequence $(h_{i,j,1},h_{i,j,2},\ldots,h_{i,j,K})$ for the input vector sequence of $s_{i,j}$.

In order to combine the hidden vectors $h_{i,j,k}$, the attention mechanism computes the weighted sum of the hidden vectors to obtain a single representation vector for the input sentence $s_{i,j}$ [Bahdanau et al., 2015]. Specifically, each hidden vector $h_{i,j,k}$ is given a weight $\alpha_{i,j,k}$ to estimate its contribution for the final representation of the container $c_{i}$ for the problem of police killing recognition. In this work, the weight $\alpha_{i,j,k}$ is computed by:

$$\alpha_{i,j,k}=\frac{\exp(a_{i,j,k}^{\top}w_{a})}{\sum_{k^{\prime}}{\exp(a_{i,j,k^{\prime}}^{\top}w_{a})}}$$

(1)

where:

$$a_{i,j,k}=\tanh(W_{att}h_{i,j,k}+b_{att})$$

(2)

In such equations, $W_{att}$, $b_{att}$ and $w_{a}$ are the attention parameters at the word level that are learnt in the training process. Eventually, the representation vector $u_{i,j}$ of the sentence $s_{i,j}$ is:

$$u_{i,j}=\sum_{k}\alpha_{i,j,k}h_{i,j,k}$$

(3)

Once the word level component has been completed, every sentence $s_{i,j}$ in the container $c_{i}$ would have a corresponding representation vector $u_{i,j}$. Such sentence representation vectors $u_{i,j}$ altogether form a new sequence of vector $(u_{i,1},u_{i,2},\ldots,u_{i,L})$ to represent the container $c_{i}$. At the sentence level, $(u_{i,1},u_{i,2},\ldots,u_{i,L})$ is processed in the same way we process the vector sequence $(x_{i,j,1},x_{i,j,2},\ldots,x_{i,j,K})$ at the word level to produce the sentence representation vector $u_{i,j}$. In particular, $(u_{i,1},u_{i,2},\ldots,u_{i,L})$ would be first passed to a bidirectional LSTM model to obtain the hidden vector sequence $(h_{i,1},h_{i,2},\ldots,h_{i,L})$. This is in turn fed into the attention component to obtain the attention weights $(\beta_{i,1},\beta_{i,2},\ldots,\beta_{i,L})$ (i.e, similar to Equations 1 and 2). Finally, we compute the vector representation $v_{i}$ for the sentence container $c_{i}$ via the weighted sum: $v_{i}=\sum_{j}\beta_{i,j}h_{i,j}$. As the sentences in $c_{i}$ are sorted by their appearance time, the attentional bidirectional LSTMs at the sentence level attempt to estimate the contribution of each sentence $s_{i,j}$ in $c_{i}$ for the representation vector $v_{i}$ with respect to its past and future context information (i.e, the sentences before and after $s_{i,j}$ in $c_{i}$).

The container representation vector $v_{i}$ for $c_{i}$ allows us to compute the probability $P_{i}$ of $e_{i}$ being killed by police: $P_{i}=P(y_{e_{i}}=1\mid c_{i})=\sigma(W_{out}v_{i}+b_{out})$ where $W_{out}$ and $b_{out}$ are the model parameters, and $\sigma$ is the logistic function. In order to train the hierarchical LSTM model in this section, we use the cross-entropy between the predicted labels and the golden labels as the loss function:

$$L_{c}=-\sum_{e_{i}}y_{e_{i}}\log(P_{i})+(1-y_{e_{i}})\log(1-P_{i})$$

(4)

The previous section has described the supervised attention mechanism for the hierarchical LSTM model. It remains to investigate the methods to obtain the intuitive attention weights. An important characteristics of the intuitive attention weights is that they should assign high weights to the linguistically important words for police killing recognition. In this work, we first start by selecting the important words in the sentences based on our intuition. Afterward, non-negative scores are given to the selected words and their neighbors in the sentences, leaving zero scores for other words. These scores are then normalized to ensure a sum of 1. In order to generate the non-negative scores for the selected words and their neighbors, we employ a Gaussian distribution $\mathcal{N}(\mu,\sigma^{2})$[Mi et al., 2016, Liu et al., 2017b] with the mean $\mu$ and the standard deviation $\sigma$ so that the closer neighboring words would receive higher scores.

This section presents two methods to select important words in the sentences of the containers for the problem of police killing detection. The first method relies on the semantical aspect while the second method concerns the syntactical heuristics. We will compare these methods in the experiments.

Semantical Aspect: The two concepts that are most related to our problem of police killing recognition would naturally be “police” and “killing”. It is therefore intuitive to consider these two words (“police” and “killing”) and their similar ones as the important words for our problem. Consequently, for every sentence in the containers, we search for such important words and use the matched word as the selected words. Following [Keith et al., 2017], we generate the lists of similar words for “police” and “killing” by looking up the nearest words in cosine distance via the word vectors pre-trained with word2vec and its corpus. Note that this method also includes the names of the entities $e_{i}$ in the list of selected important words due to their roles in specifying the entities of interests in the sentences.

Syntactical Aspect: The semantical aspect with the lists of similar words for police killing detection might be helpful for the sentences that express a police killing instance (positive sentences) and contain the similar words. However, for the negative sentences that do not mention police killing incidents, the appearance of the similar words in the lists for “killing” and “police” might be harmful to the supervised attention mechanisms as the emphasis on such words might lead to an incorrect impression to consider the sentence as actually positive. For instance, consider the following negative sentence with the words in the similar word lists written in bold³³3Throughout this work, the names of the entities of interests would be replaced by “TARGET” while the other entity mentions in the sentences would be substituted by “PERSON” for generalization.:

Marion Police Department have arrested TARGET , 20 , of Marion , in connection with the fatal hit.

In this sentence, the extreme emphasis on “Police”, “TARGET” and “fatal” might lead to the incorrect prediction that this sentence is expressing a fatal event caused by police. In order to overcome this issue, we observe that police killing recognition can be seen as a relation identification problem [Bunescu and Mooney, 2005], attempting to decide whether the entities of interests (i.e, the “TARGET”) has a semantical relation of “killed by” with the similar words of “police” (if any) in the sentence or not. In such relation identification problem, it has been shown that the shortest dependency path connecting the two word of interests (i.e, the words “TARGET” and “police” in our case) in the dependency trees involve the most important context words for the problem [Bunescu and Mooney, 2005]. Consequently, in this work, we propose to select the words along the shortest dependency paths between the entity name of interests (i.e, “TARGET”) and the similar words of “police” in the sentence as the guidance words for the supervised attention mechanism for police killing recognition (dependency tress are obtained via Stanford CoreNLP [Schuster and Manning, 2016] in this work).

In the example sentence above, the shortest dependency path between “TARGET” and “police” is: Police $\rightarrow$ Department $\rightarrow$ arrested $\rightarrow$ TARGET. It is clear in this situation that the words along the path do not suggest a police-caused killing (i.e, not a “killed by” relation between “TARGET” and “Police”). Consequently, the attention of the models to such words would lead to a correct prediction in this case.

On the other hand, for the positive sentences, it might be the case that the words along the dependency paths help to include some words that are crucial to police killing prediction, but do not appear in the semantical word lists.

For experimental purposes, we call the hierarchical LSTM model without supervised attention as “H-LSTM”. The hierarchical LSTM model with the supervised attention mechanism would then be called “H-LSTM+SemAtt” and “H-LSTM+SynAtt” depending on whether the semantical aspect (i.e, the lists of similar words) or the syntactical aspect (i.e, the dependency paths) is used to obtain the important words. In order to further demonstrate the benefits of supervised attention for police killing detection, we consider two more baseline models when the model attentions at the word level are excluded from “H-LSTM+SemAtt” and “H-LSTM+SynAtt”, resulting in the models “Mean+SemAtt” and “Mean+SynAtt”. In particular, in such models, the sentence representation vector $u_{i,j}$ for the sentence $s_{i,j}$ would not be obtained via the attention-based weighted sum in Equation 3. In contrast, $u_{i,j}$ would be computed using the mean vector of the vector set $\{h_{i,j,k_{1}},h_{i,j,k_{2}},\ldots,h_{i,j,k_{I}}\}$ from LSTMs where $k_{1},k_{2},\ldots,k_{I}$ are the indexes of the selected important words using the semantical or syntactical aspect for the sentence $s_{i,j}=(t_{i,j,1},t_{i,j,2},\ldots,t_{i,j,K})$: $u_{i,j}=1/I\sum_{m=1}^{I}h_{i,j,k_{m}}$.

Figure 2 indicates the attention weights computed by the models H-LSTM, H-LSTM+SemAtt and H-LSTM+SynAtt for the words in an example sentence. This sentence corresponds to an entity in the test set that is correctly predicted (as being killed by police) by H-LSTM+SynAtt, but is incorrectly predicted by H-LSTM and H-LSTM+SemAtt. As we can see from the figure, H-LSTM fails in this case as it reserves a very high weight for the “TARGET” and essentially ignores the other words. This phenomenon is quite popular for H-LSTM and demonstrates the needs for supervised attention mechanisms as being motivated in the previous sections. In addition, H-LSTM+SemAtt cannot use this sentence as an evidence to make a correct prediction in this case as it mainly attends to the words in the similar word lists (i.e,“TARGET”, “shot” and “police”) and misses the word “death”. This is undesirable as “death” is the only clue showing that the victim of the shooting in this sentence is actually dead. H-LSTM+SynAtt is successful in this case as it is able to assign high weights to such important words along the dependency paths. This demonstrates our arguments in Section 3, showing the benefits of H-LSTM+SynAtt to suggest important words that cannot be captured by H-LSTM+SemAtt for police killing recognition.

The previous part has shown the advantages of H-LSTM+SynAtt over H-LSTM+SemAtt for positive entities. This section focuses on the benefits of H-LSTM+SynAtt for the negative entities. Figure 3 illustrates the attention weights that H-LSTM+SemAtt and H-LSTM+SynAtt assign to the words of an example sentence in the test data.

The entity of this sentence is negative that has been correctly recognized by H-LSTM+SynAtt, but has been incorrectly predicted by H-LSTM+SemAtt. As suggested in the figure, the failure of H-LSTM+SemAtt is due to its very high weights on “shooting”, “TARGET” and “police”, ignoring the effect of the words “said” and “arrested” that clearly negate the involvement of police in this shooting. H-LSTM+SynAtt can attend to such important words as they belong to the dependency paths between “police” and “TARGET” in this case.