End-to-end Semantic Role Labeling with Neural Transition-based Model

We define the transition states as $s=(\sigma_{l},\alpha_{l},\lambda_{p},\alpha_{r},\sigma_{r},\beta,Y)$. We denote that the $\sigma_{l}$ and $\sigma_{r}$ are stacks holding processed arguments in left-side ($l$) and right-side ($r$), respectively, $\alpha_{l}$ and $\alpha_{r}$ are the corresponding stacks temporarily storing the arguments popped out of $\sigma_{*}$ ($*$ $\in$ {$l$,$r$}) which will be pushed back later. $\lambda_{p}$ is a variable referring to a candidate predicate. $\beta$ is a buffer loading the unprocessed words in a sentence. $Y$ is a container storing the output triplets.

We design total six actions as follows.

• •

  NO-PRD, current candidate $w_{i}$ is not a predicate, so moves $w_{i}$ from $\beta$ to $\sigma_{l}$ while popping the top element out of $\sigma_{r}$.

• •

  PRD-GEN, current candidate $w_{i}$ is a predicate (i.e., $p_{j}$), so generate representation for $p_{j}$ onto $\lambda_{p}$.

• •

  LEFT-ARC, the top element in $\sigma_{l}$ is a argument (i.e, $a_{t}$), so assign an arc between $p_{j}$ and $a_{t}$, and pop the top element out of $\sigma_{l}$ into $\alpha_{l}$.

• •

  RIGHT-ARC, the top element in $\sigma_{r}$ is a argument (i.e, $a_{t}$), so assign an arc between $p_{j}$ and $a_{t}$, and pop the top element out of $\sigma_{r}$ into $\alpha_{r}$.

• •

  NO-ARC, no arc between $p_{j}$ and the top element in $\sigma_{l}$ or $\sigma_{r}$, so pop the top elements out of $\sigma_{l}$ or $\sigma_{r}$ into $\alpha_{l}$ or $\alpha_{r}$.

• •

  SHIFT, end of the detection of arguments for $p_{j}$. Pop all the elements out of $\alpha_{*}$ into $\sigma_{*}$, clear $\lambda_{p}$, and moves $w_{i}$ from $\beta$ to $\sigma_{l}$ while popping the top element out of $\sigma_{r}$.

The vanilla transition framework generally process the words in a sentence incrementally from left to right, while such reading scheme may be rigid for SRL parsing. Intuitively, it is always easier for humans to first grasp the core ideas then dig into more details. To this end, we consider a close-first parsing procedure (Goldberg and Elhadad 2010; Cai and Lam 2019; Kurita and Søgaard 2019). We first perform recognition for any possible predicate, e.g., $p_{j}$, and then the process of detecting the arguments of $p_{j}$ starts down the path from near-to-$p_{j}$ to far-to-$p_{j}$, until all the arguments are determined.

Given a sentence, before conducting the transitions, the $\sigma_{r}$ will pre-load all the words from $\beta$ except the first word in reverse order. When a predicate is determined, the system starts searching for its arguments in leftward and rightward alternatively, from near to far. Once an arc is confirmed, a labeler assigns a role label for the argument. It is worth noticing that only a subset of actions are legal to a certain transition state. And invalid actions can lower encoding efficiency, and lead to bad directed graph structure. For example, a NO-PRD action must be followed by either NO-PRD or PRD-GEN action, and *-ARC must starts by a PRD-GEN action. We thus also design some constraints for each action. Figure 2(a) shows the transition process based on the example sentence in Figure 1.

We first use the word form representation under two types of settings: $\bm{v}^{w}_{i},\tilde{\bm{v}}^{w}_{i}$, where $\bm{v}^{w}_{i}$ is the representation initialized with a pre-trained embedding vector for word $w_{i}$, which further can be trained, and $\tilde{\bm{v}}^{w}_{i}$ is the fixed version. We also use convolutional neural networks (CNNs) to encode characters inside a word $w_{i}$ into character-level representation $\bm{v}^{c}_{i}$. In addition, given the input sentence $S$, we exploit the POS tag of the tokens, and use the embedding $\bm{v}^{pos}_{i}$ via a lookup table.

Besides, we employ a TreeLSTM (Tai, Socher, and Manning 2015; Miwa and Bansal 2016) to encode the dependency structural feature representation $\bm{v}^{syn}_{i}$ for each corresponding word $w_{i}$. We concatenate these representations into unified input representation: $\bm{x}_{i}=[\bm{v}^{w}_{i};\tilde{\bm{v}}^{w}_{i};\bm{v}^{c}_{i};\bm{v}^{pos}_{i};\bm{v}^{syn}_{i}]$ Then, a BiLSTM is used as encoder for learning the context representations. We then concatenate two hidden states at each time step $i$ as the sequence representation: $\bm{h}_{i}=\{\bm{h}_{1},\bm{h}_{2},\cdots,\bm{h}_{n}\}$, which will be used for state representation learning.

The actions are predicted based on the neural transition state representations, as depicted in Figure 2(b). We first use a BiLSTM to generate representations $\bm{r}^{\beta}$ for each word $w_{i}$ in $\beta$. Another BiLSTM is used for generating predicate representation $\bm{r}^{\lambda_{p}}$ at $\lambda$. Taking the $9$-th transition step as example, the current representations of predicate ‘stay’ in $\lambda$ and the word ‘witnesses’ in $\beta$ are from BiLSTMs, respectively.

We use Stack-LSTM (Zhang et al. 2019; Yuan, Jiang, and Tu 2019) to encode the elements in stacks (e.g., $\sigma_{*},\alpha_{*}$). For instance, the representations of the top elements in stacks (i.e., ‘When’ in $\sigma_{l}$, ‘witnesses’ in $\alpha_{l}$, ‘nothing’ in $\sigma_{r}$, ‘,’ in $\alpha_{r}$) is obtained from the Stack-LSTM, respectively. Besides of the regular states introduced in $s$, we also maintain the trace of action histories $\delta$ via a stack. We summarize all these state representations via concatenation as the initiation for the next action prediction:

$$\bm{g}_{t}=[\bm{r}^{\sigma_{l}}_{t};\bm{r}^{\sigma_{r}}_{t};\bm{r}^{\alpha_{l}}_{t};\bm{r}^{\alpha_{r}}_{t};\bm{r}^{\lambda_{p}}_{t};\bm{r}^{\beta}_{t};\bm{r}^{\delta}_{t}]\,.$$

(1)

Based on state $\bm{g}_{t}$, the model predicts the action probability, as in Figure 2(b):

$$\bm{P}^{a}=\text{softmax}(\text{FFNs}(\bm{g}_{t}))\,,$$

(2)

where FFNs($\cdot$) is a feed-forword network. At the meantime, once an arc is determined, a labeler¹¹1 In traditional transition methods, the arcs are determined jointly with their role labels by joint actions, e.g., LEFT-ARC-A1, whereas we find in our preliminary experiments that separating these two predictions can bring better performances. will assign a role label for the argument:

$$\bm{P}^{r}=\text{softmax}(\text{FFNs}(\bm{g}_{t}))\,.$$

(3)

During training, we set the goal to maximize the likelihood of actions from the current states under the gold-standard actions. We first convert gold output structures of training data into action sequences. We minimize the following loss:

$$\mathcal{L}=-\begin{matrix}\sum_{t}\end{matrix}\hat{\varphi}_{t}\text{log}p(\varphi_{t}|\Theta)+\frac{\zeta}{2}||\Theta||^{2}\,,$$

(4)

where $\hat{\varphi}$ denotes gold-standard actions, $\Theta$ is the set of all model parameters, and $\zeta$ is a co-efficient. Instead of greedily choosing the output with maximum probability, we apply beam-search algorithm (Zhang and Clark 2008; Lyu, Zhang, and Ji 2016) for decoding, where top-B partial outputs are cached for each transition decision, avoiding achieving sub-optimal results.

When removing the character encoder, POS tags and syntactic dependencies, respectively, the performances drop consistently. We notice that the dependency structural knowledge is of the greatest importance for both the argument and predicate recognition, among other features, which coincides with the prior findings of the structural syntax information (Marcheggiani, Frolov, and Titov 2017; Zhang, Wang, and Si 2019; Fei et al. 2020).

We ablate the argument-predicate scoring distribution $\bm{o}^{a}$ and $\bm{o}^{r}$ to see the contribution of the proposed interaction mechanism. We can find that both these two distributions plays key role for arguments role labeling, especially the $\bm{o}^{r}$. When removing such incremental high-order feature away (i.e., vanilla model w/o $\bm{o}^{a}$&$\bm{o}^{r}$), the results drop significantly. This verifies the usefulness of the proposed high-order feature.

Our model performs argument parsing around the predicate down the ‘from-near-to-far’ direction. Compared with the traditional ‘left-to-right’ reading order, or the reversed ‘right -to-left’ order, we can find that the close-first parsing scheme is much more effective.

We mainly make comparisons with the recent previous end-to-end SRL models. In Table 1, first of all, we find that the syntactic dependency feature configurations universally contribute to both the argument recognition and predicate disambiguation, compared with the syntax-agnostic models. Most importantly, our high-order model outperforms these baselines on both two subtasks. The improvements are more significant on the setting with dependency features, i.e., 90.2% (Arg.) and 95.5% (Prd.). Also we notice that our model is especially better at predicate disambiguation. Besides, our transition systems beat competitors with higher decoding speed, nearly two times faster.

Table 3 shows the performances on UPB for other languages. The experiments are conducted with dependency features. Overall, our high-order model wins the best averaged F1 score (74.2%). Also the high-order feature universally helps to improve the vanilla transition model with average 1.3%(=74.2-72.9) F1 score. The improvements so far by our model demonstrate its effectiveness.

Table 4 compares the performances on out-of-the-domain test set of CoNLL09 data with dependency features. Overall, the similar trends are kept as that on in-domain dataset. Our model still performs the best, yielding 80.0% (Arg.) and 82.7% (Prd.) F1 scores, verifying its generalization capability on capturing the semantic preferences of the task.

We next compare the results in Table 5 for standalone argument role labeling, where the models are given by the gold pre-identified predicates as input, and output the argument boundaries and roles. Compared with the baseline pipeline systems, the end-to-end systems can perform better, while our transition model gives a new state-of-the-art 90.7% F1 score.

Finally, we take advances of the recent contextualized word representations, ELMo, BERT and XLNet. As shown in Table 6, all the end-to-end SRL models can benefit a lot. With enhanced word representations, we see that the graph-based model by Li et al. (2019) can achieve better results (91.5%) with ELMo. while our model can obtain the best performances by BERT (92.2%), and XLNet (92.0%).

We introduce the incremental high order interaction for improving the argument recognition, by informing the detection of following and farther arguments with previous recognition history information, that is, the later decisions are influenced by former decisions. Such uni-directional propagation naturally spurs a potential question: Is it necessary to use the later refined decisions to inform and refine back the former decisions? Theorically speaking, the negative influences only exist at the initial transition decisions, because our proposed interaction mechanism combined with beam search strategy can largely refined the future decisions. One the other hand, the close-first parsing scheme can ensure that the nearer-to-predicate arguments recognized earlier are more likely to be correct. In addition, in our experiment we proves the unnecessity of the back refining. Figure 4 shows the performance under varying surface distance from predicate to arguments. It is clear that within the distances of 3-4, the performances by our full model changes very little. Even when removing the interaction mechanism (into vanilla model), the decreases under short surface distances is not much significant. Lastly, our high-order model handles better the long-distance dependency issues, compared with baselines.

We now look into the incremental high order interaction mechanism. We consider conducting empirical visualization of the transition parsing steps. We compare the transition results by high-order model, and the vanilla model without high-order feature, as illustrated in Figure 5. At the earlier steps with short distance between predicate and argument, e.g., second and fourth iteration steps, both two transition models can make correct prediction. At 7${}^{\text{th}}$ step, the vanilla model wrongly assigns a A3 for argument ‘you’, and further falsely determines the ‘sometime’ as AM-TMP argument at 8${}^{\text{th}}$ step. On the contrary, our high-order model with such interaction can perform correct role labeling, and yield accurate argument detection even in remote distance.

To further explore the strengths of our transition model, we study the argument role violation (Punyakanok et al. 2004; FitzGerald et al. 2015; He et al. 2018). Consider categorizing the role labels into three types: (U) unique core roles (A0-A5, AA), (C) continuation roles and (R) reference roles. 1) If a core role appears more than once, U is violated. 2) If C-X role not precedes by the X role (for some X), C is violated; 3) R is violated if R-X role does not appear. Table 7 shows the role violation results.

First of all, our high-order model gives the most consistent results with gold ones, with minimum role violations. This partially indicates that the system can learn the latent constraints. But without the incremental high order interaction, performances by vanilla model for continuation and reference roles get hurt. This is reasonable, since generally the non-core roles are more remote from their predicate. We also find that our transition model with close-first parsing order performs better, especially for the unique core role, compared with the ‘left-to-right’ parsing order. Finally, the graph-based baselines tend to predict more duplicate core arguments, and also recognize fewer reference argument roles.