A Boundary Regression Model for Nested
Named Entity Recognition

In a neural network model, the values of inputs represent the tense of signals. Therefore, words in a sentence are traditionally represented as high-dimensional one-hot vectors. At current, deep neural network has two advantages to support automatically extracting semantic features from raw inputs: word embedding and feature transformation. Word embedding is applied to map every word into a distributed representation, which encodes semantic information learned from external resources. In feature transformation, many types of neural layers (CNN [21], LSTM [15] or Attention [44]) can be stacked to support designed feature transformation for capturing syntactic and semantic features of a sentence, which avoid the need for manually feature engineering.

For feature extraction, there are two differences between object detection and NE recognition. First, the detection of objects is mainly based on internal features of objects. Therefore, an object moving in an image exerts less influence on the object detection. However, entities have a strong semantic dependency in a sentence. Second, in nested objects, features in the bottom object are blocked by upper objects, which makes challenge for bottom object detection. While nested NEs share the same context in a sentence. It is important to lean dependent features relevant to considered NEs. Therefore, comparing with object detection, encoding semantic dependencies in a sentence is more important for NE recognition. In deep architectures, a recurrent neural network or an attention network is helpful to capture the semantic dependency between words.

In our BR model, we adopted a basic network consisting of an embedding layer and a Bi-LSTM layer. The embedding layer is adopted to map a sentence into a distributed representation, where words (or characters) are embedded into vectors by a lookup table pretrained with unsupervised algorithms. Then, a Bi-LSTM is implemented to encode the semantic dependencies in a sentence. To simplify the region proposal step, we set the length of the input sentence as a fixed number, denoted as $L$. Longer or shorter sentences are trimmed or padded, respectively.

The output of the basic network is denoted as recu feature maps, where a feature map is an abstract token representation of the input. In the BR model, it also denotes as the feature map layer, which represent high-order abstract features of a sentence integrated with dependent semantics between words. In computer vision, images have an invariance property for a zooming operation. Object detection can benefit from multigranularity representations, where the region proposal can be implemented on multiscale feature maps to generate bounding boxes with different scales. In natural language processing, it is difficult to condense a textual sequence into multigranularity representations. At present, for each input sentence, we generate a single recu feature map layer, which can be seen as a high-order abstract representation of an input sentence. Each feature map position corresponds to a possible entity boundary. Because we adopt continuous location representation, the position of feature maps is normalized into the interval $[0,1]$ to support regression operation.

The feature map layer is mainly applied to support region proposal for generating abstract NE representations. Instead of directly generating NE candidates from a sentence (e.g., Sohrab et al. [39]), generating abstract NEs from the feature map layer can share parameters in the bottom network. It reduces the computational complexity and enables more potent nonlinear function approximators to enhance model discriminability.

A feature map corresponds to an abstract representation of a possible NE boundary in a sentence. Each feature map can be set as a start position and combined with right feature maps to generate bounding boxes with different lengths. In this paper, for every feature map, we enumerate $K$ bounding boxes from left to right. The value $K$ is a predefined parameter indicating the longest NE candidate. It is similar to an exhaustive enumeration method, which verifies every possible NE candidate up to a certain length (e.g., Sohrab et al. [39]). The difference is that bounding boxes are referred by their spatial locations in a sentence, which can be used to filter bounding boxes those are unlikely to be a truth box (discussed in Section III-D). It reduces the computational complexity and decreases the influence caused by negative examples.

To show the potential ability of the BR model to locate NEs those are missed in the region proposal process, in this experiment, we also propose an interval enumeration strategy. For every feature map, we enumerate bounding boxes from left to right with lengths [1, 3, 5, 7, 11, 15, 20]³³3We set the longest length of NEs to be 20. They cover 99.05% of NEs in the Chinese ACE corpus.. In the training process, all ground truth boxes in training data are included to train the classifier. In the testing process, only bounding boxes with lengths [1, 3, 5, 7, 11, 15, 20] are verified. Comparing with exhaustively enumeration with lengths from 1 to 20, the interval enumeration reduces about 65% computational overhead. For convenience, we refer to the BR model with interval enumeration as “BR${}_{\text{int}}$”. The BR model implemented on exhaustive enumeration is referred to as “BR${}_{\text{exh}}$”.

A bounding box is a high-order abstract representation of a possible NE generated from feature maps by region proposal. Because feature maps are transformed by a basic network which consists of a recurrent neural network or an attention network, each bounding box contains contextual features about a possible NE. Using class labels and location parameters of bonding boxes, a softmax layer and a linear layer can be set to predict their class probabilities and learn the location offset relative to a ground truth box. In the following, we give formal definitions about the bounding box.

Let $\mathbf{D}=\{d_{1},d_{2},\cdots,d_{M}\}$ denote a bounding box set generated from an input sentence $S$. $M$ is the size of $\mathbf{D}$. Each bounding box $d_{i}\in\mathbf{D}$ has 3 parameters: $d_{i}^{s}$, $d_{i}^{l}$ and $\mathbf{c}_{i}$. Parameters $d_{i}^{s}$ and $d_{i}^{l}$ are two real numbers denoting the start position and length of $d_{i}$ in a sentence, respectively. The end position of $d_{i}$ can be computed as $d_{i}^{s}+d_{i}^{l}$. Parameter $\mathbf{c}_{i}=(c_{i}^{1},c_{i}^{2},\cdots,c_{i}^{Z})$ $(c_{i}\in\{0,1\})$ is a one-hot vector representing the entity type of $d_{i}$, where $Z$ is the number of entity types. Therefore, a bounding box $d_{i}$ can be referred to as a three-tuple $d_{i}=\langle d_{i}^{s},d_{i}^{l},\mathbf{c}_{i}\rangle$. If a bounding box corresponds to a true NE, it is referred to as a ground truth box and represented as $g_{j}=\langle g_{j}^{s},g_{j}^{l},\mathbf{c}_{j}\rangle$.

Bounding boxes are labelled with location parameters. Borrowed from the intersection over union (IoU) developed in computer vision [8], the overlapping ratio between two bounding boxes can be measured by the IoU value.

Let $d_{i}$ and $g_{j}$ be a bounding box and a ground truth box, respectively. The IoU value between them is computed as:

where the function $span(d_{i})$ represents the range of a bounding box in feature maps. If a bounding box has a large IoU value, it is highly overlapped with a ground truth box. A high overlapping ratio indicates that a bounding box contains adequate contextual features about a true NE, which guarantees learning of the location offset relevant to a truth box. Otherwise, if the IoU value of a bounding box is lower than a predefined threshold, it denotes a false NE. It is used to train a classifier for identifying false NEs.

Let $\mathbf{D}_{G}$ represent the set of all ground truth boxes in $\mathbf{D}$. We define two sets as follows:

where $\gamma$ is a predefined threshold. $\mathbf{D}_{G}$ is a subset of $\mathbf{D}_{p}$, where $\gamma$ is equal to 1.

In this paper, $\mathbf{D}_{p}$ is referred to as the positive bounding box set, and $\mathbf{D}_{n}$ is referred to as the negative bounding box set. In region proposal, a large number of negative bounding boxes will be generated, which lead to a significant data imbalance problem. This is also computationally expensive. In the training process, we collect $\mathbf{D}_{p}$ and $\mathbf{D}_{n}$ with a ratio of 1:3 for balancing positive and negative samples. This guarantees faster optimization and a stable training process.

Given a bounding box $d_{i}$, its relative ground truth box is identified as:

Given a ground truth box $g_{j}$, all bounding boxes of $\mathbf{D}_{p}$ satisfying Equation 3 are referred to as $\mathbf{D}_{g_{j}}$. They are the neighbourhoods of $g_{j}$. This is formalized as:

It is important to know that, in the training data, all bounding boxes in $\mathbf{D}_{g_{j}}$ are labelled with a positive class tag the same as $g_{j}$. This labelling strategy is different from the traditional method in which, if the start and end boundaries of an NE candidate are not precisely matched to a true NE, it is labelled with a negative class tag. The reason for this will be discussed in detail in Section III-E. For consistency, in this paper, we use the term “positive box” referring to a bounding box with an IoU value relevant to a ground truth box larger than $\gamma$. The term “truth box” refers to a bounding box, which has a location precisely matched to a real NE.

Based on $\mathbf{D}_{G}$ and Equation 3, $\mathbf{D}_{p}$ can be partitioned into a set $\mathcal{D}_{p}=\{\mathbf{D}_{g_{1}},\mathbf{D}_{g_{2}},\cdots\}$. Therefore, $\mathcal{D}_{p}$ is a partition of $\mathbf{D}_{p}$. If $i\neq j$, then $\mathbf{D}_{g_{i}}\cap\mathbf{D}_{g_{j}}=\emptyset$. Every bounding box $d_{i}\in\mathbf{D}_{p}$ belongs to a $\mathbf{D}_{g_{j}}\in\mathcal{D}_{p}$.

For convenience, Table I lists the definitions about different bounding box sets. Their roles to support boundary regression will be discussed in the following subsection.

Let $\hat{d}_{ij}^{s}=(g_{j}^{s}-d_{i}^{s})/g_{j}^{l}$ and $\hat{d}_{ij}^{l}=log(g_{j}^{l}/d_{i}^{l})$ be the normalized position offset and shape offset between $d_{i}$ and $g_{j}$. Given a bounding box $d_{i}$, the BR model outputs 3 parameters: $\Delta d_{i}^{s}$, $\Delta d_{i}^{l}$ and $\tilde{\mathbf{c}}_{i}$. $\Delta d_{i}^{s}$ and $\Delta d_{i}^{l}$ denote the predicted position offset and shape offset of $d_{i}$ relative to $g_{j}$. $\tilde{\mathbf{c}}_{i}$ is a confidence score that reflects the confidence that a box contains a true NE. As Figure 3 shows, $\Delta d_{i}^{s}$ and $\Delta d_{i}^{l}$ are regressed by a linear layer, while the classification confidence score $\tilde{\mathbf{c}}_{i}=(\tilde{c}_{i}^{0},\tilde{c}_{i}^{1},\cdots,\tilde{c}_{i}^{Z})$ is predicted by a softmax layer.

For every $d_{i}\in\mathbf{D}_{p}$, the location offset of $d_{i}$ relates to a ground truth box that is predicted by a linear layer. A characteristic function $E_{ij}^{z}=\{0,1\}$ is defined to indicate that a default box $d_{i}$ is matched to a relative ground truth box $g_{j}$ selected by Equation 3. In the training process, the regression operation updates $\Delta d_{i}^{s}$ and $\Delta d_{i}^{l}$ for the purpose of approaching $\hat{d}_{ij}^{s}$ and $\hat{d}_{ij}^{l}$, respectively. The location loss can be computed as:

where $N=|\mathbf{D}_{g_{j}}|$ is the cardinality of $\mathbf{D}_{g_{j}}$. It is used to normalize the weight between $g_{j}\in\mathbf{D}_{G}$. $\text{Smooth}_{L_{1}}$ is a robust $L_{1}$ loss that quantifies the dissimilarity between $d_{i}$ and $g_{j}$. It is less sensitive to outliers [13].

Equation 5 shows that only the positive bounding box set $\mathbf{D}_{p}$ is adopted to compute the location loss. For every ground truth box $g_{j}\in\mathbf{D}_{G}$, its neighbourhoods $\mathbf{D}_{g_{j}}$ are used to generate the location loss. This setting is natural because neighbourhoods contain sufficient contextual features about an NE to support boundary regression. On the other hand, negative bounding boxes are far from any ground truth box. Because the vanishing gradient problem, it is difficult to precisely regress their location offsets.

When minimizing the location loss, bounding boxes belonging to $\mathbf{D}_{g_{j}}$ will approach the ground truth box $g_{j}$. Therefore, all bounding boxes in $\mathbf{D}_{g_{j}}$ are given a class tag which is the same as ground truth box $g_{j}$.

Confidence loss is a softmax loss over multiple class confidences. It is given as follows:

where $\tilde{c}_{i}^{z}=exp(\tilde{c}_{i}^{z})/\sum_{z=0}^{Z}exp(\tilde{c}_{i}^{z})$, $\tilde{c}_{i}^{0}$ is the confidence score indicating that an example is negative. A key issue about the classification is that the confidence score should be estimated based on NE representations with refined spatial locations in a sentence.

The total loss function combines the location loss and confidence loss:

where $\alpha$ is a predefined parameter balancing the weight between the location loss and confidence loss. The training objective is to reduce the total loss of the location offset and class prediction. In the training process, we optimize their locations to improve their matching degree and maximize their confidences.

In the prediction process, the BR model outputs a set of bounding boxes for each input sentence, referred to as $\mathbf{D}=\{d_{1},d_{2},\cdots,d_{M}\}$. Every box $d_{i}\in\mathbf{D}$ has 3 outputs: $\Delta d_{i}^{s}$, $\Delta d_{i}^{l}$ and $\tilde{\mathbf{c}}_{i}$, respectively indicating the position offset, shape offset and class probability of $d_{i}$ relative to a truth box. After $\Delta d_{i}^{s}$ and $\Delta d_{i}^{l}$ are resized as $\Delta s_{i}$ and $\Delta l_{i}$, an predicated NE can be located as: [$s_{i}+\Delta s_{i}$, $l_{i}+\Delta l_{i}$, $\tilde{c_{i}}$]⁴⁴4An example of the output is shown in Figure 3..

The output $\mathbf{D}$ contains a large number of boxes, but many of them are overlapped. Non-maximum suppression (NMS) is implemented in the prediction process to produce the final decision, which selects truth boxes from overlapped neighbourhoods. The NMS algorithm is shown in Table II.

It is a one-dimensional NMS algorithm that selects nested NEs from overlapped positive boxes. The NMS algorithm searches local maximized elements from overlapping neighbourhoods in which a smaller number of high-confidence boxes are collected. The threshold is adopted to control the overlapping ratio between neighbourhoods. In our experiments, the value of $\lambda$ is set as 0.6.

To show the superiority of our model, we first compare the BR model with related work. It is first evaluated on the Chinese corpus. Then, to show the extensibility of the BR model, the BR model is transformed to the English corpus for further assessment.

In natural language processing, continuous location representation has not been used denoting to positions of linguistic units in a sentence. Therefore, the regression operation is rarely used to support information extraction. As we known, the BR model is the first attempt to locate linguistic units in a sentence by regression operation. To analyse the mechanism of boundary regression for nested NE recognition, we design a traditional NE classification model named as bounding box classifier (BBC) for comparison. It is generated by omitting the linear layer from the deep architecture in Figure 3. In the output, only a softmax layer is adopted to predict the class probability for every bounding box.

In this section, three experiments are conducted to show the usefulness of the regression operation. We first conduct two ablation studies to show the feasibility of boundary regression. In the first experiment, exhaustive enumeration is adopted in region proposal. The BBC model and the BR model are compared to show the ability of BR to refine spatial locations of NEs in a sentence. In the second experiment, the BR model is implemented on intermittently enumerated bounding boxes. The experiment shows the ability of the BR model to locate true NEs from mismatched NE candidates. The BR model is mainly designed to supported nested NE recognition. It also can be used to recognize flattened NEs. Therefore, in the third experiment, we evaluate the BR model on flattened NE recognition. The first experiment and the second experiment are conducted on the ACE Chinese corpus. The third experiment is implemented on two English corpora with flattened NE annotation: the OntoNotes 5.0 [32] corpus and the CoNLL 2003 [35] corpus.

Because the IoU value and the NMS algorithm are influential on the BR model, in this section, we conduct experiments to analyse the influences of IoU and NMS.

In object detection of computer vision, compared with multistage pipeline models (e.g., R-CNN [14]), an end-to-end framework is hailed due to the superiority of high speed (e.g., Faster R-NN [34]). The reason for this is that the background of an image is learned in a single pass in the training process. It is shared by all proposed regions in an image.

To show the time complexity of boundary regression, in this experiment, we compare our model with those of Zheng et al. [52] and Wang et al. [46]. Zheng et al. [52] present a boundary-aware neural model which detects entity boundaries. Boundary-relevant regions are then utilized to predict entity categorical labels. The boundary detection and region prediction share the same bidirectional LSTM for feature extraction. Wang et al. [46] present a pyramid-shaped model stacked with neural layers. This model directly implements NE span prediction. Therefore, it has a higher speed.

In this experiment, we implement these models on the ACE English corpus with the same data split, settings and GPU platform. The times required to train these models are shown in Figure 7, where the height of the histograms represents the time cost in seconds.

In comparison with them, boundary regression achieves the least time complexity. The BR model has two characteristics which support high speed recognition. First, feature maps are generated from a basic network. They are shared by all bounding boxes in a sentence. In fact, all bounding boxes are mutually overlapped. They are parts of feature maps, which considerably reduce model parameters. Second, every bounding box has location parameters. Therefore, in the learning process, the IoU value can be adopted to filter negative bounding boxes. This strategy is effective to reduce the time complexity.

For a better understanding of boundary regression and investigating more details of the BR model, in the follows, we present a visualization about boundary regression.

A sentence “埃及是中东地区最重要的国家”⁶⁶6It can be translated as: “Egypt is the most important country in the Middle East area”. is selected from the testing data. It contains four nested NEs: “埃及” (Egypt, GPE), “中东地区最重要的国家” (the most important country in the Middle East area, GPE), “中东地区” (the Middle East area, LOC), and “中东” (the Middle East, GPE). A bounding box is denoted by 3 parameters $s_{i}$, $l_{i}$ and $c_{i}$, which represent the starting position of the box, the length of the box and the class probability of the box, respectively. To visualize bounding boxes, a bounding box is drawn as a rectangle. The horizontal ordinate represents the boundary positions of the bounding boxes in a sentence, which are normalized to $[0,1]$. The vertical coordinate represents the classification confidence score. The colours of the bounding box represent NE types. To generate bounding boxes, the selected sentence is predicted by a pre-trained BR model. All outputted bounding boxes are collected and drawn with respect to the sentence. The result is shown in Figure 8.

In Figure 8(a), bounding boxes were predicted by the BR model without training (0 iterations). From Figure 8(b) to Figure 8(f), the BR model is trained with different rounds (denoted in the titles of subfigures). Because the regression operation may output negative values for parameters $s_{i}$ and $l_{i}$, we filter bounding boxes with $l_{i}\leq 0$ or $s_{i}+l_{i}>1$ (beyond the sentence range).

In Figure 8(a), there is no tendency between bounding boxes. They are distributed evenly across the whole sentence and all NE types. In Figure 8(b), the BR model is implemented on the training data in only one round. One interesting phenomenon is that red bounding boxes and blue bounding boxes are grouped around NEs quickly. Furthermore, other true entity types are appropriately depressed. From Figure 8(c) to Figure 8(f), when the number of iterations is increased, there are two tendencies in bounding boxes. First, the BR model becomes more confident in the entity type prediction, which increases the classification confidence of the bounding boxes. Second, the locations of bounding boxes are approaching the true NEs. This indicates that the regression operation to locate NEs is feasible.

Overlapped bounding boxes are the key to solving the nested NE problem. Figure 8(f) shows that nested NEs are distinguished appropriately. We have tracked several bounding boxes and found that bounding boxes do not approach true NEs smoothly and directly. There are some oscillations between them. In the training process, a bounding box may match the true NE perfectly, while moving away in the next iteration. However, in accordance with the increase in the number of training steps, the oscillation tends toward stability.