RAPS: A Novel Few-Shot Relation Extraction Pipeline with Query-Information Guided Attention and Adaptive Prototype Fusion

We employ BERT (Devlin et al., 2019) as the encoder to map the instances into a low-dimensional vector space and better capture the semantic information of support set $\mathcal{S}$ and $\mathcal{Q}$. For instances in $\mathcal{S}$ and $\mathcal{Q}$, we concatenate the hidden states corresponding to start tokens of two entity mentions following Baldini Soares et al. (2019), i.e., $h=\text{concat}(h_{1},h_{2})\in\mathbb{R}^{2d},$ where $h_{i}\in\mathbb{R}^{d},i=1,2$ and $d$ is the size of the representation vector (or euqally, the hidden size of BERT output), as the instance representation.

For each relation, we combine its name and description and feed the sequence into the BERT encoder. We treat the embeddings of the [CLS] tokens $\left\{\mathbf{r}_{i}^{1},1\leq i\leq N\right\}$, and the average embeddings of all tokens $\left\{\mathbf{r}_{i}^{2},1\leq i\leq N\right\}$, as two components of the relation representation. Then, the final representation of each relation is derived by direct concatenation of $\mathbf{r}_{i}^{1}$ and $\mathbf{r}_{i}^{2}$, i.e.,

$$\mathbf{r}_{i}=\text{concat}(\mathbf{r}_{i}^{1},\mathbf{r}_{i}^{2})\in\mathbb{R}^{2d}.$$

Traditionally, the naïve prototype for relation $i$ is computed simply by averaging the representations of the $K$ instances under relation $i$ in $\mathcal{S}$ as

$$\mathbf{p}_{i}=\frac{1}{K}\sum_{k=1}^{K}\mathbf{s}_{k}^{i},$$

(1)

where $\mathbf{s}_{k}^{i}$ is the embedding of the $k$th support instance of relation $i$. However, simply averaging all support instances as the relation prototype regardless of the query instances will lose abundant interaction between them. In other words, different relation prototypes will be computed based on the semantic relevance between the support instances and the query instances.

In order to generate more informative prototypes, we propose a query-information guided attention module. This specific module aims to utilize the semantic relevance between the support instances and the query instances to help compute the customized relation prototypes. Specifically, the prototype can be represented as:

$$\mathbf{p}_{i}=\sum_{k=1}^{K}\alpha_{k}^{i}\mathbf{s}_{k}^{i},$$

(2)

where $\mathbf{p}_{i}$ is the attention-based prototype of relation $i$, $\mathbf{s}_{k}^{i}$ is the representation of the $k$th support instance of relation $i$, $\alpha_{k}^{i}$ is the weight indicating the semantic relevance between the $k$th support instance of relation $i$ and the query instances. The weight $\alpha_{k}^{i}$ is computed by

$$\alpha_{k}^{i}=\frac{\exp\left(-\frac{1}{|\mathcal{Q}|}\sum_{j=1}^{|\mathcal{Q}|}d(\mathbf{s}_{k}^{i},\mathbf{q}_{j})\right)}{\sum_{k=1}^{K}\exp\left(-\frac{1}{|\mathcal{Q}|}\sum_{j=1}^{|\mathcal{Q}|}d(\mathbf{s}_{k}^{i},\mathbf{q}_{j})\right)},$$

(3)

where $\mathbf{q}_{j}$ is the embedding of the $j$th query instance in the query set, $|\mathcal{Q}|$ is the total number of instances in $\mathcal{Q}$, and $d(\cdot,\cdot)$ means the Euclidean distance. We call the prototype obtained in this way Initial Prototype, which is able to store some knowledge about the query. The benefit of the initial prototype for subsequent classification will be proved in Subsection 4.4.

Inspired by the adaptively spatial feature fusion approach proposed by Songtao Liu and Wang (2019), we design a novel adaptive prototype fusion mechanism to obtain the integrated prototype. Specifically, the integrated prototype $\mathbf{p}_{i}^{w}$ is an weighted average of $\mathbf{p}_{i}$ and $\mathbf{r}_{i}$:

$$\mathbf{p}_{i}^{w}=w_{1}\mathbf{p}_{i}+w_{2}\mathbf{r}_{i},$$

(4)

where $w_{1},w_{2}\in\mathbb{R}$ are two learnable weights, which depend on the representation of category information to ensure that the calculated integrated prototype $\mathbf{p}_{i}^{w}$ is compatible and not redundant. Unlike Songtao Liu and Wang (2019) forces the sum of the weights equal to 1, our adaptive prototype fusion has no constraint on $w_{1}$ and $w_{2}$. This provides more degrees of freedom to learn from data and adjust the weights. The benefits of mechanism will be demonstrated in Subsection 4.5.

With the representation of query and integrated prototypes of $N$ relations, the model uses the vector dot product way to compute the probability of the relations for the query instance representation $\mathbf{q}_{j}$ as follows:

$$P(y=i|\mathbf{q}_{j})=\frac{\exp(\mathbf{q}_{j}\cdot\mathbf{p}_{i}^{w})}{\sum_{n=1}^{N}\exp(\mathbf{q}_{j}\cdot\mathbf{p}_{n}^{w})}.$$

The training loss $\mathcal{L}$ is defined as regular cross entropy loss as follows:

$$\mathcal{L}=-\sum_{j=1}^{|\mathcal{Q}|}\log\frac{\exp(\mathbf{q}_{j}\cdot\mathbf{p}_{i}^{w})}{\sum_{n=1}^{N}\exp(\mathbf{q}_{j}\cdot\mathbf{p}_{n}^{w})}.$$

In the prediction stage, query $\mathbf{q}_{j}$ is assigned to relation $i$ with the highest probability:

$$\text{label}=\arg\max_{i}P(y=i|\mathbf{q}_{j}).$$

Datasets We use FewRel 1.0 (Han et al., 2018) and FewRel 2.0 (the domain adaption portion) (Gao et al., 2019b) to evaluate our model. FewRel 1.0 is a large-scale FSRE dataset, which contains 100 relations with 700 instances each relation. We follow the official split to use 64, 16 and 20 relations for training, validation, and testing. In order to study the domain transferability of RAPS, we also evaluate our model on FewRel 2.0. The training set of FewRel 2.0 is the same as FewRel 1.0, and the validation set is collected from the biomedical domain which contains 10 relations and 100 instances for each relation. We evaluate our model on FewRel 1.0 and FewRel 2.0 in terms of the accuracy under multiple $N$-way $K$-shot meta tasks. We select $N$ to be 5 and 10, $K$ to be 1 and 5 to form four test scenarios according to Gao et al. (2019a).

Training We use BERT${}_{\texttt{BASE}}$ and CP (Wang et al., 2020) as the sentence encoder, and set the total train iteration number as 30,000, validation iteration number as 1,000, batch size as 4, learning rate as $1\times 10^{-5}$ and $5\times 10^{-6}$ for BERT and CP respectively.

Table 2 presents the experimental results on FewRel 1.0 validation set and test set. As shown in the upper part of Table 2, our method outperforms the strong baseline models significantly. To be specific, RAPS achieves the average of 1.19 points improvement in terms of accuracy on the test sets of four meta tasks, compared to the second best method (HCRP), demonstrating the superior generalization ability. In addition, we evaluate our approach based on the model CP (Peng et al., 2020), where the BERT encoder is initialized with the pre-trained parameters by a contrastive pre-training approach. The lower part of Table 2 shows that our approach achieves a consistent performance boost when using CP pre-trained model. It is worth to mention that RAPS (CP) has 1.17 and 1.24 points improvement compared with HCRP (CP), under the 5-way 1-shot and 10-way 1-shot scenarios, respectively, which demonstrates that our approach is more suitable for few-shot scenarios. Our method also achieves the competitive performance compared with HCRP on FewRel 2.0, as shown in Table 3, which shows that RAPS is also transferable to the domain adaptation setting. It can be seen that RAPS is overall better than HCRP with the CP as encoder. The possible reason why RAPS is worse than HCRP when using BERT as encoder is that the FewRel 2.0 with domain adaptation setting only provides the name of relations without a specific description, which has a relatively harmful impact on the adaptive fusion mechanism to generate a strong relation representation for the relation prototypes. Nevertheless, HCRP seems to be more robust to the lack of relation description due to the complex hybrid prototype learning module.

In this subsection, we further explore four different types of adaptive prototype fusion methods, including:

• •

  Unconstrained Adaptive Scalar (UAS): $\mathbf{p}_{i}^{w}=w_{1}\mathbf{p}_{i}+w_{2}\mathbf{r}_{i},w_{1},w_{2}\in\mathbb{R}$.

• •

  Constrained Adaptive Scalar (CAS): $\mathbf{p}_{i}^{w}=w_{1}\mathbf{p}_{i}+w_{2}\mathbf{r}_{i},w_{1},w_{2}\in\mathbb{R},\text{subject to}\,w_{1}+w_{2}=1$.

• •

  Unconstrained Adaptive Matrix (UAM): $\mathbf{p}_{i}^{W}=\mathbf{W}_{1}\mathbf{p}_{i}+\mathbf{W}_{2}\mathbf{r}_{i},\mathbf{W}_{1},\mathbf{W}_{2}\in\mathbb{R}^{2d\times 2d}$.

• •

  Constrained Adaptive Matrix (CAM): $\mathbf{p}_{i}^{W}=\mathbf{W}_{1}\mathbf{p}_{i}+\mathbf{W}_{2}\mathbf{r}_{i},\mathbf{W}_{1},\mathbf{W}_{2}\in\mathbb{R}^{2d\times 2d},\text{subject to}\,\mathbf{W}_{1}+\mathbf{W}_{2}=\mathbf{I}_{2d}$.

For UAS and CAS, $w_{1}$ and $w_{2}$ are two learnable scalars. For UAM and CAM, $\mathbf{W}_{1}$ and $\mathbf{W}_{2}$ are two learnable matrices, $\mathbf{I}_{2d}$ is the identity matrix of order $2d$. Note that UAM and CAM are equivalent to add a fully-connected layer without bias term on $\mathbf{p}_{i}$ and $\mathbf{r}_{i}$, respectively. We evaluate these four types of adaptive prototype fusion methods on FewRel 1.0 validation set. The results are shown in Table 4. We can see that UAS gets the best performance in all scenarios. Compared with CAS, UAS has more degrees of freedom to learn the trade-off between relation prototype and relation information, thus leading to more accurate prototype and better performance than CAS. It can be seen that matrix-based fusion methods (UAM and CAM) are consistently inferior to scalar-based fusion methods (UAS and CAS), which may attribute to the introduction of useless parameters ($O(d^{2})$ for UAM and CAM vs. $O(1)$ for UAS and CAS).

In this subsection, we conduct an ablation study on 5-way 1-shot and 10-way-1-shot based on BERT with the validation set, to demonstrate the effectiveness of the proposed query-information guided attention module and adaptive prototype fusion mechanism (abbreviated as QIA and APF for further reference, respectively). We consider three ablation experiments including w/o QIA, w/o APF, and w/o QIA and APF.

• •

  w/o QIA: calculate the prototype using (1) instead of (2) + (3).

• •

  w/o APF: calculate $\mathbf{p}_{i}^{w}$ using direct addition $\mathbf{p}_{i}^{w}=\mathbf{p}_{i}+\mathbf{r}_{i}$ instead of (4).

• •

  w/o QIA and APF: calculate the prototype using (1) instead of (2) + (3), and calculate $\mathbf{p}_{i}^{w}$ using direct addition $\mathbf{p}_{i}^{w}=\mathbf{p}_{i}+\mathbf{r}_{i}$ instead of (4).

From the results in Table 5, We can obtain several observations. First, the model performance drops more or less without any of QIA or APF, which demonstrates the effectiveness of RAPS. Second, QIA seems to be more important under 10-way 1-shot scenario than 5-way 1-shot, with the loss of performance 0.15% and 0.76%, respectively. Third, APF seems to be more important under 5-way 1-shot scenario than 10-way 1-shot, with the loss of performance 1.35% and 1.13%, respectively. Based on the above discussion, it may be a future research direction to choose appropriate prototype computation method under different $N$-way $K$-shot settings.

In order to further explore the reason why our proposed method has excellent discriminative capability, we give the visualization results in Figure 2 with BERT on 5-way 1-shot of the validation set of FewRel 1.0. Figure 2 shows the t-SNE (van der Maaten and Hinton, 2008) visualization of query instances, where different colors represent different relation classes. The left subfigure means the original statements of query instances encoded by Proto-BERT (Snell et al., 2017), and the right subfigure means the statements of query instances encoded by RAPS.

It can be seen from the left subfigure that, although the instances can also be divided into different classes, the intra-class distances are not close enough, and there are multiple error points (i.e., yellow points in red cluster). After introducing the relation information and training with the proposed query-information guided attention and adaptive prototype fusion mechanism (right), we can see that the error points are reduced while the representations of the same class are closer, and the inter-class boundaries are more distinct. The observation shows that our proposed method is indeed beneficial to the improvement of the model.