Extract and Attend: Improving Entity Translation in
Neural Machine Translation

Extracting the translation candidates for entities in the source sentence provides explicit references when generating the target sentence in NMT. There are two steps when extracting the entity translation candidates, where the entities in the source sentences are first detected by NER and then translated to the target language. If the entity is found in the bilingual dictionary, we retrieve its translation(s). Although there may be multiple translation candidates for one entity, the entity usually links to a single sense after disambiguating by entity type, and the multiple candidates in the dictionary for one named entity are commonly all correct. For example, “John Wilson” can be translated to “约翰·维尔逊” or “约翰·威尔森”. During training, we consider the one with shortest Levenshtein distance³³3https://en.wikipedia.org/wiki/Levenshtein_distance compared to the ground truth translation to encourage the decoder to copy the given candidate. During inference, considering that only the source sentence is available, we select the one with highest frequency in the training set.

The coverage in the dictionary is limited for some types of entities (e.g, person names). Meanwhile, a large number of named entities (e.g., person names and some of locations) are translated by transliteration (i.e., translated according to the pronunciations). Accordingly, we consider to use transliteration to handle such entities if they are uncovered by the dictionary. Transliteration in different countries often follow different rules. For example, names of Chinese persons are transliterated into English via Pinyin, while names of Korean persons are often transliterated via McCune-Reischauer⁴⁴4https://en.wikipedia.org/wiki/McCune-Reischauer. Current transliteration models Kundu et al. (2018); Karimi et al. (2011); Le et al. (2019) do not consider different nationalities for a single language pair, which is an important cause for transliteration errors. Considering this, we develop a nationality-aware transliteration pipeline, which consists of a nationality classifier and a nationality-aware transliteration model. As shown in Fig. 2, the nationality classifier takes the source entity and source sentence as input, and predicts the nationality of the entity. Then, the nationality tag is concatenated with the entity and translated by the word-level transliteration model.

We consider to let the decoder attend to the extracted translation candidates via self-attention, which has shown to be more effective in improving entity translation compared to alternative designs (see Section 5.3). Accordingly, we concatenate extracted candidate translations with “[SEP]” and place it before the “¡bos¿” token of the decoder input. In order to identify the alignments between the translation candidates and the corresponding entities in the source sentence, we add entity type embeddings to word embeddings of the entities in the source sentence as Modrzejewski et al. (2020), and concatenate the corresponding translation candidates in the same order as they are in the source sentence. We demonstrate that our model can correctly align the entities and the corresponding translation candidates in Appendix A.1 via case study. We use independent position embeddings for the translation candidates and the target sentence as shown in Fig. 1. The loss on the tokens of translation candidates is ignored. In this way, the decoder can attend to the translation candidates through the attention mechanism in the decoder, which helps improve the performance of the model on translating entities.

We conduct experiments on English-Chinese (En-Zh) and English-Russian (En-Ru) translation. We chose language pairs so that the source and target languages come from different scripts⁵⁵5English uses Latin script, Chinese uses Logographic script, and Russian uses Cyrillic script., because cross-script entity translation is more challenging. Following Modrzejewski et al. (2020), three types of named entities are considered, i.e., person name, organization and location. Note that the proposed framework is not limited to the three types and can be applied to other entities (e.g., domain entities).

The nationality classifier is fine-tuned from pre-trained BERT checkpoint (base, cased) available on HuggingFace¹²¹²12https://huggingface.co/bert-base-uncased. Both the NMT model and the nationality-aware transliteration model use Transformer base architecture Vaswani et al. (2017) with $6$-layer encoder and decoder, hidden size as $512$ and $8$ attention heads.

To evaluate the overall translation quality, we compute BLEU and COMET Rei et al. (2020) scores¹³¹³13The wmt22-comet-da model is used to calculate COMET scores. To evaluate the translation quality on entities, we consider using error rate of entity translation as the evaluation criterion. Following Modrzejewski et al. (2020), we evaluate entity error rate by recognizing named entities from the reference sentence, and then checking occurrence in the output sentence, where it is regarded as error if it does not occur.

We compare our Extract-and-Attend approach with the following baselines¹⁴¹⁴14The entity resources used in Transformer with Dictionary, Replacement and Placeholder are obtained as is Section 3.1:

• •

  Transformer. The Transformer model is directly trained on parallel corpus.

• •

  Transformer with Dictionary. The entity dictionary is directly added to the parallel corpus to train a transformer model.

• •

  Replacement. After identifying entities in the source sentence with NER and aligning them with target tokens, the corresponding tokens are replaced by translation candidates.

• •

  Placeholder Yan et al. (2019); Li et al. (2019). It first replaces the entities in the source sentence with placeholders based on NER and then restores the placeholders in the output sentence with the extracted translation candidates.

• •

  Annotation Modrzejewski et al. (2020). Entity type embeddings are added to the original word embeddings for the tokens of entities in the source sentence.

• •

  Multi-task Zhao et al. (2020a) It improves the entity translation in NMT by multi-task learning on machine translation and knowledge reasoning.

BLEU, COMET and entity error rates of the Extract-and-Attend approach with the baselines are shown in Table 3 and Table 4, where the proposed approach consistently performs the best on all the metrics and language pairs. From the results, it can be observed that: 1) The proposed method reduces the error rate by up to $35\%$ and achieves a gain of up to $0.85$ BLEU and $13.8$ COMET compared to the standard Transformer model; 2) Compared with the annotation method Modrzejewski et al. (2020), which annotates the entities in the source sentence based on NER without incorporating any additional resources (e.g., dictionary), the proposed Extract-and-Attend approach takes advantage of the entity dictionary and nationality-aware transliteration pipeline, and reduces the entity error rate by up to $26\%$ while achieving up to $0.77$ points gain on BLEU and $3.0$ points on COMET; 3) Compared with the replacement and placeholder Yan et al. (2019); Li et al. (2019) methods, the Extract-and-Attend approach is more robust to NER errors (see A.3) than hard replacement and reduces the error rate by up to $16\%$ while gaining up to $2.1$ BLEU and $7.2$ COMET; 4) Compared to the multi-task Zhao et al. (2020a) method, the Extract-and-Attend approach explicitly provides the translation candidates when decoding, which reduces the entity error rate by up to $35\%$ and improves BLEU by up to $0.8$ points and COMET up to $4.4$ points. We also provide the error rates for different entity types in Appendix A.2, and analyze the effect of dictionary coverage in Appendix A.4

Entity error rates calculated according to Section 4.3 may incur false negative errors, which has two main causes. First, as noted by Modrzejewski et al. (2020), it is common for NER models to make erroneous predictions. Second, there may be multiple correct translations for one entity, but the ones different from that in the reference sentence are regarded as errors. For example, BMA (British Medical Association) can either be copied in the target sentence, or translated into its Chinese form “英国医学会”. Therefore, we also perform human evaluation wmttest150 (see Table 5), where 150 sentence pairs with entities are randomly sampled from the $En\rightarrow Zh$ test set. Compared to automatic evaluation results in Table 4, entity error rates based on human evaluation become lower after eliminating the false negatives, while the relative performance of different models remain almost consistent. Therefore, though there are false negatives in the automatic evaluation as in Section 4.3, it is still a valid metric for evaluating entity translation. Moreover, we observe that the Extract-and-Attend approach performs the best on all three entity types and reduces the total error rate by $32\%$.

To investigate the effectiveness of our transliteration pipeline, we implement a variant denoted as Extract-and-Attend (w/o Transliteration), in which we only extract translation candidates covered by the dictionary. From Table 6, we can see that the translation quality of person names is significantly improved, reducing the error rate by $37\%$; transliteration is also effective for locations, reducing the error rate by $9\%$. Overall, the transliteration model improves BLEU by 0.33 and COMET by 4.1.

Considering that different transliteration rules may be applied for different countries, we propose to incorporate nationality information during transliteration. To evaluate the effectiveness of utilizing the nationality information in the transliteration pipeline, we compare the performance of the proposed nationality-aware transliteration pipeline with the transliteration model trained on paired entities without nationality information. As shown in Table 7, adding nationality information during transliteration consistently improves transliteration quality across all language pairs, and is most helpful for $Zh\rightarrow En$, where the transliteration accuracy is improved by $9\%$.

We also conduct experiments to evaluate the effect of attending translation candidates in the encoder compared to the decoder. Similar to Zhong and Chiang (2020), we append translation candidates to the source tokens, where the position embeddings of the translation candidates are shared with the first token of the corresponding entities in the source sentence. Relative position embeddings denoting token order within the translation candidate are also added. As shown in Table 8, adding the translation candidates to the decoder is better than adding to the encoder. Intuitively, attending to translation candidates in the encoder may incur additional burden to the encoder to handle multiple languages.

Some entities have multiple translation candidates in the entity dictionary. To study whether to provide multiple candidates for each named entity, we extract up to three candidates from the entity dictionary. To help the model distinguish different candidates, we use a separator between candidates of the same entity, which is different from the one used to separate the candidates for different entities. Table 9 shows that adding multiple translation candidates slightly reduces the translation quality in terms of BLEU, COMET and entity error rate. Intuitively, all the retrieved translation candidates for an entity are typically correct, and using one translation candidate for each entity provides sufficient information.

Extracting translation candidates requires additional inference time, including the delays from NER and transliteration pipeline. Specifically, the average inference time for standard Transformer, Replacement, Placeholder, Annotation, Multi-task and our method are 389ms, 552ms, 470ms, 416ms, 395ms, 624ms¹⁵¹⁵15Evaluated on a P40 GPU with batch size of 1, other experimental settings same as Section 4.