MirrorAlign: A Super Lightweight Unsupervised Word Alignment Model
via Cross-Lingual Contrastive Learning

For a given source-target sentence pair $(\mathbf{s},\mathbf{t})$, $s_{i},t_{j}\in\mathbb{R}^{d}$ represent the $i$-th and $j$-th word embeddings for the source and target sentences, respectively. Luong et al. (2015a); Ding et al. (2020b) illustrate that modelling the neighbour words within the local window helps to understand the current words. Inspired by this, we perform a extremely simple but effective mean pooling operation with the representations of its surrounding words to capture the contextualized information. Padding operation is used to ensure the sequence length. As a result, the final representation of each word can be calculated by element-wisely adding the mean pooling embedding and its original embedding:

$$\begin{split}x_{i}=&\textsc{MeanPool}([s_{i}]^{win})+s_{i},\\ \end{split}$$

(1)

where $win$ is the pooling window size. We can therefore derive the sentence level representations $(x_{1},x_{2},...,x_{|s|}),(y_{1},y_{2},...,y_{|t|})$ for $\mathbf{s}$ and $\mathbf{t}$. In addition to modeling words, modeling structured information (such as syntactic information) may be helpful to enhance the sentence representation Li et al. (2017); Marcheggiani and Titov (2017); Ding and Tao (2019), thus improving the word alignment. We leave this exploration for future work.

Bidirectional symmetric attention is the basic component of our proposed model. The aim of this module is to generate the source-to-target (aka. s2t) and target-to-source (aka. t2s) soft attention maps. The details of the attention mechanism: given a source side word representation $x_{i}$ as query $q_{i}\in\mathbb{R}^{d}$ and pack all the target tokens together into a matrix $V_{t}\in\mathbb{R}^{\left|t\right|\times d}$. The attention context can be calculated as:

$$\textsc{Attention}\left(q_{i},V_{t},V_{t}\right)=(a_{t}^{i}\cdot V_{t})^{\intercal},$$

(2)

where the vector $a_{t}^{i}\in\mathbb{R}^{1\times|t|}$ represents the attention probabilities for $q_{i}$ in source sentence over all the target tokens, in which each element signifies the relevance to the query, and can be derived from:

$$a_{t}^{i}=\textsc{Softmax}\left(V_{t}\cdot q_{i}\right)^{\intercal}.$$

(3)

For simplicity, we denote the attention context of $q_{i}$ in the target side as $att_{t}(q_{i})$. s2t attention map $A_{s,t}\in\mathbb{R}^{|s|\times|t|}$ is constructed by stacking the probability vectors $a_{t}^{i}$ corresponding to all the source tokens.

Reversely, we can obtain t2s attention map $A_{t,s}$ in a symmetric way. Then, these two attention matrices $A_{s,t}$ and $A_{t,s}$ will be used to decode alignment links. Take s2t for example, given a target token, the source token with the highest attention weight is viewed as the aligned word.

Intuitively, the two attention matrices $A_{s,t}$ and $A_{t,s}^{T}$ should be very close. However, the attention mechanism suffers from symmetry error in different direction Koehn and Knowles (2017).

To bridge this discrepancy, we introduce agreement mechanism Liang et al. (2006), acting like a mirror that precisely reflects the matching degree between $A_{s,t}$ and $A_{t,s}$, which is also empirically confirmed in machine translation Levinboim et al. (2015). In particular, we use an agreement loss to bind above two matrices:

$$\mathcal{L}\textit{oss}_{disagree}=\sum_{i}\sum_{j}(A^{s,t}_{i,j}-A^{t,s}_{j,i})^{2}.$$

(4)

In Section 4.6, we empirically show this agreement can be complementary to the bidirectional symmetric constraint, demonstrating the effectiveness of this component.

Suppose that $(q_{i},att_{t}(q_{i}))$ is a pair of s2t word representation and corresponding attention context sampled from the joint distribution $p_{t}(q,att_{t}(q))$ (hereinafter we call it a positive pair), the primary objective of the s2t training is to maximize the alignment degree between the elements within a positive pair. Thus, we first define an alignment function by using the $\mathrm{sigmoid}$ inner product as:

$$\textsc{Align}(q,att_{t}(q))=\sigma(\langle q,att_{t}(q)\rangle),$$

(5)

where $\sigma(\cdot)$ denotes the $\mathrm{sigmoid}$ function and $\langle\cdot,\cdot\rangle$ is the inner product operation. However, merely optimizing the alignment of positive pairs ignores important positive-negative relation knowledge Mikolov et al. (2013).

To make the training process more informative, we reform the overall objective in the contrastive learning manner Oord et al. (2018); Saunshi et al. (2019) with Noise Contrastive Estimation (NCE) loss Mikolov et al. (2013), which has been widely used in many NLP tasks Xiong et al. (2021); Gao et al. (2021); Wang et al. (2022). Specifically, we first sample $k$ negative word representations $q_{j}$³³3In the contrastive learning setting, $q_{j}$ and $att_{t}(q_{i})$ can be sampled from different sentences. If $q_{j}$ and $att_{t}(q_{i})$ are from the same sentence, $i\neq j$; otherwise, $j$ can be a random index within the sentence length. For simplicity, in this paper, we use $q_{j}$ where $i\neq j$ to denote the negative samples, although with a little bit ambiguity. from the margin $p_{t}(q)$. Then, we can formulate the overall NCE objective as following:

		$\displaystyle\mathcal{L}\textit{oss}_{s\rightarrow t}^{i}=-\mathop{\mathbb{E}}_{\{att_{t}(q_{i}),q_{i},q_{j}\}}~{}[\log$		(6)
		$\displaystyle\frac{\textsc{Align}(q_{i},att_{t}(q_{i}))}{\textsc{Align}(q_{i},att_{t}(q_{i}))+\sum_{j=1}^{k}\textsc{Align}(q_{j},att_{t}(q_{i}))}],$

It is evident that the objective in Eq. (6) explicitly encourages the alignment of positive pair $(q_{i},att_{t}(q_{i}))$ while simultaneously separates the negative pairs $(q_{j},att_{t}(q_{i}))$.

Moreover, a direct consequence of minimizing Eq. (6) is that the optimal estimation of the alignment between the representation and attention context is proportional to the ratio of joint distribution and the product of margins $\frac{p_{t}(q,att_{t}(q))}{p_{t}(q)\cdot p_{t}(att_{t}(q))}$ which is the point-wise mutual information, and we can further have the following proposition with repect to the mutual information:

The detailed proof can be found in Oord et al. (2018). Proposition 1 indicates that the lower bound of the mutual information $I(q,att_{t}(q))$ can be maximized by achieving the optimal NCE loss, which provides theoretical guarantee for our proposed method.

Our training schema over parallel sentences is mainly inspired by the bilingual skip-gram model Luong et al. (2015b) and invertibility modeling Levinboim et al. (2015). Therefore, the ultimate training objective should consider both forward ($s\rightarrow t$) and backward ($t\rightarrow s$) direction, combined with the mirror agreement loss. Technically, the final training objective is:

	$\displaystyle\mathcal{L}\textit{oss}$	$\displaystyle=\sum_{i}^{\left|t\right|}\mathcal{L}\textit{oss}_{s\rightarrow t}^{i}+\sum_{j}^{\left|s\right|}\mathcal{L}\textit{oss}_{t\rightarrow s}^{j}$		(8)
		$\displaystyle+\alpha\cdot\mathcal{L}\textit{oss}_{disagree},$

where $\mathcal{L}\textit{oss}_{s\rightarrow t}$ and $\mathcal{L}\textit{oss}_{t\rightarrow s}$ are symmetrical and $\alpha$ is a loss weight to balance the likelihood and disagreement loss.

We perform our method on three widely used datasets: English-French (EN-FR), Romanian-English (RO-EN) and German-English (DE-EN). Training and test data for EN-FR and RO-EN are from NAACL 2003 share tasks Mihalcea and Pedersen (2003). For RO-EN, we add Europarl v8 corpus, increasing the amount of training data from 49K to 0.4M. For DE-EN, we use the Europarl v7 corpus as training data and test on the gold alignments. All above data are lowercased and tokenized by Moses. The evaluation metrics are Precision, Recall, F-score (F1) and Alignment Error Rate (AER).

Besides two strong statistical alignment models, i.e. FastAlign and GIZA++, we also compare our approach with neural alignment models where they induce alignments either from the attention weights or through feature importance measures.

For our method (MirrorAlign), all the source and target embeddings are initialized by Xavier method Glorot and Bengio (2010). The embedding size $d$ and pooling window size are set to 256 and 3, respectively. The hyper-parameters $\alpha$ is tested by grid search from 0.0 to 1.0 at 0.1 intervals. For FastAlign, we train it from scratch by the open-source pipeline⁴⁴4https://github.com/lilt/alignment-scripts. Also, we report the results of NNSA and machine translation based model (Section 4.2). All experiments of MirrorAlign are run on 1 Nvidia P40 GPU. The CPU model is Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz. Both FastAlign and MirrorAlign take nearly half a hour to train one million samples.

Table 2 summarizes the AER of our method over several language pairs. Our model outperforms all other baseline models. Comparing to FastAlign, we achieve 1.3, 0.5 and 2.2 AER improvements on EN-FR, RO-EN, DE-EN respectively.

Notably, our model exceeds the naive attention model in a big margin in terms of AER (ranging from 8.2 to 26.1) over all language pairs. We attribute the poor performance of the straightforward attention model (translation model) to its contextualized word representation. For instance, when translating a verb, contextual information will be paid attention to determine the form (e.g., tense) of the word, that may interfere the word alignment.

Experiment results in different alignment directions can be found in Table 1. The grow-diag symmetrization benifits all the models.

Take the experiment on EN-FR dataset as an example, MirrorAlign converges to the best performance after running 3 epochs and taking 14 minutes totally, where FastAlign and GIZA++ cost 21 and 230 minutes, respectively, to achieve the best results. Notably, the time consumption will rise dozens of times in neural translation fashion.

To further explore the effects of several components (i.e., bidirectional symmetric attention, agreement loss) in our MirrorAlign, we conduct an ablation study. Table 3 shows the results on EN-FR dataset. When the model is trained using only $\mathcal{L}\textit{oss}_{s\rightarrow t}$ or $\mathcal{L}\textit{oss}_{t\rightarrow s}$ as loss functions, the AER of them are quite high (20.9 and 23.3). As expected, combined loss function improves the alignment quality significantly (14.1 AER). It is noteworthy that with the rectification of agreement mechanism, the final combination achieves the best result (9.2 AER), indicating that the agreement mechanism is the most important component in MirrorAlign.

To better present the improvements brought by adding each component, we visualize the alignment case in Figure 3. As we can see, each component is complementary to others, that is, the attention map becomes more diagonally concentrated after adding the bidirectional symmetric attention and the agreement constraint.