Layer-Wise Multi-View Learning for Neural Machine Translation

1. Multi-view. Multi-view learning has achieved great success in conventional machine learning by exploiting the redundant views of the same input data [Xu et al., 2013], where one of the keys is view construction. In our scenario, a view is the hidden representation of the input sentence (an array of hidden vectors for each token, e.g., $H^{*}$). In this work, we further propose to take the off-the-shelf output of each encoder layer (i.e., $H^{(l)}$ ²²2To be precise, there is a newly added layer normalization following $H^{(l)}$ to keep consistent with $H^{*}$.) to construct the redundant views. In NLP, previous implementations of view construction generally require the model to recalculate on the reconstructed input, such as using different orders of n-grams in the bag-of-word model [Matsubara et al., 2005], randomly masking the input tokens [Clark et al., 2018]. As opposed to them, our method is very cheap as the by-product of the standard layer-by-layer encoding process. According to the definition of a view, we can regard the vanilla Transformer as a single-view model since only the topmost encoder layer (also called primary view) is fed to the decoder. In contrast, MV-Transformer additionally contains an intermediate layer $M_{a}$ ($1\leq M_{a}<M$) as the auxiliary view ³³3In this work, we only consider the case of one auxiliary view. More auxiliary views may be more helpful, while it requires more training costs. We leave this issue in future work.. The choice of $M_{a}$ can be arbitrary, and we discuss its effect in § 4.2. Our goal is to learn a better single model with the help of the auxiliary view.

2. Partially shared parameters. In the encoder, except for the last layer normalization, all other parameters are shared in the two views to obtain the corresponding view representations by encoding only once. However, the situation is different for the decoder. We empirically find that a fully shared decoder has no sufficient capacity to be compatible with two different views simultaneously, especially in medium or large translation tasks (see § 4.4). On the other hand, it can be seen that the difference between the two views only directly affects the CANs in the decoder and has nothing to do with other sublayers (i.e., SANs, FFNs). Therefore, using a separate decoder for each view will cause an enormous waste of decoder model parameters. To trade-off, we extend the decoder network by using independent CANs for each view but share all SANs and FFNs (see Figure 1) ⁴⁴4In earlier experiments, we tried to add view embedding to make the decoder aware of identity of each view, i.e. $H^{*}+\mathcal{E}_{pri}$ for primary view, where $\mathcal{E}_{pri}\in\mathcal{R}^{d}$, but it did not work well..

3. Two-stream decoder. Given the two views, we use a two-stream decoder during training. Like Eq. 2, the auxiliary view is queried as:

   $$\ddot{Z}_{a}^{(l)}=\dot{Z}_{a}^{(l)}+\textrm{CAN}_{a}\big{(}\textrm{LN}_{a}(\dot{Z}_{a}^{(l)}),H_{a}^{*}\big{)}$$

   (4)

   where the subscript $a$ indicates used for auxiliary view. In each decoding step, one stream queries $\dot{Z}^{(l)}$ from the primary view $H^{*}$ like a standard Transformer, while the other stream queries $\dot{Z}_{a}^{(l)}$ from the auxiliary view $H_{a}^{*}$ through separate CAN sublayers. In this way, each stream yields distinct predictions based on different context semantics in the views. Here we use $p_{pri}(\cdot)$ and $p_{aux}(\cdot)$ to denote the prediction distribution by the primary view and the auxiliary view, respectively.

4. Training. To jointly train the two views and transfer the knowledge between them, the training objective of MV-Transformer consists of two items. The first item $\hat{\mathcal{L}}_{nll}$ is similar to the negative log-likelihood in Eq. 3, but additionally considers the log-likelihood of the auxiliary view prediction:

   $$\hat{\mathcal{L}}_{nll}=\frac{1}{2}\times(\mathcal{L}_{nll}^{pri}+\mathcal{L}_{nll}^{aux})$$

   (5)

   where $\mathcal{L}_{nll}^{pri}$, $\mathcal{L}_{nll}^{aux}$ are based on the distribution of $p_{pri}(\cdot)$ and $p_{aux}(\cdot)$ respectively, and $1/2$ is used to numerically scale $\hat{\mathcal{L}}_{nll}$ to $\mathcal{L}_{nll}$. The second item $\hat{\mathcal{L}}_{cr}$ is the consistency regularization loss between views, where we use Kullback–Leibler (KL) divergence to let the student (played by the auxiliary view) imitate the prediction of the teacher (played by the primary view):

   $\displaystyle\begin{split}\hat{\mathcal{L}}_{cr}&=-\sum_{j=1}^{n}\textrm{KL}(p_{aux}^{(j)}||p_{pri}^{(j)})=-\sum_{j=1}^{n}\sum_{v\in\mathcal{V}}{p_{pri}^{(j)}(v)\textrm{log}\frac{p_{pri}^{(j)}(v)}{p_{aux}^{(j)}(v)}}\end{split}$

   (6)

   where $p^{(j)}(v)$ is the probability of generating token $v$ at step $j$⁵⁵5We use temperature $\tau$=1 in all experiments, e.g. $p(i)=\textrm{exp}(z_{i}/\tau)/\sum_{j\in\mathcal{V}}\textrm{exp}(z_{j}/\tau)$, $z$ is the logit.. We note that our consistency regularization is different from traditional knowledge distillation, where a typical implementation is to detach the teacher’s prediction $p_{pri}^{(j)}$ as a constant [Hinton et al., 2015]. On the contrary, our method takes $p_{pri}^{(j)}$ as a variable that requires gradients during back-propagation. To this end, the entire model parameters are optimized to give good predictions in two views instead of considering only one, which implicitly makes the model learn from different encoder layers. Some people may say that it is enough for the student to learn from the teacher, but the reverse is unreasonable. However, we believe that the information in different views is complementary, so the potential for mutual learning of views may be greater than one-way learning. And our empirical comparison in § 3.3 also confirms this assumption. Finally, we can interpolate these two losses with the hyper-parameter $\alpha$ to obtain the overall loss function for multi-view learning:

   $\displaystyle\hat{\mathcal{L}}=(1-\alpha)\times\hat{\mathcal{L}}_{nll}+\alpha\times\hat{\mathcal{L}}_{cr}$

   (7)

   Intuitively, when $\alpha$ is low, the loss degrades into Eq. 5, which only focuses on the ground-truth labels. On the contrary, a high $\alpha$ overemphasizes the consistency of the entire vocabulary between the two views, resulting in neglecting to learn from the provided ground-truth. We discuss $\alpha$’s effect in  § 4.2.

5. Inference. Instead of maintaining both views like training, we can shift to any single view at inference time. Considering the primary view as an example: We can straightforwardly discard all the modules attached to the auxiliary view, including $\textrm{CAN}_{a}$ and $\textrm{LN}_{a}$ in the decoder as well as the newly added layer normalization in the encoder. It makes the decoding speed to be precisely the same as that of the standard model. Likely, we can also switch to the auxiliary view composed of fewer encoder layers for slightly faster speed, but with the risk of performance degradation.

In this section, we discuss why our method works from two aspects: robustness to encoding representation and dark knowledge. See  § 4.1 for more experimental analysis.

1. Robustness to encoding representation. Over-reliance on the top encoding layer (primary view) makes the model easier to over-fit [Wang et al., 2018]. Our method attempts to reduce the sensitivity to the primary view by feeding an auxiliary view. Figure 2 shows that the vector similarity between the $i$-th encoder layer and the topmost layer grows as the increase of $i$. Therefore, we can regard the middle layer’s auxiliary view as a noisy version of the primary view. Training with noises has been widely proven to effectively improve the model’s generalization ability, such as dropout [Srivastava et al., 2014], adversarial training [Miyato et al., 2017, Cheng et al., 2019] etc. We also experimentally confirm that our model is more robust than the single view model when injecting random noises into the encoding representation.

2. Dark knowledge. Typically, the prediction target in $\mathcal{L}_{nll}$ is a one-hot distribution: Only the gold label is 1, while the others are 0. A better alternative is label smoothing [Szegedy et al., 2016], which reduces the probability of gold label by $\epsilon$ and redistributes $\epsilon$ to all non-gold labels on average. However, label smoothing ignores the relationship between non-gold labels. For example, if the current ground-truth is “improve”, then “promote” should have high probability than “eat”. In contrast, in our method, the target in the auxiliary view is the primary view’s prediction, which contains more information about non-gold labels, also known as dark knowledge [Hinton et al., 2015].

1. Datasets. We conducted experiments on five translation tasks: Korean$\rightarrow$English (Ko$\rightarrow$En, 96k)⁶⁶6https://sites.google.com/site/koreanparalleldata/, IWSLT’14 German$\rightarrow$English (De$\rightarrow$En, 160k), WMT’17 Turkish$\rightarrow$English (Tr$\rightarrow$En, 205k), WMT’16 Romanian$\rightarrow$English (Ro$\rightarrow$En, 601k)⁷⁷7https://github.com/nyu-dl/dl4mt-nonauto and WMT’16 English$\rightarrow$German (De$\rightarrow$En, 4.5M)⁸⁸8https://drive.google.com/uc?export=download&id=0B_bZck-ksdkpM25jRUN2X2UxMm8. We use the officially provided development sets and test sets for all tasks. We pre-process and tokenize all the data sets using the Moses toolkit.

2. Models and hyperparameters. We tested all models in shallow networks based on PostNorm and deep networks based on PreNorm, respectively. Concretely, for shallow models, we use $M$=$N$=3 for the small-scale Ko-En task ⁹⁹9We failed to train with $M$=$N$=6 in early experiments., while $M$=$N$=6 for other tasks. We use Small configuration (embed=512, ffn=1024, head=4) for {Ko, De, Tr}-En, and Base configuration (embed=512, ffn=2048, head=8) for Ro-En and En-De. As for deep models, we double the encoder depth as the corresponding PostNorm counterparts. E.g., suppose we use a 6-layer encoder in vanilla Transformer. In that case, we turn it to 12-layer in deep Transformer. For MV-Transformer, we use 1/3/6-th encoder layer as the auxiliary view when the encoder depth is 3/6/12, respectively. Following ?), we use the inverse_sqrt learning rate schedule with warm-up and label smoothing of 0.1. Some training hyperparameters are distinct across tasks due to the different data sizes. Detailed hyperparameters are listed in Appendix A.

3. Decoding and evaluation. To compare with previous works, we use the beam size of 4 and average last 5 checkpoints on De$\rightarrow$En, while for other tasks, we use the beam size of 5 and the best checkpoint according to the best BLEU score on the development set. For evaluation, except that Ko$\rightarrow$En uses sacrebleu ¹⁰¹⁰10BLEU+c.mixed+l.ko-en+#.1+s.exp+tok.13a+v.1.2.21, all other datasets are evaluated by multi-bleu.perl. Only De$\rightarrow$En is reported by case insensitive BLEU.

• •

  Oneway-KD. Similar to Eq. 7 but detach the teacher’s prediction, i.e., gradients of the teacher’s prediction is not tracked, posing a one-way transfer from primary view to auxiliary view.

• •

  Seq-KD. Train the large model first and then translate the original training set by beam search to construct the distilled training set for the small model [Kim and Rush, 2016].

• •

  Ensemble. Independently train the two models and combine their predictions at inference time, e.g., by algorithmic average.

Experiments are done on IWSLT’14 De$\rightarrow$En, where the small model has a 3-layer encoder. As shown in Table 3, we can see that: (1) Oneway-KD suffers from severe degradation than MV when detaching the primary view, which indicates that making mutual learning between the primary view and auxiliary view is critical; (2) Seq-KD is almost useless or even badly hurts the performance (vs. Baseline (3L)), which is against the previous belief that Seq-KD helps the small model a lot by learning from the teacher. We suspect that the reason is that our student’s performance has already been closed to the teacher; (3) Ensemble can achieve significant performance improvement than a single model but at the cost of almost twice the slower decoding speed. However, our approach can achieve comparable or even better results than the model ensemble but maintain the decoding speed as a single model.

1. Robustness to encoding noises. In § 2.3, we suppose that MV-Transformer is less sensitive to noise encoding representations due to the introduction of the auxiliary view. To verify it, we add random Gaussian noises sampled from $\mathcal{N}(0,\epsilon)$ into the normalized input in the last layer normalization of the encoder¹²¹²12Original layer normalization is $y=g\odot N(x)+b$, while our noised version is $y=g\odot\big{(}N(x)+\epsilon\big{)}+b$. Our purpose is to avoid different scales of $g$ and $b$ between models.. As shown in Figure 3, we can see that while both models degrade performance along with stronger noises, MV-Transformer is less sensitive, e.g., the maximum gap is 15.72/2.09 when $\epsilon$=0.8/1.0 for the 6/12-layer encoder respectively. It indicates that our model has a better generalization even if the test distribution is largely different from the training distribution. We also observed that the PreNorm-style Transformer is less sensitive than the PostNorm counterpart, e.g., when $\epsilon$=1.0, the PostNorm Transformer decreases 33.55 BLEU points, while the PreNorm one only decreases 6.86.

2. Dark knowledge. As shown in Table 3, we study the effect of dark knowledge in our multi-view learning. First, we test the case where we only use gold knowledge (ground-truth label, #2) and dark knowledge (non-ground truth labels, #3). Obviously, without the help of dark knowledge, multi-view learning fails to boost the performance. Going further along this line, we study which part of dark knowledge is the most important (#4-5). Specifically, we split all non-gold labels into two parts from high to low according to their probability: [1,100] and [100,]. We can see that our approach’s success lies in those non-gold labels at the top, not the long tail part.

1. Position of auxiliary view. In Figure 5, we plot the BLEU score curves against the auxiliary view position $L_{a}$ on IWSLT’14 De$\rightarrow$En. In general, we can obtain better performance when $L_{a}$ is closer to $L_{e}$, but this is not always true. It is intuitive: When $L_{a}\ll L_{e}$, the auxiliary view is numerically far different from the primary view, which is difficult for a partially shared decoder to learn. On the contrary, if $L_{a}$ is too close to the top, the slight difference may not bring too many learnable signals. In this work, we use the middle layer as the auxiliary view, while it should be noted that we could obtain better results if we tune $L_{a}$ more carefully (e.g., $L_{a}$=10 vs. $L_{a}$=6 in the 12-layer encoder).

2. Interpolation coefficient. In Figure 5, we show the curve of BLEU score against the hyperparameter $\alpha$ on IWSLT’14 De$\rightarrow$En test set. First of all, we can see that the BLEU score is improved along with the increase of $\alpha$, while it starts to decrease when $\alpha$ is too large (e.g., $\alpha>$ 0.6). In particular, we failed to train the model when $\alpha$=1.0. On the other hand, a large $\alpha$ can reduce the gap between the two views as expected. We note that even though it is difficult to know the optimal $\alpha$ in advance, we empirically found that $\alpha\in[0.3,0.5]$ is robust across these distinct tasks.

In addition to the Transformer, we also test our method on the recently proposed DynamicConv [Wu et al., 2019]. Original DynamicConv is composed of a 7-layer encoder and a 6-layer decoder. Our method takes the topmost layer as the primary view as before and uses the 4-th layer as the auxiliary view. The results on IWSLT’14 De$\rightarrow$En task are listed in Table 5. It can be seen that DynamicConv with multi-view is stably higher than that of a single-view model by about 0.5 BLEU score, which indicates that our method is transparent to network architecture and has the potential to be widely used.