Improving Prediction Backward-Compatiblility in NLP Model Upgrade with Gated Fusion

To improve backward compatibility in model upgrade, it’s crucial to have a mechanism that detects potential regression errors and mitigates them when making predictions. We propose Gated Fusion (GF) to achieve this by learning a gate as a soft switch to choose between generating predictions by the new model or resorting to outputs of the old model. Gated Fusion is inspired by the gating mechanism widely used in other applications. For example, mixing word copying mode with word generation mode for language modeling (Merity et al., 2016) and summarization (See et al., 2017).

The intuition behind Gated Fusion is that when $f_{new}$ makes a mistake while $f_{old}$ produces the correct output, the gate $g_{\theta}$ will learn to put more weight on $f_{old}$ in order to minimize the final classification loss. This process effectively mitigates potential negative flips introduced by the model upgrade and thus improves the backward compatibility of final predictions.

In practice, training Gated Fusion with randomly initialized $f_{new}$ would make the shallow gating network quickly converge to favor the fully-trained $f_{old}$. To prevent this, we only train $f_{new}$ for the first few epochs to ensure its competence before jointly training $g_{\theta}$ and $f_{new}$ using $l^{*}_{GF}(x)$. In addition, we found that stopping gradient flow from $g_{\theta}$ to $f_{new}$ can further prevent the performance decrease of the new model within Gated Fusion:

At inference time, Gated Fusion produces logits from $f_{old}$ and $f_{new}$ as well as the gate value $\alpha_{gate}$ to make output predictions:

Our proposed Gated Fusion requires $f_{old}$ to be hosted together with the new model. In reality, one could have a resource-constrained setting and request the old model to be discarded at inference. We note that in real applications, repetitive inputs are commonly seen in live traffic (Batrinca and Treleaven, 2015) and the backward compatibility of model upgrade entails that correct predictions can be preserved on the legacy instances already seen and predicted by the old model.

To simulate real scenarios, we randomly cache old model’s logits on a portion of test inputs. When getting out-of-cache instances, we use new model’s output embedding $\mathcal{E}_{new}(x)$ as key and euclidean distance as metric to search for the nearest cached instance. The cached old-model logits can then be used for Gated Fusion to make predictions without hosting $f_{old}$ at inference.

We conduct experiments on two representative model upgrade scenarios: (a) upgrade to a larger pretrained model of the same type, where we use BERT_base to BERT_large. (b) upgrade to a distinct pretrained model with the same size. We use BERT_base to ELECTRA_base (Clark et al., 2020) as this challenging model upgrade scenario for backward-compatibility, as they are pretrained under different self-supervised learning paradigms. The former uses masked language modeling (MLM) with reconstruction loss, while the latter is pretrained in generative-contrastive (adversarial) fashion with distributional divergence as the loss (Liu et al., 2021).

We evaluate our approach across three datasets. They represent different sentence-level classification tasks, from single-sentence to sentence-pair classification, with varying dataset sizes. We use: (a) Stanford Sentiment Treebank (SST-2), a single-sentence task to classify movie review sentiment, with $67$k train and $0.9$k dev set (Socher et al., 2013). (b) Microsoft Research Paraphrase Corpus (MRPC) (Dolan and Brockett, 2005), a sentence-pair classification task for identifying paraphrases, with $3.7$k train and $0.4$k dev set. (c) Question Natural Language Inference (QNLI), a question-paragraph pair task to determine whether the paragraph contains the answer to the question, with $100$k train and $5.5$k dev set. Datasets are taken from GLUE Benchmark (Wang et al., 2018) and processed with scripts from Hugging Face²²2https://huggingface.co/datasets/glue.

For implementation, we use the sequence classification and pre-trained model parameters from Hugging Face Transformers³³3https://huggingface.co/docs/transformers/index. Experiments are done in PyTorch (Paszke et al., 2019) with Tesla V100 GPUs and results are averaged over $5$ random seeds. Learning rate, batch size, and train epoch are tuned during training new model alone on given tasks and then fixed for all backward-compatible solutions. In Gated Fusion, we first train $f_{new}$ alone for first $(N-1)$ epochs and then jointly train $g_{\theta}$ and $f_{new}$ with Gated Fusion logits $l^{*}_{GF}$ in the last epoch. Further implementation details can be found in the Appendix.

We compare our approach with several strong baselines. (a) Train the new model directly on the target task without any adjustment, i.e. $f^{o}_{new}$. (b) The specialized distillation method proposed in Xie et al. (2021), where the KL-divergence of prediction probabilities between old and new models is applied when $p_{old}(y=y_{i}|x_{i})>p_{new}(y=y_{i}|x_{i})$. (c) Model ensemble via majority-voting that was shown to be very effective (Yan et al., 2021; Xie et al., 2021). Similarly, we use $5$-seed new model ensemble as a strong baseline. (d) The ensemble of the old and new models probabilities, $p^{*}(y|x)=(1-\alpha)\cdot p_{old}(y|x)+\alpha\cdot p_{new}(y|x)$, as well as ensemble of the old and new models logits, $l^{*}(y|x)=(1-\alpha)\cdot l_{old}(y|x)+\alpha\cdot l_{new}(y|x)$, where $\alpha$ is searched among $[0.5,0.6,0.7,0.8,0.9]$ to maximize backward-compatibility while achieving accuracy on par with the vanilla $f^{o}_{new}$.

Our first model upgrade scenario scales up the size of underlying pretrained language models. We experiment with BERT_base to BERT_large, where the model size is tripled (110M vs 340M) and the model depth is doubled (12 vs 24 layers).

Table 1 shows the results. For $f^{o}_{new}$, we can observe that the negative flip rates $\mathcal{R}_{NF}$ are usually much larger than the accuracy gains across tasks, which could be the reason to hinder new model adoptions in real-world applications. Besides, when dividing $\mathcal{R}_{NF}$ over the error rate $(1-\textit{accuracy})$, we can observe that around $30\%$ to $40\%$ of all $f^{o}_{new}$ prediction errors are in fact the new errors introduced during model upgrade. For improving prediction backward-compatibility, our proposed Gated Fusion outperforms other competing methods to considerably reduce $\mathcal{R}_{NF}$ without degradation on accuracy. Note that best $\alpha$ values found for the two variants of old-new ensemble are both $0.5$, hence producing identical results.

Compared to the vanilla new model, gated fusion obtains absolute $\mathcal{R}_{NF}$ reductions of −$1.40\%$ on SST-2, −$2.94\%$ on MRPC, and −$1.99\%$ on QNLI. These translate to reducing the total negative flip cases by $64.2\%$, $71.4\%$, $73.2\%$, respectively. Compared to the strongest baseline (old-new ensemble), we obtain further absolute $\mathcal{R}_{NF}$ reductions of −$0.28\%$ on SST-2, −$0.49\%$ on MRPC, and −$0.31\%$ on QNLI, which translate to further reducing $12.8\%$, $11.9\%$, and $11.4\%$ of negative flip cases. These results show the effectiveness of our method to mitigate a significant amount of regression errors during model upgrade.

A more challenging upgrade scenario is when old and new models are pretrained under distinctive paradigms, producing two representation spaces of fairly different characteristics (Meng et al., 2021b). We experiment with BERT_base to ELECTRA_base in this scenario, where two models have the same size but are pretrained under utterly different schemes, i.e. generative versus adversarial.

Table 2 shows the results. For $f^{o}_{new}$, compared with upgrading to BERT_large, we observe larger accuracy gains and lower $\mathcal{R}_{NF}$ on SST-2 and MRPC. However, on QNLI, upgrading to ELECTRA_base achieves a higher accuracy gain but an even a higher $\mathcal{R}_{NF}$. This implies that boosting accuracy and improving backward compatibility could be two related but different objectives.

For mitigation strategies, Gated Fusion achieves the lowest negative flip rates across datasets without any accuracy loss. We obtain absolute $\mathcal{R}_{NF}$ reductions of −$0.92\%$ on SST-2, −$1.33\%$ on MRPC, and −$2.01\%$ on QNLI over the vanilla setup, reducing $56.4\%$, $35.7\%$, and $71.3\%$ of overall negative flips, respectively. Compared with upgrading to BERT_large, we observe that upgrading to ELECTRA_base has much smaller relative negative flip reductions on SST-2 and MRPC, showing that it could be indeed harder to improve backward-compatibility when upgrading to a distinct pretrained model. In contrast, similar relative negative flip reductions are observed on QNLI across two upgrade scenarios. This could be attributed to the abundant training data of the downstream task.

Our proposed method requires the old model to be hosted together with the new model. A natural question is whether we could train Gated Fusion with the old model and then discard it at inference time to host the new model only.

We first experiment with directly dropping the old model within Gated Fusion at inference time. Results in Table 4 show that dropping old model in Gated Fusion can still achieve comparable accuracy across the board, suggesting no performance degradation. Nonetheless, we observe that the negative flip rates also fall back to similar positions as training the new model in the vanilla setting.

However, in real application scenario, live inputs are often repetitively seen across time and ensuring backward-compatibility means that correct predictions on same instances can be preserved after model upgrade. We experiment with the caching method introduced in section 3.3 to store old model’s logits on random $X\%$ of test instances where Gated Fusion can later access them for inference. Results in Table 5 show that with higher percentage of cache, $\mathcal{R}_{NF}$ is gradually reduced towards $\mathcal{R}_{NF}$ of the original Gated Fusion, which is equivalent to $100\%$ cache. Still, we observe a notable gap in $\mathcal{R}_{NF}$ between the partial caching and full settings. We leave the examination of ways to achieve the upper bound in reduction in $\mathcal{R}_{NF}$ with smaller cache to the future work.

In previous works (Yan et al., 2021; Xie et al., 2021), new model ensemble via majority voting is shown to effectively reduce negative flips and posed as a difficult-to-beat baseline. Here, we increase the number of models in ensemble to examine its limitations. Results in Table 3 show that ensemble with more models generally help to obtain lower $\mathcal{R}_{NF}$. However, $\mathcal{R}_{NF}$ converges quickly as number of models increased, where a notable gap remains between new model ensemble and Gated Fusion. Moreover, the results show once more that boosting accuracy does not necessarily improve the backward compatibility in model upgrade.

In principle, two sources could cause negative flips during model upgrade (a) the stochasticity during model training, including initializations, data loading order, and optimization process (Somepalli et al., 2022). (b) the distinctions between old and new model hypotheses, including architecture and pretraining data and procedure, leading to different representation space structures and prediction behaviors in terms of decision boundaries. Without an explicit connection to $f_{old}$, new model ensemble can only reduce negative flips primarily caused by the first factor, while our proposed Gated Fusion directly learns to mitigate regression errors regardless of their causes.

Besides, as large-scale generative models become more and more powerful and popular (Raffel et al., 2020; Brown et al., 2020; Su et al., 2021), it would be difficult to fine-tune them multiple times on a target task for ensemble.

Comparing $f^{o}_{new}$ with $f^{*}_{GF}$, we can calculate the fix rate and new fault rate of our Gated Fusion method. During an upgrade, if there are $20$ negative flips with $f^{o}_{new}$ and $16$ out of them can be mitigated by $f^{*}_{GF}$, we obtain the fix rate to be $16/20=80\%$. Similarly, if $f^{*}_{GF}$ introduces another $4$ new negative flips which are not present with $f^{o}_{new}$, the new fault rate is computed to be $4/20=20\%$. We calculate the $5$-seed average of these two rates across different classification tasks and upgrade scenarios. In BERT_base to BERT_large, the averaged fix rates by Gated Fusion are $68.4\%$ on SST-2, $83.8\%$ on MRPC, and $82.9\%$ on QNLI, with new fault rates being $4.1\%$ on SST-2, $11.3\%$ on MRPC, and $9.7\%$ on QNLI. In BERT_base to ELECTRA_base, Gated Fusion achieves the averaged fix rates $58.0\%$ on SST-2, $50.8\%$ on MRPC, and $75.6\%$ on QNLI, with new fault rates being $2.8\%$ on SST-2, $15.2\%$ on MRPC, and $4.0\%$ on QNLI. These results show that, on average, Gated Fusion is able to eliminate $69.9\%$ of total regression errors while adding only $7.9\%$ new ones, comparing with doing model upgrade without any treatment, i.e. $f^{o}_{new}$.

Table 6 shows a few regression error cases fixed by our proposed approach. In general, Gated Fusion can mitigate negative flips happened on different classes across diverse tasks as well as on inputs with variable lengths. With closer inspections of $f^{*}_{GF}$, we found that when $f_{new}$ produces incorrect predictions and $f_{old}$ gives correct outputs, $g_{\theta}$ is capable of putting larger weights on $f_{old}$ to ensure the backward compatibility. We also observed that the gate $g_{\theta}$ is more prone to over-fitting when the downstream tasks have smaller training set, e.g. MRPC, or are more difficult in nature, e.g. single-sentence task SST-2 versus sentence-pair tasks, which causes Gated Fusion to introduce more new errors, i.e. higher new fault rates.