E2TIMT: Efficient and Effective Modal Adapter for Text Image Machine Translation

As shown in Figure 1 (a), given an input image $I$, a convolutional neural network (CNN) based image encoder extracts the image embedding $E_{I}$ by transforming image pixels into feature vectors:

where $I\in\mathbb{R}^{H\times W\times C}$ and $E_{I}\in\mathbb{R}^{l_{I}\times c}$. $H$, $W$, and $C$ denote the height, width, and channel of the input image respectively. $l_{I}$ represents the length of image embedding, which is calculated as $l_{I}=h\times w$, where $h$, $w$, and $c$ denote the height, width, and channel of the encoded feature map separately.

The image encoder mainly extracts the local features of the input images, while the image sequential encoder aims to model contextual information by considering the whole input sequence:

where $\text{Seq}_{I}(\cdot)$ represents the image sequential encoder and transformer encoder [32] is utilized in our implementation. $S_{I}\in\mathbb{R}^{l_{S}\times d_{S}}$ denotes the sequential features in OCR model. $l_{S}$ and $d_{S}$ denote the length and dimension of sequential features respectively.

Finally, the OCR decoder generates recognized tokens auto-regressively given sequential features:

where $\text{Dec}_{I}(\cdot)$ represents the OCR decoder, and transformer decoder [32] is utilized in our implementation. $D_{I}$ denotes the outputs of the decoder. $W_{I}\in\mathbb{R}^{|\mathcal{V}_{X}|\times d_{I}}$ represents the linear transformation that maps the decoder features into corresponding recognized tokens, $\mathcal{V}_{X}$ is the recognition vocabulary, and $d_{I}$ is the dimension of decoder hidden states.

MT model translates the source language into the target language as shown in Figure 1 (b). Given a source language sentence $T$, the text encoder first maps the input words into a sequence of word embeddings:

where $E_{T}\in\mathbb{R}^{l_{E}\times d_{E}}$ denotes the text embedding. $l_{E}$ and $d_{E}$ represent the sequence length and the dimension of text embedding respectively.

Text sequential encoder further extracts contextual features based on text embeddings:

where $\text{Seq}_{T}(\cdot)$ represents the text sequential encoder, which is a transformer encoder in our implementation. $S_{T}$ denotes the encoded text sequential features.

MT decoder finally generates the target tokens auto-regressively given sequential features:

where $\text{Dec}_{T}(\cdot)$ represents the MT decoder, and the transformer decoder is utilized in our implementation. $D_{T}$ is the output of the decoder and $W_{T}\in\mathbb{R}^{|\mathcal{V}_{Y}|\times d_{T}}$ is the linear transformation. ${\mathcal{V}_{Y}}$ represents the target language vocabulary and $d_{T}$ denotes the dimension of the hidden states.

The embedding modal adapter is placed in the middle of the OCR image encoder and the text sequential encoder as shown in Figure 2 (a). First, the EmbMA transforms the image embedding into the text embedding space. Second, to better meet the feature distribution of the MT processing flow, the output of EmbMA is constrained by the text embedding through a cross-modal contrastive loss $\mathcal{L}_{\text{CMC}}^{\text{EmbMA}}$. As so, the output of EmbMA given $i$-th image embedding should be similar to the $i$-th text embedding, and apart from the other text embeddings in the mini-batch:

where $\text{EmbMA}(\cdot)$ utilizes the same modal adapter architecture as in Equation 7. $H_{\text{EmbMA}}^{(i)}$ represents the output of the EmbMA. $E_{I}^{(i)}$ and $E_{T}^{(i)}$ denote the image and text embedding of $i$-th sample respectively. $K$ denotes the size of the mini-batch. $\tau$ stands for the temperature parameter and $d(q,k)$ represents the similarity metric which we utilize cosine similarity in our implementation.

Aligned with text embedding, the outputs of EmbMA are further fed into the text sequential encoder to obtain the contextual feature $S_{\text{EmbMA}}^{(i)}$. Through EmbMA, the image embeddings are transformed into the MT processing flow, and MT decoder finally generates target translation:

Different from EmbMA, SeqMA is designed to align the sequential features of the OCR and MT models. As shown in Figure 2 (b), SeqMA first transforms the image sequential features into text sequential feature space. Then, the MT decoder generates target language tokens given transformed image sequential features:

where $\text{SeqMA}(\cdot)$ uses the same structure as in Equation 7. $H_{\text{SeqMA}}^{(i)}$ denotes the output of the sequential modal adapter and $S_{I}^{(i)}$ represents the output of the image sequential encoder of the $i$-th sample in the mini-batch. $D_{\text{SeqMA}}^{(i)}$ denotes the output of text decoder given the hidden states from SeqMA.

Since the hidden states of the SeqMA are further fed into the MT decoder, the feature distribution of $H_{\text{SeqMA}}^{(i)}$ should be similar to the hidden states of $S_{T}^{(i)}$. To bridge the feature gap between the OCR and MT tasks, a cross-modal contrastive loss is utilized to align the feature distribution of the transformed image sequential feature and text sequential feature:

where $d(\cdot)$ and $\tau$ are the same similarity metric and temperature parameter as introduced in Equation 8.

During model training, only parameters in modal adapters are updated, while the parameters in the OCR and MT models are all fixed. Through parameter-efficient modal adapter tuning, the pre-trained OCR encoder and MT decoder are able to transfer to the TIMT task with ease. Specifically, multi-task learning is utilized by optimizing end-to-end text image translation loss and cross-modal contrastive loss. Formally, the end-to-end text image translation loss and the overall loss functions are:

where $\mathcal{L}_{\text{CMC}}$ is introduced as in Equation 8 and Equation 11. $\lambda_{\text{CMC}}$ denotes the hyper-parameter, which balances the weight of end-to-end text image translation loss and cross-modal contrastive loss. Note that the $P(Y^{(i)}|I^{(i)})$ in end-to-end text image translation loss $\mathcal{L}_{\text{TIMT}}$ and cross-modal contrastive loss $\mathcal{L}_{\text{CMC}}$ are calculated based on the corresponding training workflow of SeqMA and EmbMA.

During model inference, as the blue arrow lines shown in Figure 2, the input images are first fed into the OCR encoder to obtain the image features. Second, the modal adapter transforms the image features into the MT feature space, and the MT decoder finally generates translation results. Note that the OCR decoder and the MT encoder are not utilized during inference resulting in a fast decoding speed with the end-to-end processing architecture as shown in Figure 1 (c). By bridging OCR encoder and MT decoder, modal adapter based method can take full advantage of pre-trained OCR and MT models.

OCR, MT, and end-to-end TIMT datasets are utilized in our experiments. OCR and MT datasets are used to train the OCR and MT models respectively. While the TIMT dataset is used to train the parameters in the modal adapter.

We implement the image encoder based on the code release by  [2]. The MT model is utilized the same architecture proposed in  [32]. The OCR and MT models are firstly trained with OCR and MT datasets respectively. Parameters of OCR and MT models are then frozen during fine-tuning. The implementation of the modal adapter is utilized a similar architecture as the transformer encoder with the hidden dimensions of 512, 8 attention heads, and a dropout rate of 0.1. The initial learning rate is set to 2e-3, the batch size is 64, and the training step is set to 300,000. Parameters of the modal adapter are initialized with Xavier initiation method  [8] and optimized with Adam optimizer [15] on single NVIDIA V100 GPU. Detokenized BLEU [21] calculated by sacre-BLEU ⁶⁶6https://github.com/mjpost/sacrebleu is utilized as the metric to evaluate the performance of text image translation models.

Table 1 shows the BLEU scores of text image translation models on various evaluation datasets. Three OCR models are utilized in the cascade models: CRNN [27], TPS+ResNet+BiLSTM+Attention (TRBA) [2], and TPS+ResNet+
Transformer (TRT). While transformer-base [32] is utilized for MT model. The performance of the OCR and the MT models are shown in Section 4.4. Five architectures are compared in end-to-end TIMT setting. TRBA [2] represents the OCR architecture trained with end-to-end TIMT dataset. CLTIR [5] model trains end-to-end TIMT with auxiliary OCR task. RTNet [29] utilizes a feature transformer to link OCR encoder and decoder but ignores the task gap modeling. MTETIMT [19] represents the machine translation enhanced end-to-end TIMT model, which utilizes multi-task learning with auxiliary translation task. While MHCMM[4] proposes a multi-hierarchy cross-modal mimic framework for the end-to-end text image translation, which incorporates external text translation corpus and utilizes text MT model as teacher guidance for TIMT model. The modal adapter in Table 1 bridges the pre-trained TRT OCR encoder and transformer MT decoder. Experimental results show that our proposed sequential and embedding modal adapter outperforms two-stage cascade models on three translation domains with an average improvement of 1.01 BLEU scores. Meanwhile, modal adapter improves the TIMT performance on various language directions (En$\Rightarrow$Zh and En$\Rightarrow$De), revealing the method is robust to different language settings. For Zh$\Rightarrow$En translation direction, modal adapter based method achieves similar results as the previous machine translation enhanced multi-task training model, indicating modal adapter method can take full advantage of the pre-trained MT model without multi-task training.

Furthermore, the embedding modal adapter performs better than the sequential modal adapter, and we attribute that EmbMA retains the cross-attention flow between the original text sequential encoder and decoder. This shows it is vital not only to eliminate the gap between the OCR and MT tasks but also to maintain the consistency of structures within each task.

OCR and MT models in cascade models are firstly trained with corresponding OCR and MT datasets. Parameters in pre-trained OCR and MT models are then frozen during modal adapter training. Three OCR models CRNN, TRBA, and TRT are all trained with the same scene text recognition and synthetic text line recognition datasets introduced in section 4.1. Table 2 shows the performance of various OCR models. Transformer based TRT model achieves the best recognition performance, indicating the strong sequential encoder is essential for optical character recognition. For MT models, the transformer-base and the transformer-big [32] are utilized to translate the source language into the target language. Table 3 shows the performance of text translation, and the transformer-big achieves better translation BLEU.

To evaluate the generalization of our proposed method, the modal adapter is studied by bridging various OCR encoders and MT decoders. As shown in Figure 3, modal adapter tuning outperforms the cascade models on different OCR and MT combinations, revealing the good generalization of modal adapter tuning methods. Figure 3 (a) shows the text image translation results by combining different OCR models and transformer base MT model. Better OCR image encoder can extract more information into image features, leading to better text image translation performance.

Figure 3 (b) depicts various OCR models with transformer big MT models. Similar to Figure 3 (a), better OCR models achieve better results with transformer big MT models. Furthermore, stronger MT decoders can further improve the translation performance in Figure 3 (b) compared with Figure 3 (a). As a result, our proposed modal adapter tuning method has strong scalability by bridging better OCR and MT models.

Cascade models have redundant parameters and slow decoding speed. By removing the OCR decoder and the MT encoder, the modal adapter tuning method has fewer parameters and a faster decoding speed. As shown in Table 4, the end-to-end model, which is trained from the scratch, has fewer parameters and a faster decoding speed compared with the cascade model. Fine-tuning model is also an end-to-end model, which is initialized with the OCR encoder and MT decoder. Then the fine-tuning model is trained with the end-to-end text image translation dataset. Since the modal adapter bridges the OCR encoder and the MT decoder directly, it has a faster decoding speed than the cascade model. Meanwhile, after removing the OCR decoder and MT encoder, modal adapter models have fewer parameters than the cascade model. For the comparison of fine-tuning methods, modal adapter tuning outperforms fine-tuning model, because modal adapter models the task consistency between the OCR encoder and MT decoder, which alleviates the gap between OCR and MT tasks.

Adapter tuning [24] is an effective parameter-efficient fine-tuning method. Different from adapter tuning, which inserted bottleneck modules inside the pre-trained transformer layers, the modal adapter is designed outside the pre-trained models by bridging the separated OCR encoder and the MT decoder. As shown in Table 5, the modal adapter significantly outperforms adapter tuning with 5.81 BLEU for the synthetic domain and 3.59 BLEU for the subtitle domain. To offer a more similar architecture, we also put the bottleneck-based adapter outside the pre-trained models, which is similar to our proposed modal adapter tuning. Bottleneck-based modal adapter tuning also outperforms the vanilla adapter tuning, revealing the effectiveness of explicitly modeling the transformation mapping from the OCR feature space to the MT feature space. Finally, self-attention based modal adapter outperforms the bottleneck-based modal adapter, which we attribute to the strong encoding ability of stacked self-attention layers.

Parameters of the modal adapter are trained on the end-to-end TIMT dataset and the amount of end-to-end data has a great impact on performance. Figure 4 shows the performance of modal adapter tuning with different amounts of end-to-end TIMT datasets. When the end-to-end data is low-resource (around 200 thousand image-text pairs), the performance of modal adapter tuning is limited. We attribute the reason to the non-convergence of modal adapter given low-resource end-to-end data. As the amount of end-to-end TIMT data increases, the modal adapter achieves better results, revealing the modal adapter needs enough data to learn the transformation from the OCR feature space to the MT feature space. When the end-to-end image-text translation data achieves more than 800 thousand pairs, the TIMT results tend to be stable and perform the best translation results. Thus, one million end-to-end text image translation pairs are suitable to train a good end-to-end TIMT model.

Hyper-parameter $\lambda_{\text{CMC}}$ is an important parameter to balance the end-to-end TIMT optimization object and cross-modal contrastive learning object. Figure 5 shows the evaluation of hyper-parameter $\lambda_{\text{CMC}}$. From this hyper-parameter evaluation, the optimal value of $\lambda_{\text{CMC}}$ is 0.4 for both embedding modal adapter and sequential modal adapter. When $\lambda_{\text{CMC}}=0$, parameters in the modal adapter are only optimized by the end-to-end TIMT loss, which ignores the task gap between OCR and MT, leading to performance drop. Specifically, without cross-modal contrastive learning, SeqMA drops 2.35 BLEU scores and EmbMA drops 2.68 BLEU scores, indicating that cross-modal contrastive learning can effectively alleviate the feature gaps between the OCR and MT tasks. When $\lambda_{\text{CMC}}=1$, the overall loss function becomes $\mathcal{L}_{\text{All}}=\mathcal{L}_{\text{CMC}}$, and the performance drops, indicating end-to-end loss is also vital to modal adapter tuning. Thus, the optimization of the modal adapter should be guided both from direct translation object $\mathcal{L}_{\text{TIMT}}$ and cross-modal contrastive learning object $\mathcal{L}_{\text{CMC}}$.

TIMT models are mainly divided into the cascade and end-to-end models. Cascade models deploy OCR and MT models respectively [26, 10, 1, 3, 7]. Specifically, the source language text images are first fed into OCR models to obtain the recognized source language sentences [28, 27, 2, 35, 36, 13, 14]. Second, the source language sentences are translated into the target language with the MT model [31, 32, 38, 37]. Cascade directly connects separated OCR and MT models leading to model redundancy and slow decoding speed. Furthermore, recognition errors made by OCR models are further propagated through MT models, causing severe translation mistakes.

For end-to-end models, the naive approach is to take the OCR model to translate source language text images by training with source language images and corresponding target sentences like TRBA [2]. Furthermore, multi-task learning is proposed to incorporate external OCR datasets [29, 5] or MT datasets [19] to enhance the performance of end-to-end models. MHCMM[4] further improves the feature representation through cross-modal mimic learning on the basis of incorporating external MT data.

However, existing methods still have limitations in fusing cascade and end-to-end models. In this paper, our proposed modal adapter bridges OCR encoder and MT decoder in cascade method through an end-to-end framework, which can take advantage of both cascade and end-to-end methods. Experimental results show modal adapter based TIMT effectively improves translation performance with efficient architecture and fast decoding speed.

Pre-trained models have been explored to achieve good performance after fine-tuning on down-stream tasks [6, 22, 17, 23]. To simplify and speed up the fine-tuning process, efficiency tuning methods are proposed by just updating partial parameters of the model [34]. Another parameter-efficient tuning research keeps the parameters of pre-trained models unchanged and incorporates external modules to meet the downstream tasks like adapter tuning [24], LoRA [11], BitFit [33], prefix tuning [18], and so on. These fine-tuning methods just optimize the parameters of external modules, which makes the fine-tuning process more efficient.

Except for fine-tuning unified pre-trained models, existing research also tried to bridge pre-trained encoder and decoder [25].  [30] proposed to bridge pre-trained mBERT and mGPT through a Graft module to achieve text machine translation. While  [16] explores combining ASR encoder and MT decoder with vanilla adapter for end-to-end speech translation. Inspired by recent research on bridging encoder and decoder, we propose a modal adapter to bridge the OCR encoder and the MT decoder.