Augmented Transformers with Adaptive n-grams Embedding for Multilingual Scene Text Recognition

Existing work on STR [14] can be categorized into language-free and language-based approaches [9]. Language-free methods do not consider the linguistic information between characters and focus on the visual textures for recognition [15]. Yao et al. [16] combine various feature descriptors with localized individual characters, and present a multi-scale representation for STR. Shi et al. [17] propose a unified neural network architecture framework for STR that integrates feature extraction, sequence modeling, and transcription, considering STR as a pixel-wise classification task. Language-based methods focus on the language mode with different structures, or consider the relationship between vision and language. For example, attention-based methods [18, 19] are adopted for STR by following an end-to-end neural network model in recurrent neural networks [20]. Fang et al. [21] propose a text recognizer based on convolutional neural networks (CNNs) by fusing visual and language-based methods to boost the recognition performance.

Little work on multilingual scene text recognition has been reported with only a few exceptions. Busta et al. [10] design probably the first method, called E2E-MLT, for multilingual scene text recognition on the basis of a single fully convolutional network, which is demonstrated to be competitive even on rotated and vertical text instances. An E2E approach to script identification with different recognition heads, called Multiplexed Multilingual Mask TextSpotter, was proposed in [22], which can support the removal of existing or inclusion of new languages.

However, due to the diversity of scene texts, complexity of the background, large amounts of uncertainty [23], and different sizes of different language scripts, existing methods fail to work effectively on multilingual scene text recognition tasks.

To benefit from the generated visual features following linguistic rules, increased research interests have been dedicated to using transformers for STR recently [24, 4, 25], where the encoder extracts visual features and the decoder predicts characters in images. Owing to their parallel self-attention and prediction mechanisms, transformers can overcome the difficulties of sequential inference with diverse scene text features to some extent [1].

Sporadic research efforts on adapting transformers to addressing various challenges in monolingual STR have been reported. For instance, to deal with images with different resolutions, Raisi et al. [26] develop a transformer-based architecture for recognizing texts in images by using a 2D positional encoder so that the spatial information the features can be preserved. Biten et al. [27] propose a layer-aware transformer with a pre-training scheme on the basis of text and spatial cues only and show that it works well on scanned documents to handle multimodality in scene text visual question answering. Based on ViT [4], Tan et al. [28] propose a mixture experts of pure transformers for processing different resolutions for scene text recognition. Atienza [5] presents a new transformer, called ViTSTR, for scent text recognition, which only uses the encoder architecture and emphasizes the balance between the performance and computational efficiency. Recently, Wang et al. [29] explore the linguistic information between the local visual patches and propose transformer-based language rectification modules for optimizing word length and guiding rectification in monolingual scene text recognition.

Despite the progress made on the application of visual transformers to STR tasks, it remains an open question how to explicitly identify inter-patch correlations for multilingual scene text recognition containing multi-scale, multi-orientation and low-resolution texts.

Our goal is to introduce a new vision transformer architecture for handling complex multilingual STR tasks. The proposed TANGER, as shown in Fig. 2, consists of a primary transformer, which is a normal vision transform, and a supplementary pyramid transformer with adaptive n-grams embeddings. The primary vision transform focuses on the patch-based image features without considering the associated local patches, in which a dropout layer is adopted to alleviate overfitting. The supplementary pyramid transformer concentrates on effectively extracting visual features from the neighbouring image patches with the help of the adaptive n-grams embedding, allowing us to deal with multi-scale and multi-orientation visual appearances of multilingual texts. Similar to [5], we use a single model architecture in the primary transformer, while in the supplementary transformer, three layers of pyramid transformer encoders are stacked to control the scale of feature maps in three stages. As shown in Fig. 2, $F_{2}$, $F_{3}$ and $F_{4}$ and the feature maps extracted from the previous stage and serve as the input into encoder layers $L_{1}$, $L_{2}$, and $L_{3}$, respectively. To reduce the computational complexity of TANGER, the encoder in the primary transformer and all three encoder layers in the supplementary transformer share the same parameters.

In our implementation, we first divide an input RGB image into non-overlapping patches, and the features of a patch is regarded as a concatenation of the raw pixel RGB values. For example, we use a patch size of $m\times m$, and split a given input image of size $H\times W\times 3$ into $\frac{HW}{m^{2}}$ patches, where $m$, $H$ and $W$ are parameters to be defined. After that, a linear embedding layer is applied on this raw-valued feature to project it onto an arbitrary dimension $C$ for each patch. In this way, we can flexibly pass the embedded patches of different dimensions as well as a position embedding to the input of the two transformers.

The vision transformer in [5] only considers single image patches as a set of isolated vision words or characters without taking into account the correlation between the features in the neighboring image patches, thus limiting the model’s ability to recognize complex scene texts. For obtaining efficient relevant features of the image patches, we treat each image patch as a bag of visual words [30] to capture the relationship between neighboring patches. To this end, we introduce an adaptive $n$-grams embedding method for determining an optimal $n$ for each image patch.

As illustrated in Fig. 2, the pyramid transformer architecture [31] with $n$-grams embedding is employed to flexibly capture the linguistic information in the neighboring patches. However, the value of $n$ for the $i$-th image patch $p_{i}$ must be determined individually to best capture the correlations between its neighboring patches. In this work, we set $n=\{2,3,4,5\}$ according to our pilot studies. Then, we choose the optimal $n_{i}$ for $p_{i}$ as follows:

where $P(.)$ is the probability at which $p_{i}$ is correlated with its previous $(n-1)$ patches, $p_{i-n+1}p_{i-n+2}...p_{i-1}$ represents $n-1$ sequential local visual patches. We can estimate $P(.)$ using a feature histogram of a group of continuous patches built by using the bag of visual words (BVW) as suggested in [30, 32]:

where $VW_{k}$ is the $k$-th visual word, $K$ is the total number of visual words in the histogram, and $N(VW_{k})$ refers to the frequency of features of the input image patches $p_{i-n+1}p_{i-n+2}...p_{i-1}p_{i}$ that is similar to each of the visual word in the histogram. An illustrative example is given in Fig. 3, in which the input image is partitioned into nine patches. To determine the number of grams for $p_{6}$, we calculate $N(VK_{k}),k=1,\cdots,5$ when $n=2,3,4,5$, respectively. Finally, the maximum probability is obtained when $n=2$. Consequently, bi-gram embedding is adopted for $p_{6}$.

Since the length of the local patches differs from patch to patch (in the above example, the length varies from 2 to 5), we use a spatial pyramid pooling layer to obtain the final n-grams embedding representation that conforms to the input size of the transformer. This way, we can flexibly adjust the length of the local image patches most relevant to the current patch, enabling it to extract the most effective features for complex multilingual scene texts.

For multilingual scene text recognition, we must pay particular attention to the variability of the extracted features of two transformers for different languages. To achieve this, we design a loss function by including an extra cross-language rectification loss in addition to the vision loss:

where $Loss_{v}$ means the loss for the vision model, which is set following [5], $Loss_{clr}$ represents the cross-language rectification loss, and $\alpha$ is the weight coefficient, which is set to 0.01 based on our pilot studies. $Loss_{clr}$ is defined as follows:

where $L_{score}$ is the loss of the word coherence scores, and $L_{class}$ is the loss of language class identification:

where $MLP(\cdot)$ is a multilayer perceptron (MLP) in the transformers that performs feature extraction, and $T_{pt}$ and $T_{st}$ denote the extracted cross-language features from the vision transformer and pyramid transformer, respectively, $SoftCE(.)$ is the soft cross entropy of language class. With the help of the supplementary transformer, the coherence scores of the predicted characters in different patches are obtained by the linear inference layer in MLP, and the loss of the word coherence scores for multilingual texts can be estimated as follow:

where $y_{i},i\in[1,maxlen]$ represent inference result of the $i$-th character, $maxlen$ is the maximum length of the predicted characters.

Here, we discuss further the relationship between TANGER and ViTSTR [5]. Like ViTSTR, TANGER is an image-based transformer model without relying on language resources such as dictionaries or corpus. Similar to the traditional vision transformer [4], ViTSTR is mainly concerned with extracting features from a sequence of separate image patches. By contrast, TANGER explicitly takes into account the potential correlation between neighbouring visual patches, which is of paramount importance for feature extraction in complex scenes, such as multi-scale, multi-orientation and multilingual texts [33].

Datasets We validate the effectiveness of TANGER on three multilingual datasets, MLT17 [10], CTW1500 [11], RCTW17 [12], one monolingual dataset Totaltext [13], as well as a new multilingual dataset TsiText. MLT2017 [35], which comes from ICDAR 2017 Robust Reading Competition, contains 7200 training, 1800 validation and 9000 testing natural scene images in Arabic, Latin, Chinese, Japanese, Korean, and Bangla. CTW1500 [11] includes 1000 natural scene images for training and 500 for testing in English and Chinese languages. RCTW17 [12] contains 8034 training and 4229 testing images, including primarily on-scene texts in Chinese and a few in English. Totaltext [13] contains 1255 training images and 300 images for testing with 11459 annotated text instances of wild scenes in English.

TsiText contains 3600 training, 600 validation, and 1800 testing natural images collected from tourism scenes. All images in TsiText are downloaded from the Internet and do not contain sensitive information. There are an average of four text instances in per scene image, with a maximum of 61 text instances. Each text instance contains four different attributes, including position, character content, scene class, and language, and is annotated in a similar way to [11]. TsiText aims to provide complex multilingual scene texts containing rich and diverse travel-related scenes such as food, shop signs, traffic signs, directions, landmarks, and posters. It also includes multiple text types, including handwriting, print, and complex artistic styles. Finally, there is certain degrees of variety in text shapes, such as horizontal, multi-directional, curved, circular, and partially obscured texts, among others. To the best of our knowledge, this is the richest dataset for multilingual tourism scene texts.

Metrics and Parameter Configurations  We adopt the character-level accuracy as the metric for comparing the algorithms on multilingual scene text recognition tasks. In addition, the frequency of the word-level edit distance between the predicted and ground truth words is employed to compare the recognition performance of the algorithms under comparison on Latin and non-Latin language scene texts. To verify the effectiveness of TANGER on monolingual scene texts, we compare it with ViTSTR in terms of model parameters, speed, FLOPS as well as accuracy.

During the training, we adopt Adam with a mini-batch size of 192 to train the transformer models for 300 epochs from scratch with a learning rate of 0.001 on two A100 GPUs.

Table I lists the performance of the proposed TANGER with three state-of-the-art algorithms for multilingual scene text recognition, namely ABCNet+D [34], E2E-MLT [10], and Multiplexed [22]. ABCNet+D [34] is a dictionary-based recognition algorithm that incorporates dictionaries before handling ambiguous cases. E2E-MLT [10] is an end-to-end approach with a single fully convolutional network applicable to both Latin and non-Latin languages for multilingual scene text. Multiplexed [22] proposes a unified loss with disentangled loss and integrated loss, and trains multiple text recognition heads in an end-to-end manner for script identification in different languages. Note that E2E-MLT and Multiplexed are end-to-end methods for both text detection and recognition, however, this work focuses on the recognition performance of the compared algorithms.

As listed in Table I, we can see that TANGER achieves the best character-level accuracy on four multilingual scene text recognition tasks. Compared with ABCNet+D, a dictionary-guided approach, TANGER can significantly improve the accuracy by 14.8%, 11.9%, 10.2%, and 10.1% on MLT17, LSVT19, RCTW17, and TsiText datasets, respectively. The impressive results of TANGER may be attributed to the fact that visual feature extraction is more effective than using lexicons when dealing with a series of multilingual texts in complex scenes. Experimental results also show that TANGER consistently outperforms two end-to-end approaches on all multilingual scene text recognition tasks. Specifically, TANGER achieves a state-of-the-art accuracy of 55.1% and 89.9% on the MLT17 and TsiText datasets, respectively. We surmise that the competitive performance of TANGER comes mainly from the more effective representation of the neighboring visual features, as well as the use of cross-language rectification for complex multilingual scene texts, which has also been verified in our ablation studies.

Figure 4 exemplifies the recognition results by TANGER on six complex testing images from the TsiText dataset. We see that TANGER can successfully recognize the texts in multiple languages in various complex scenes. On these six images, TANGER achieves an accuracy of 99.1%, while ABCNet+D, E2E-MLT, and Multiplexed achieve 91.2%, 94.6% and 95.1%, respectively, confirming the benefits of the proposed adaptive n-grams embedding and cross-language rectification mechanisms.

To further demonstrate the performance of TANGER for multilingual scene text recognition on both Latin and non-Latin languages, we examine the histograms of edit distance [36] between the pairs of predicted and the ground truth words for Arabic (non-Latin) and Latin language texts on the MLT17 dataset. Note that the smaller the edit distance, the smaller the recognition error of the algorithm is, and the words are recognized correctly when the edit distance is 0. From Fig. 5, we observe that TANGER can obtain the maximum frequency of occurrences at an edit distance of 0 among all compared methods for both Arabic and Latin languages. In addition, the edit distance of TANGER is always smaller than 5 for Latin languages. Overall, we can conclude that TANGER outperforms the compared methods by observing the histograms of the edit distance, further demonstrating the effectiveness of TANGER for multilingual scene text recognition.

Here we evaluate TANGER for monolingual scene text recognition on the Totaltext dataset by comparing it with ViTSTR [5]. Table II reports the comparative results in terms of the accuracy, speed, model parameters, and FLOPS for three variants of the two compared models, tiny, small, and base versions. Since the transformer-based ViTSTR model is limited to Latin language text recognition we compared the two algorithms on Totaltext, which contains English texts only. From the results in Table II, we can find that TANGER enhances the recognition accuracies by 2.1%, 3.6% and 1.1% for the tiny, small and base versions, respectively, compared to ViTSTR, without considerably slowing down the inference speed.

Interestingly, we find that the coherence score loss designed for cross-language rectification in our method is also helpful for enhancing the recognition performance for monolingual tasks, which may be attributed to the supplementary transformer that takes neighboring patches into account for complex scene text extraction. Figure 6 provides some selected cases, where ViTSTR makes mistakes whilst TANGER correctly recognizes the scene texts in some artistic fonts, or multi-scale and multi-orientation texts.

In this section, we conduct ablation studies on MLT17 and TsiText datasets. Our goal is two-fold. First, to assess the importance of the proposed adaptive n-grams embeddings for multilingual text representation, we adopt different but fixed values of $n$ on local neighboring image patches for multilingual text representation compared to the proposed adaptive n-grams embeddings in TANGER. Then, we examine the effect of the proposed cross-linguistic loss function on TANGER’s recognition performance.

Adaptive n-grams Patch Embedding The proposed adaptive n-grams embedding method aims to choose an optimal $n$ for each image patch. To demonstrate its benefit, we compare the performance of TANGER variants when $n$ is set to $n=2,3,4,5$, respectively. The comparative results are given Table III, from which we can clearly see that the adaptive n-grams embedding has led to significant improvements in the performance for multilingual scene text recognition on both MLT17 and TsiText. Specifically, the adaptive n-grams embedding can achieve a maximum performance increase of 1.9% and 3.2% on MLT17 and TsiText, respectively, compared with the best variant with a fixed $n$. A possible explanation is that the font sizes of different scene texts in different languages may differ dramatically, which requires an adaptive setting of the n-grams embedding.

Cross-language Rectification  Next we consider the effect of the proposed cross-language loss functions on the performance of TANGER on the MLT17 dataset. We consider two metrics for comparison, including the character-based accuracy and the frequencies of the word-level edit distance. From the results in Table IV, we see that the accuracy of TANGER has an increase of 3.5% on MLT17 and 2.3% on TsiText with cross-language rectification loss. These results indicate that the cross-language loss is able to improve recognition performance on multilingual scenes.