Ensemble Learning for Vietnamese Scene Text Spotting in Urban Environments

In recent decades, the growth of deep learning significantly contributes to the advancement of scene text spotting. There are typically approaches: two-stage and end-to-end approaches. The two-stage approach consists of two major stages: scene text detection and scene text recognition. A plethora of effective algorithms have been proposed for each stage, including DB/DB++ [14, 15], SAST [16], EAST [17], etc. for detection and SPIN [18], SRN [19], SVTR [20], ABINet [21], etc. for recognization. Efforts have also been directed towards integrating detection and recognition processes, with works like TextBoxes by Liao et al. [22] combining a single-shot detector and a text recognizer. Furthermore, Nguyen et al. [10] leveraged a dictionary to generate a list of potential results and identify the most visually compatible outcome with the text’s appearance to train and improve the recognition stage. Additionally, He et al. contributed VinText dataset [10], a challenging benchmark for Vietnamese scene text spotting. Despite these advancements, existing techniques treat detection and recognition as independent tasks, lacking seamless information exchange between them.

Meanwhile, end-to-end approach focuses on merging detection and recognition into a unified system. Peng et al. [23] presented an end-to-end scene text spotting method that approaches scene text spotting as a sequence prediction task. Wang et al. proposed PGNet [24], which revolves around developing a model that combines a detection unit and a recognition module, allowing for shared CNN features and joint training. Huang et al. [1] introduced a novel end-to-end scene text spotting framework called SwinTextSpotter. By integrating both functionalities into a single algorithm, the resulting end-to-end model becomes more compact and efficient, leading to improved speed and performance recognition.

Ensemble learning represents a powerful methodology that amalgamates the strengths and benefits of multiple approaches, culminating in a superior model. As early as 1996, Rosen proposed [25] a technique called decorrelation network training to enhance the accuracy of regression learning in ensemble neural networks. Deng et al. [26] introduced linear and log-linear stacking methods for ensemble learning, focusing on applications to speech class posterior probabilities computed by convolutional, recurrent, and fully-connected deep neural networks. Recently, Casado-García and Heras [27] explored ensemble methods for object detection, addressing the challenges associated with ensembling object detectors, such as the limitations of existing ensemble approaches that depended on specific detection models or frameworks. Leveraging the advantage of different methods, we introduce an ensemble learning framework to integrate results from individual models of both the two-stage and end-to-end approaches to improve the performance of scene text spotting.

We present an ensemble learning framework designed to effectively combine the outputs of multiple scene text spotting methods. Fig. 2 illustrates the overview of our method, which consists of three main components: data converter, base models, and meta-model.

In the first stage of the workflow, we transform the raw images and labels of the VinText dataset [10] into different formats suitable for training and predicting of each base model. The data used to train the recognition model is images with a single word, while the dataset includes images with many different text lines which contain more than one word. Therefore, we crop the original images to smaller images that contain only one word and also create new labels for the new images.

After that, base models are trained on converted data and used for prediction to obtain initial results. Subsequently, the results of all models are combined with an ensemble technique and are essential data to create a meta-model. The meta-model is then used for prediction and generates final result, which is expected to be better than the initial ones.

Our posed Meta-model undergoes two processes: merging non-overlapping text boxes and merging overlapping text boxes. Given an image and $n$ base models, the input of our ensemble algorithm is a list of prediction results: $D=[D_{1},\ldots,D_{n}]$ where each $D_{i}$, with $i\in\{1,\ldots,n\}$, is a list of predicted text boxes $D_{i}=[B_{1},B_{2},\ldots]$. In general, each $D_{i}$ comes from the predictions of a scene text spotting model using a particular method $M_{i}$ for a given image.

All experiments were performed on Google Colab with the following configuration GPU T4 and 13GB of RAM. Detection methods were pre-trained on the ICDAR 2015 [28] for $1,000$ epochs. The initialized learning rate was $7\times 10^{-3}$ and followed a decay learning rate schedule with a decay factor of $0.9$. After that, the models were fine-tuned for $30$ epochs on VinText dataset [10], and weight decay regularization was applied with a value of $10^{-4}$. The dataset underwent decoding in BGR format, and various data augmentation techniques were applied, such as horizontal flipping, affine transformations (rotations between $-10^{\circ}$ to $10^{\circ}$), and resizing between $0.5$ and $3$ times the original size.

Recognition methods were pre-trained on MJSynth [29, 30] and SynthText [31] datasets. After that, they underwent a total of 20 epochs of fine-tuning on the VinText dataset [10]. We utilized the Adam optimizer [32] with a $\beta_{1}$ value of 0.9 and $\beta_{2}$ value of $0.99$. Gradient clipping with a norm threshold of $20.0$ was applied to prevent exploding gradients. The learning rate followed a piecewise schedule, decaying at epoch 20 tow $10^{-4}$ and $10^{-5}$ for subsequent steps. $L_{2}$ regularization with a factor of 0 was employed.

The VinText dataset [10], a recently proposed Vietnamese text dataset, was used to evaluate methods. This benchmark dataset has a total of 56K text instances and 2,000 images, which consists of 1,200 training images, 500 testing images, and 300 unseen test images. Most text instances have usually too many patterns and chaotic scenes with scene text instances of various types, appearances, sizes, and orientations. We evaluated methods on unseen test images.

In order to evaluate the performance and effectiveness of methods, we used Character Accuracy ($CA$) metric for recognition task, Precision ($P$), Recall ($R$), and F-measure ($F1$) metrics for detection task.

We carefully analyzed the results of detection and recognition methods in Table I and Table II to select appropriate models for our ensemble learning framework. The objective is to identify models that excel in distinct aspects and can synergistically complement each other when combined.

In the evaluation of detection methods from Table I, we meticulously analyze their performance on both the ICDAR2015 [28] and Total-Text datasets [34]. Notably, DB++[15], EAST[17], and SAST [16] stand out in different metrics. DB++ [15] showcases high precision, while EAST [17] demonstrates good recall, and SAST exhibits a strong F-measure specifically on the Total-Text dataset [34].

Shifting our focus to the recognition methods from Table II, we assess their average accuracy. SPIN [18], ABINet [21], and RobustScanner [39] stand out as top performers, demonstrating remarkable accuracy in text recognition. However, upon merging the RobustScanner [39] with the detection methods, we observe challenges with overlapping text boxes, leading to a decline in overall performance. As a result, we make the decision to exclude the RobustScanner [39] from our final ensemble, striking a balance between detection and recognition performance.

After the individual evaluations, we selected DB++ [15], EAST [17], SAST [16])for the detection task, SPIN [18], ABINet [21], SRN [19] for the recognition task. Additionally, PSNet [24] was chosen as an end-to-end model capable of addressing both detection and recognition aspects. Subsequently, we employed our ensemble algorithm to combine model pairs, harnessing the unique strengths of each end-to-end model. The results of this comprehensive ensemble are presented in Table III, showcasing a powerful and complementary fusion of models for Vietnamese scene text spotting.

While potential of ensemble learning is vast, it is crucial to recognize that mere combination of more models does not always guarantee improved results. In fact, some models, when merged, may yield even worse performance compared to their individual counterparts (see Table III). Our rigorous experimentation has led us to discover that the most optimal ensemble result is achieved by combining pairs of methods DB++[15] / SPIN [18] and SAST [17]/ ABINet[21]. This carefully curated selection of effective ensemble pairs sheds light on the delicate interplay between methods and underscores the significance of thoughtful model combination for achieving remarkable performance gains in scene text spotting. Indeed, our ensemble framework delivers improved performance in all metrics (See Fig. 3).