Geometric Perception based Efficient Text Recognition
^††thanks: Identify applicable funding agency here. If none, delete this.

The task of Optical Character Recognition (OCR) has been widely explored within the field of computer vision. Every character recognition task can be classified into two subtasks: character localization and character classification. With the introduction of the Region Proposal Network (RPN) [1] and other advancements in object detection, the performance of the character spotting task has been steadily increasing. Also, with the attention mechanism [2] and autoencoder-based segmentation mechanisms [3],the performance of the character recognition task improved.

However, this performance improvement was often accompanied by a significant increase in model size and computational complexity, despite the underlying task remaining the same. Specifically, State Of The Art (SOTA) models such as RF-Learner [4] and SPIN [5] have the parameter space on the scale of $10^{6}$. The usage of sizable models can be problematic in situations where real-time or near real-time operation is required. For example, deploying models on edge devices or training models in a federated learning fashion can benefit greatly from having a lightweight model architecture.

In this paper, we introduce an efficient and high-performance Deep Learning (DL) model architecture that is specifically designed to minimize the model parameter space. In addition to the very high inference speed, due to the significantly small model size, the model could be deployed on edge devices and even be trained on edge devices in a federated learning fashion. Since data privacy is a high priority and data drift is an unavoidable hazard in Machine Learning (ML), training an OCR model in a federated learning schema is profitable in both research and industry contexts. With the introduction of our architecture, the OCR model could achieve SOTA performance via federated training, as well.

Unlike other image recognition tasks, the only significant factor in the text/character recognition task is the geometry of the character. Factors such as color, depth, or hierarchical features have minimal influence on the output of the task. However, most models extract localized features from the RGB pixel space without explicitly designing for an efficient feature-extracting process. As a result, to achieve higher accuracy, the model size has to be increased. Consequently, the model training sample space has to be expanded to minimize the estimation error in empirical risk minimization. As a solution, the proposed model extracts geometric features from the grayscale image to identify the sequence of characters, without explicitly processing character locations.

This paper introduces a novel model architecture composed of a novel image feature encoding strategy and feature projection methods. Furthermore, the model is trained on images with a fixed number of characters, and thus outputs a fixed number of object labels, without specific training or predicting the object locations/ bounding boxes. Ultimately, this is targeting significantly smaller parameter spaces, and less computationally complex operations to tackle the real-time inference scenarios and optimal training in a federated learning setup. The model consists of two novel implementations: (1) a geometrical shape-based encoder, and (2) a feature localization unit for ground-truth label sequence prediction.

The model assumes the input image only contains a row of characters in the horizontally aligned format. The proposed model’s input format is a standard image format in the Scene Text Recognition (STR) field. However, any other text alignment such as curved or tilted can be used for model training with a preprocessing mechanism such as Thin Plate Spline transformation (TPS) [6]. Furthermore, the proposed model can be used in an end-to-end text recognition task, simply by adding a bounding box detection model such as SegLink [7] or BoundaryE2E [8] before the recognition model.

For the first novel feature, the paper introduces a very lightweight encoder unit that encodes the shape of each character found in the image. In the real world, we could safely say that someone learned a shape, by examining how someone can reorganize a striped image of a shape, back together [1(b)]. As a deep learning counterpart, we propose LSTM-based encoders trained to learn the aforementioned principal idea. Furthermore, the following layer configurations and the objective function are defined to ensure the encoder learns as expected.

The second novelty is that the encoded feature space is mapped into a logit layer by a unique spatial attention mechanism. This submodel consists of several unique innovations to achieve SOTA performance. As unveiling those new implementations from input to end, the model is composed of an interpretable feature region to target the label mapping block. Not only is this block mathematically stable to prevent the model from collapsing but it’s also very lightweight compared to the general attention [2] mechanism. On the other hand, the paper introduces a fully-convolutional label prediction mechanism, loosely resembling the auto-encoder architecture [9]. Furthermore, the proposed architecture reveals a superior logit layer format to traditional multi-class multi-label classification [10] logit format in several aspects.

In addition to the novel model architecture, the paper also introduces a sophisticated synthetic text sample generation python library called digitgen. digitgen library builds as a solution to several drawbacks found in well-known datasets like SynthText [11] and MJSynth [12]. The implemented image synthesizing algorithm enables the generation of a dataset of any size, irrespective of the memory size of the system. Furthermore, it is able to generate a COCO-structured annotation file [13] to rebuild the data loaders under any deep learning framework. Additionally, the characters in the images are generated completely pseudo-randomly, unlike in MJSynth or SynthText. This random character definition is proposed to mitigate the propensity added by the lexicon-based scene text generation. Despite synthetic generation, the proposed library provides a variety of augmentations such as adding random noise, adding shadow patches, adding light bursts, changing font family, font size, etc.

To summarize, the paper introduces a novel model architecture to address the task of regular scene text recognition (GeoTRNet), and a new synthetic regular scene text image generating python library (digitgen). With the aforementioned introduction, the paper empirically proves the model’s SOTA performance under near real-world image datasets. Additionally, the experimental results under federated learning with low client sample sizes, serve as evidence for the model’s capability in estimation error minimization. Furthermore, the conducted experiments on metric learning for the signature verification task show the wide usability of the proposed architecture.

The field of scene text recognition has gained a lot of progress throughout the years and many solutions model the task with a sequence prediction objective function. Graves et. al. [14] have approached this task by handcrafted geometrical features such as mean gray value and gradient of pixel intensity. Su and Lu [15] achieve the feature extraction using HOG features obtained from the cropped images of the words. Most later works have turned towards deep learning to replace hand-crafted feature extraction. Shi et. al. [16] were the first to utilize a combination of CNNs and RNNs to tackle the task of text recognition (CRNN). The CNN was used to extract a set of feature maps from the input image. The feature map was then broken down into one-pixel-wide columns of pixels and the $i^{t}h$ pixel columns of each feature map were concatenated to form the feature vectors which were then fed into an RNN. They further improved on this by swapping the RNN for a Bidirectional LSTM which showed better performance. This setup proved to be highly compact and efficient due to the removal of fully connected layers. Shi et. al. [16] use a CTC layer [17] to define the conditional probability of the sequence of labels. However, the computationally suboptimal nature of the CTC loss makes the model training highly volatile and time-consuming. To address this issue, Xie et. al. [18] introduced the Aggregation Cross-Entropy loss which was highly efficient. In [19], Baek et. al. introduce a four-stage STR framework that most STR models fit into. Of these, the first stage is defined to be the irregular/curved text image to regular text image transformation. In the case of irregular text, the transformation stage is a significant necessity to ease the processing for subsequent stages. However, in this paper, we will build on tasks where the model receives regular text as input. The task of transformation is handled by different models using various methods. Shi et. al. [7] address the case of oriented text detection using the SegLink model which detects individual characters, their orientations, and the links between characters of the same word. BoundaryE2E [8] detects the boundary points around the text and uses them to warp the image to horizontal text. A similar mechanism is used by Qiao et. al. [20], where the text boundaries and corners are detected using semantic segmentation, prior to the boundary point detection. Liu et. al. [21] designed ABCNet to predict Bezier curves that would fit the text on the image. This was followed by the BezierAlign module which aligned the text as required. Zhang et. al. [5] achieve both chromatic and geometric rectification using the SPIN network. As noted in [5], this network can work in complement with other transformation models or with a standalone recognition model such as the one proposed in this paper.

The GeoRTNet model, even in its current form, opens a large number of doors with solutions in the trending deep learning topics such as privacy-aware model training, high-speed inference, and open-world object detection. Specifically, the simplistic and very-lightweight nature of the model lays the ground for optimal data distributed training in tasks such as credit-card security number reading and barcode number panel recognition. On the other hand, microsecond scale inference time comes in handy with tasks like highway vehicle number plate detection. Furthermore, the highly reliable nature and the well-explainable nature make the model suitable to operate on sensitive datasets.

However, the current model is restricted to only the digit recognition step on regular text scenes. The introduced model could be used to improve any multi-step or end-to-end STR model, just by replacing the model recognition header with the proposed model. This upgrade is guaranteed to provide impressive performance with drastic inference time reduction. Stated model upgrades can be done with minimal intervention in STR models such as BoundaryE2E [8] and SegLink[7].

GeoTRNet merged with text localization and a pre-processing model [8] could lead to significant improvements in the model accuracy, model reliability, and inference speed, because the highly reliable nature of the GeoTRNet, and the significant model weight reduction, even in the recognition header, leads to the stated claims respectively. As stated in the introduction, with transformations like SPT, the updated STR model supports irregular text images as well.

As empirically proven, the GeoTRNet model could also achieve outstanding performance in few-shot learning tasks such as federated learning and metric learning. The proposed model inherently could be used in tasks like signature verification, and hand-written letter quality/shape evaluation tasks, especially under any task that humans perceive solely from the object shape, with a very simple change to the core model, as shown in experiments.

In this paper, we introduced GeoTRNet, a novel model architecture for geometry-based text recognition. GeoTRNet is capable of recognizing a set of characters with SOTA accuracy within a fraction of a millisecond. This performance was maintained on text recognition tasks under standard and synthetic datasets with a model of only 1% of the parameter size. Furthermore, the model was proven to perform well, even under the few-shot learning schemes such as federated learning and metrics learning. Also, GeoTRNet comes with high model explainability and very high resilience to adversarial attacks. Hopefully, the proposed architecture can open new doors in various other related tasks in the computer vision field. We are certain that the core concepts we introduced here could be used to build smart models in all areas of deep learning.

We thank S.M.Chandanayake and L.T.N. Wickremasinghe for their support on this project.