Complex Handwriting Trajectory Recovery: Evaluation Metrics and Algorithm

There is not much work on evaluation metrics for trajectory recovery. Many techniques rely on the human vision to qualitatively assess the recovery quality [25, 4, 7, 26, 14, 21]. However, these non-quantitative human evaluations are expensive, time-consuming and inconsistent.

Trajectory recovery has been proved beneficial to handwriting recognition. As a byproduct, one may use the final recognition accuracy to compare the recovery efficacy [1, 15, 8, 12]. Instead of directly assessing the quality of different recovery trajectories, they compare the accuracy of the recognition models among which the only difference is the intermediate recovery trajectory. Though, to some extent, this method reflects the recovery quality, it is usually disturbed by the recognition model, and only provides a relative evaluation.

Most of direct quantitative metrics only focus on the evaluation of the writing order. Lau et al.  [16] designed a ranking model to assess the order of stroke endpoints, and Nel et al.  [20] designed a hidden Markov model to assess the writing order of local trajectories such as stroke loops. However, these methods are unable to assess the sequence points’ deviation to the groundtruth. Hence, these two evaluations are seldom adopted in the subsequent studies.

Metrics borrowed from other fields have also been used, such as the RMSE borrowed from signal processing and the DTW from speech recognition [10]. RMSE directly calculates distances between two sequences and strictly limits the number of trajectory points, which makes it hard to use in practice. It is too strict to require the recovered trajectory to recall exactly all the groundtruth points one-by-one, since trajectory recovery is an ill-posed problem that the unique recovery solution cannot be obtained without constraints [27]. DTW introduces an elastic matching mechanism to obtain the most possible alignment between two trajectories. However, DTW is not robust to the number of trajectory points, and prefers trajectories with fewer points. Actually, the number of points is irrelevant to the writing order or glyph fidelity, and shouldn’t affect the judgement of the recovery quality.

Aforementioned quantitative metrics only focus on the evaluation of the writing order, and neither involves the glyph fidelity. Note that the glyph correctness is also an essential aspect of trajectory recovery, since glyphs reflect the content of characters and the writing styles of a specific writer. Only one of the latest trajectory recovery work [2] borrows the metric LPIPS [33] which compares the deep features of images. However, LPIPS is not suitable for such a fine task of trajectory recovery, as we observed that the deep features are not informative enough to distinguish images with only a few pixel-level glyph differences.

Early studies in the 1990s relied on heuristic rules, often including two major modules: local area analysis and global trajectory recovery [4, 26, 13, 14, 27]. These algorithms are difficult to devise, as they rely on delicate hand-engineered features. Rule-based methods are sophisticated, and not flexible enough to handle most of the practical cases, hence these methods are considered not robust, in particular for characters with both complex glyph and long trajectory.

Inspired by the remarkable progress in deep-learning-based image analysis and sequence generation over the last few years, deep neural networks are used for trajectory recovery. Sumi et al. [30] applied variational autoencoders to mutually convert online trajectories and offline handwritten character images, and their method can handle single-stroke English letters with simple glyph structures. Nevertheless, instead of predicting the entire trajectory from a plain encoding result, we can consider employing a selection mechanism (e.g., attention mechanism) to analyze the glyph structure between the prediction of two successive points, since the relative position of continuous points can be quite variable. Zhao et al.  [36, 37] proposed a CNN model to iteratively generate stroke point sequence. However, besides CNNs, we also need to consider applying RNNs to analyze the context in handwriting trajectories, which may contribute to the recovery of long trajectories.

Bhunia et al. [3] introduced an encoder-decoder model to recover the trajectory of single-stroke Indic scripts. This algorithm employs a one-dimensional feature map to encode characters, however, it needs more spatial information to tackle complex handwritings (on a two-dimensional plane). Nguyen et al. [21] improved the encoder-decoder model by introducing a Gaussian mixture model (GMM), and tried to recover multi-stroke trajectories, in particular Japanese scripts. However, since the prediction difficulty of long trajectories remains unsolved, this method does not perform well in the case of complex characters. Archibald et al. [2] adapted the encoder-decoder model to English text with arbitrary width, which attends to text-line-level trajectory recovery, but not designed specifically for complex glyph and long trajectory sequences, either.

We propose the first glyph fidelity metric, Adaptive Intersection on Union (AIoU). It firstly performs a binarization process on the input image $I$ using a thresholding algorithm, e.g., OTSU [24], to obtain the ground-truth binary mask, denoted $G$, which indicates whether a pixel belongs to a character stroke. Meanwhile, the predicted trajectory $p$ is rendered into a bitmap (predicted mask) of width $1$ pixel, by drawing lines between neighboring points if they belong to a stroke, denoted $P$. We define the IoU (Intersection over Union) between $G$ and $P$ as follow, which is similar to the mask IoU [5] $IoU(G,P)={|G\cap P|}/{|G\cup P|}$.

An input handwritten character usually has various stroke widths while the predicted stroke widths are fixed, nevertheless, the stroke width shouldn’t influence the assessment of the glyph similarity. To reduce the impacts of stroke width, we propose a dynamic dilation algorithm to adjust the stroke width adaptively. Concretely, as shown in Fig.  2, we adopt a dilation algorithm [9] with a kernel of $3\times 3$ to widen the stroke along until the IoU score reaches the maximum, denoted $AIoU(G,P)$. Since the image $I$ is extracted as the binary mask $G$, the ground-truth trajectory of $I$ is not involved in the calculation of the AIoU, making the criteria still effective even without the ground-truth trajectory.

Variable lengths make it hard to align handwriting trajectories. As shown in Fig. 3, when comparing two handwriting trajectories with different lengths, the direct one-to-one stroke-point correspondence cannot represent the correct alignment of strokes. We modify the well-known DTW [10] to compare two trajectories whose lengths are allowed to be different, which uses an elastic matching mechanism to obtain the most possible alignment.

The original DTW relies on the concept of alignment paths, $DTW(q,p)=\min_{\phi}\left\{\sum_{t=1}^{T}d\left(q_{i_{t}},p_{j_{t}}\right)\right\},$ where the minimization is taken over all possible alignment paths $\phi$ (which is solved by a dynamic programming), $T\leq M+N$ is the alignment length and $d\left(q_{i_{t}},p_{j_{t}}\right)$ refers to the (Euclidean) distance between the two potentially matched (determined by $\phi$) points $q_{i_{t}}$ and $p_{j_{t}}$.

We observe that the original DTW empirically behaves like a monotonic function of $T$ that is usually proportional to $N$, so it in general prefers short strokes and even gives good score to incomplete strokes, which is what we want to get rid of. Intuitively, this phenomenon is interpretable: DTW is the minimization of a sum of $T$ terms, and $T$ depends on $N$. We suggest a normalized version of DTW, called the length-independent DTW (LDTW)

It is worth noting that, since the alignment problem also exists during the training process, we use a soft dynamic time warping loss (SDTW Loss) [6] to realize a global-alignment optimization, see Section 4.3.

In this part, we firstly investigate how the values of our proposed metrics(AIoU and LDTW) and other recently used metrics change in response to the errors in different magnitudes. Secondly, we analyze the impacts of the changes in the number of trajectory points and stroke width to LDTW and AIoU respectively.

Existing methods (e.g., DED-Net[3], Cross-VAE[30]) compress the feature to only one dimension vector, which are not informative enough to maintain the complex two-dimensional information of characters. Actually, every two-dimensional stroke can be projected to horizontal and vertical axes. In the double-stream parsing encoder, we construct two CRNN [29] branches denoted as $CRNN_{X}$ and $CRNN_{Y}$ to decouple handwriting images to horizontal and vertical features $V_{x}$ and $V_{y}$, which are complementary in two perpendicular directions for the parsing of glyph structure. Each branch is composed of a CNN to extract the vertical or horizontal stroke features, and a 3-layer BiLSTM to analyze the relationship between strokes, e.g., which stroke should be drawn earlier, what is the relative position between strokes. To extract stroke features of single direction in the CNN of each stream, we use asymmetric poolings, which is found to be effective experimentally. Details of proposed CNNs are shown in Fig. 5.

The stroke region is always sparse in a handwriting image, and the blank background disturbs the stroke feature extraction. To this end, we use an attention mechanism to attend to the stroke foreground. The attention mechanism fuses $V_{x}$ and $V_{y}$, and obtains the attention score $s_{i}$ of each feature $v_{i}$ to let the glyph parsing feature $Z$ focus on the stroke foreground:

where $V$ is obtained by concatenating $V_{x}$ and $V_{y}$, $v_{i}$ is the component of $V$, $|V|$ the length of $V$, $U$ is learnable parameters of a fully-connected layer. We apply a simplified attention strategy to acquire the attention score $s_{i}$ of the feature $v_{i}$.

We adopt a 3-layer LSTM as the decoder to predict the trajectory points sequentially. In particular, the decoder uses the position and the pen tip state at time step $i-1$ to predict those at time step $i$, similar to [3, 21].

During decoding, previous trajectory recovery methods [3, 21] only utilize the initial character coding. As a result, the forgetting phenomenon of RNN [11] causes the so-called trajectory-point position drifting problem during the subsequent decoding steps, especially for characters with long trajectories. To alleviate this drifting problem, we propose a global tracing mechanism by using the glyph parsing feature $Z$ at each decoding step. The whole decoding process is shown in Fig. 6.

Similar to [3, 21], we use the $L_{1}$ regression loss and the cross-entropy loss to optimize the coordinates and the pen tip states of the trajectory points, respectively. Similar to [34], during the process of optimizing pen tip states, we define three states “pen-down", “pen-up" and “end-of-sequence" respectively, which are denoted as $s_{i}^{1},s_{i}^{2},s_{i}^{3}$ of $p_{i}$. It is obvious that “pen-down" data points are much more than the other two classes. To solve the biased dataset issue, we add weights (“pen-down" is set to $1$, “pen-up" $5$, and “end-of-sequence" $1$, respectively) to the cross-entropy loss.

These hard-losses are insufficient because they require a one-to-one stroke-point correspondence, which is too strict for handwriting trajectories of variable lengths. We borrow the soft dynamic time warping loss (SDTW Loss) [6], which has never been used for trajectory recovery, to supplement the global-alignment goal of the whole trajectory and to alleviate the alignment learning problem.

The DTW algorithm can solve the alignment issue during optimization using an elastic matching mechanism. However, since containing the hard minimization operation which is not differentiable, DTW cannot be used as an loss function directly. Hence, we place the minimization operation by a soft-minimization $\min^{\gamma}\left\{a_{1},\ldots,a_{n}\right\}=\gamma\log\sum_{i=1}^{n}e^{-a_{i}/\gamma},\gamma>0.$ We define the SDTW loss

The total loss is $L=\lambda_{1}L_{1}+\lambda_{2}L_{wce}+\lambda_{3}L_{sdtw}$, where $\lambda_{1},\lambda_{2},\lambda_{3}$ are parameters to balance the effects of the $L_{1}$ regression loss, the weighted cross-entropy loss and the SDTW loss, which are set to $0.5,1$ and $1/6000$ in our experiments.

Information of datasets is given as follows, and statistics of them are in Appendix.

Chinese script. CASIA-OLHWDB(1.0-1.2) [17] is a million-level online handwritten character dataset. We conduct experiments on all of the Chinese characters from OLHWDB1.1 which covers the most frequently used characters of GB2312-80. The largest amounts of trajectory points and strokes reach 283 (with an average of 61) and 29(average of 6), respectively.

English script. We collect all of the English samples from the symbol part of CASIA-OLHWDB(1.0-1.2), covering 52 classes of English letters.

Japanese script. Referring to [21], we conduct Japanese handwriting recovery experiments on two datasets including Nakayosi_t-98-09 for training and Kuchibue_d-96-02 for testing. The largest amount of trajectory points and strokes reach 3544 (with an average of 111) and 35 (average of 6), respectively.

Indic script. Tamil dataset [3] contains samples of 156 character classes. The largest amount of trajectory points reach 1832 (average of 146).

Implementation Details. We normalize the online trajectories to [0, 64) range. In addition, in terms of the Japanese and Indic datasets, because their points densities are so high that points may overlap each other after the rescaling process, we remove the redundant points in the overlapping areas and then down-sample remaining trajectory points by half. We convert the online data to its offline equivalent by rendering the image using the online coordinate points. Although the rendered images are not real offline patterns, they are useful to evaluate the performance of trajectory recovery [3, 21]. In addition, we train our model 500,000 iterations on the Chinese and Japanese datasets, and 200,000 iterations on the English and Indic dataset, with a RTX3090 GPU. The batch size is set to 512. The optimizer is Adam with the learning rate of 0.001.

In this section, we quantitatively evaluate the quality of trajectory, recovered by our PEN-Net and existing state-of-the-art methods including DED-Net [3], Cross-VAE [30] and Kanji-Net [21], on the above-mentioned four datasets via five different evaluation metrics of which AIoU and LDTW are proposed by us.

As Table. 1 shows, our PEN-Net expresses satisfactory and superior performance compared to other approaches, with an average of 13% to 20% gap away from the second-best in all of the five evaluation criteria on the first four datasets. Moreover, to further validate the models’ effects for complex handwritings, we build two subsets by extracting 5% of samples with the most strokes from the Japanese and Chinese testing set independently, where the number of strokes of each sample is over 15 and 10 corresponding to the two languages. According to the data(Chinese/Japanese complex in the table), PEN-Net still performs better than SOTA methods. Particularly, on Japanese complex set, PEN-Net expresses superior performance compared to other approaches, with an average of 27.3% gap away from the second-best in all of the five evaluation criteria.

As the visualization results in Fig. 7, Cross-VAE [30], Kanji-Net [21] and DED-Net [3] can recover simple characters’ trajectories (English, Indic, and part of Japanese characters). However, their methods exhibit error phenomena, such as stroke duplication and trajectory deviation, in complex situations. Cross-VAE [30] may fail at recovering trajectories of complex characters (Chinese and Japanese), and Kanji-Net [21] cannot recover the whole trajectory of complex Japanese characters. In contrast, our PEN-Net makes accurate and reliable recovery prediction on both simple and complex characters, demonstrating an outstanding performance in terms of both visualization and quantitative metrics compared with the three prior SOTA works.

In this section, we conduct ablation experiments on the effectiveness of PEN-Net’s core components, including double-stream (DS) mechanism, global tracing (GLT) mechanism, attention (ATT) mechanism and SDTW loss. We use Chinese dataset to evaluate PEN-Net’s performance for complex handwriting trajectory recovery. The evaluation metrics are the same as in Section 5.3. The experiment results are reported in Table. 2 in which the first row relates to the full model with all components, and we gradually ablate each component one-by-one down to the plain baseline model at the bottom row.

Double-stream mechanism. In this test, we remove CRNNy from the backbone of the double-stream encoder and remains the CRNNx. As the 4th and 5th rows in Table. 2 show, CRNNy contributes 2.7$\%$ and 7.14$\%$ improvement on glyph fidelity metrics (AIoU and LPIPS), and 4.4$\%$, 3.41$\%$, 5.1$\%$ improvement on writing order metrics (LNDTW, DTW, RMSE). Additionally, as Fig. 8 reveals, the model without CRNNy cannot make an accurate prediction on vertical strokes of Chinese characters.

Global tracing mechanism. As the 3rd and 4th row in Table. 2 show, GLT further improves the performance based on all the metrics except RMSE. The value rise in RMSE, from 14.40 to 15.05, is because this generic metric overemphasizes the point-by-point absolute deviation, which negatively affects the overall quality evaluation of the handwriting trajectory matching. In addition, as Fig. 8 shows, drifting phenomenon occurs in the recovered trajectories if GLT is removed, while, in contrast, the phenomenon disappears vise versa.

Attention mechanism. As the 2nd and 3rd rows showed in Table. 2, ATT also improves the performance of the model. Furthermore, as the attention heat-map visualization showed in Fig. 9, the stroke region always attracts more attention(in red color) than the background area(in blue color) in a character image.

SDTW loss. As the 1st and the 2nd rows showed in Table. 2, the SDTW loss also contributes to the performance enhancement of the model.

Finally, based on these ablation studies, the PEN-Net dramatically boost the trajectory recovery performance over the baseline by 10.8$\%$ on AIoU, 23.6$\%$ on LDTW, 22.5$\%$ on DTW, 5.1$\%$ on RMSE, 19.3$\%$ on LPIPS. Consequently, we claim that the four components of PEN-Net: double-stream mechanism, global tracing, attention mechanism and SDTW loss, all play pivotal roles w.r.t. the final performance of trajectory recovery.