Keypoint based Sign Language Translation without Glosses

As we mentioned earlier, research on sign language translation has been steadily conducted. However, there are several reasons for the slow development of the SLT system. First, there is not enough data for sign language translation. Sign language is used differently depending on each country and region, so the data that must be used when translating a language is different. Therefore, it was necessary to collect sign language datasets by country directly, which was difficult because of the large cost. In addition, previous studies only dealt with translation between sign language and text, and even though video information was not included, the average of total number of words was very small, about 3,000 [14, 15, 16]. However, as algorithms developed, they laid the foundation for building sign language datasets. In particular, with the development of algorithms for weakly announcement data [17, 18] and the development of algorithms for pose estimation [19, 20], researchers felt the need for sign language data and began to build datasets.

This development led to the creation of German sign language datasets RWTH-PHOENIX-Weather 2012 [21], RWTH-PHOENIX-Weather 2014 [22], and RWTH-PHOENIX-Weather 2014-T [4] as well as the American sign language dataset ASLG-PC12 [23] and Korean sign language dataset KETI [10].

With the construction of various datasets, the SLT task has developed through Deep Learning. In many CSLR or SLT studies, the frame is encoded in feature form through Convolution Neural Networks (CNNs) [24]. Camagoz et al. [4] extracted features of frames using CNN and proposed an SLT method through the Encoding-Decoding method [7] using Recurrent Neural Networks (RNN). Similarly, Zhou et al. [25] proposed a CSLR model that combines CNN and RNN.

As Transformer [26] was in the spotlight in the NMT task, many SLT models using Transformer were developed. Yin et al. [27] achieved the state of the arts by applying Transformer to the Zhou et al. [25] model called STMC, and Camagoz et al.[28] suggest a method of applying Transformer to [4] instead of RNN. In addition, there is a study that proposed a BERT-based SLT method using a pretrained Transformer, unlike the existing aspect [29].

There is also a skeleton-based SLT study using a model of the pose estimation task, not CNN. Ko et al. [10] achieved keypoint-based SLT on large-scale videos using KETI data sets. In addition, Jang et al. [13] perform translation using the Graph Convolution Network (GCN) [30] to extract and weight important clips. Skeleton-based research continues not only in SLT but also in the task of CSLR [31, 32, 33].

Video processing is mainly performed in action recognition tasks and video recognition tasks, which greatly influence performance [34, 35, 36]. Gowda et al. [37] has proposed a score-based frame selection method called SMART in the task action recognition, achieving state-of-the-arts for the UCF-101 dataset[38], and Karpathy et al. [39] used a video processing method that treats all videos as fixed clips. Therefore, video processing is a very useful way to improve performance in a variety of tasks.

Video processing mainly utilizes a method of changing the number of frames, using a method of increasing the number of frames by converting the frames themselves [40, 41], or an augmented method of manipulating the order of frames [42]. Recently, video processing is performed using Deep Learning, and there is also a study that proposed a frame augmentation method using Generative Advertising Networks (GAN) [43] or using a genetic algorithm [44].

The frame processing method was used not only in the Action Recognition task but also in the SLT task. Park et al. [12] augmented the data in three ways: angle conversion of the camera, finger length conversion, and random keypoint removal. In addition, Ko et al. [10] used a method of increasing the number of data by adding randomness to the new skip sampling method.

We extract keypoint of the signer to obtain the feature value of the video. This method extracts skeletons points from the signer and minimizes the impact on the surrounding background, not the signer, in the video. We used a framework called Alphapose of Fang et al [20]. Alphapose can extract keypoints faster than the Cao, Zhe, et al OpenPose [19] model by first detecting signers in a top-down manner and then extracting keypoints from cropped images. We used pretrained model on a Halpe dataset [20]. There are 136 keypoints in total, and we used 123 keypoints by removing 13 keypoints to exclude the lower body. And 55 keypoints were used, excluding 68 detailed face keypoints, based on the better performances obtained when the keypoint of the face was removed in the study of Ko et al [10]. Therefore, the keypoint vector is composed of $V=(v_{1},v_{2},\ldots v_{55})$ and $v$ is the position coordinate $v\ _{i}\ =\left(v\ _{i}^{x}\ ,v\ _{i}^{y}\ \right)\ \ \ where\ \ \ i\ \in\ \left[1,55\right]$ of each keypoint. The number of the location of the keypoint follows in Figure 3.

In this paper, we propose customized normalization according to body parts considering keypoint positions. Since the body length varies depending on the person, a normalization method considering this is needed. Although the Robust Keypoint Normalization method was mentioned in Kim et al. [11], it cannot be said to be a normalization method considering all positions because the reference point was set in some parts, not all parts. Therefore, this paper proposes a normalization method that can further emphasize not only the location of each keypoint but also the movement of the hand.

We designated each reference point according to the part of the body the face, upper body, left arm, and right arm. The reference point is defined as $r=(r_{x},r_{y})$. The corresponding keypoint numbers according to each body part are shown in Table 1, and the names corresponding to each number can be referred to in Figure 3. In addition, a representative value for the entire body of the signer is defined as center point $c=(c_{x},c_{y})$ and follows (1).

$$\begin{gathered}c_{x}\ =\ \frac{1}{55}\sum_{i=1}^{55}{\ v_{i}^{x}},\ c_{y}\ =\ \frac{1}{55}\sum_{i=1}^{55}v_{i}^{y}\ \\ \end{gathered}$$

(1)

In this case, $v^{x}_{i}\ \ ,\ v^{y}_{i}$ are values of $V_{x}\ ,\ V_{y}$ vectors separated by $x$ and $y$ coordinates concerning the vector $V$. $V_{x}$ and $V_{y}$ are defined as follows:

$$\begin{gathered}V_{x}\ =\ \left(v_{1}^{x},v_{2}^{x},\ \cdots,\ v_{55}^{x}\right)\\ V_{y}\ =\ \left(v_{1}^{y},v_{2}^{y},\ \cdots,\ v_{55}^{y}\right)\end{gathered}$$

(2)

We make the elements of vectors $V_{x}$ and $V_{y}$ follows the (3).

$$\begin{gathered}S_{point}\ =\ \{\ Nose,\ Neck,\ LElbow,\ RElbow\ \}\\ d_{point}\ =\ \sqrt{\left(c_{x}\ -\ r_{x}\right)^{2}\ +\ \left(c_{y}-r_{y}\right)^{2}}\\ where\ \left(r_{x},r_{y}\right)\ \in S_{point}\\ \left(\ \widetilde{\ x}_{body},\ \widetilde{\ y}_{body}\ \right)\ =\ \left(\frac{v^{x}-c_{x}}{d_{point}},\ \frac{v^{y}-c_{y}}{d_{point}}\right)\end{gathered}$$

(3)

Equation (3) performs normalization based on the distance $d_{point}$ after obtaining the euclidean distance ${d_{point}}$ between vector $c$ and the reference point according to each part. Therefore, the keypoint for the body excluding both hands follows $\left(\widetilde{x}_{body},\ \widetilde{y}_{body}\right)$.

Next, we perform MinMax scaling to make the keypoints of both hands equal in scale and compare the movements of both hands equally. MinMax Scaling is to adjust the range to [0,1] by dividing the difference between the maximum and minimum values of coordinates by subtracting the corresponding coordinates and the minimum values. Here, to prevent a situation in which the minimum value becomes 0 and the vanishing gradient problem occurs, we subtracted -0.5 and adjusted it to the range of [-0.5,0.5]. This follows (4). Therefore, we define the keypoints of both hands that have undergone this process as $\widetilde{x}_{hand},\widetilde{y}_{hand}$. This follows (4):

$$\begin{gathered}\left(\widetilde{x}_{hand}\ ,\ \widetilde{y}_{hand}\right)\\ =\ (\frac{v^{x}\ -\ v_{min}^{x}}{v_{max}^{x}\ -\ v_{min}^{x}}\ -\ 0.5\ ,\frac{v^{y}\ -\ v_{min}^{y}}{v_{max}^{y}\ -\ v_{min}^{y}}\ -\ 0.5\ )\end{gathered}$$

(4)

where $v^{x}_{max}$ and $v^{x}_{min}$ are the maximum and minimum values of the hand keypoint, respectively. Thus, each $x$ and $y$ point to which our normalization method is applied is defined as vectors $V_{x}^{*}\ =\ \left[\widetilde{x}_{body};\widetilde{x}_{hand}\right]$ and $V_{y}^{\ast}\ =\ \left[\widetilde{y}_{body};\widetilde{y}_{hand}\right]$ and the final keypoint vector becomes $V^{*}\ =\ \left[V_{x}^{*};V_{y}^{*}\right]\in\mathbb{N}^{{55}\times 2}$.

To train through deep learning, the input size of the model must be fixed. In previous studies, for this purpose, the dimension was reduced and sized through embedding [4, 28]. However, this results in a loss of information in the video. Therefore, we use a method that reflects the information about the sign language video as it is without using embedding. To this end, this study uses a method of fixing the length of the input value by augmenting or sampling the frame.

In this paper, we propose a general-purpose method that fixes the length of the input value but applies to various datasets. In addition, we emphasize the hand-moving frame, which is the key frame of the sign language video, and propose a method of not losing it. Since the frames existing at the beginning and end of the video are those with less hand movement of most signers, we propose an augmentation methodology that emphasizes the middle frame of the image except for this. And, when the video length is very long, a sampling method that does not lose the key frame is applied. In this paper, combining these two methods, we propose an augmentation and sampling method that takes into account the difference in frames for each video. We call this method ”Stochastic Augmentation and Skip Sampling (SASS)”.

We define the $i$-th training video as $\textsc{x}_{i}\ =\ \{\ x_{t}\ \}_{t=0}^{T}$ and $L$ as the number of training videos. where, $i\in[1,L]$. We select N frames from between the frames of the training video $\textsc{x}_{i}$, which follows (5).

$$N\ =\ \frac{1}{L}\ \sum_{i=0}^{L}\textsc{x}_{i}$$

(5)

If the number of video frames $\textsc{x}_{i}$ is greater than $N$, the sampling is used, and if the number of video frame is less than $N$, the number of frames in each video is adjusted to $N$ using the augmentation. So, input value $X$ is $X\in\mathbb{R}^{L\times N\times|V^{*}|}$ This method can prevent the loss of video information, which is a disadvantage of frame sampling, and the disadvantage of frame augmentation, which can prevent a lot of memory consumption. The overall flow of our SASS method follows Figure 4.

We used a GRU-based model with a sequence to sequence (Seq2Seq) structure with an encoder and decoder in the translation step and improved translation performance by adding attention. This is a structure widely used in NMT and is suitable for translation tasks with variable-length inputs and outputs. We used Bahdanau attention [7]. The encoder-decoder structure goes through the following process when source sentence $\textsc{x}=(x_{1},x_{2},\ldots,x_{T_{x}})$ and target sentence $\textsc{y}=(y_{1},y_{2},\ldots,y_{T_{y}})$. Here, our model encodes a keypoint vector, not a sentence. First, the encoder calculates the hidden state and generates a context vector through it, and equation follows:

$$\begin{gathered}h_{t}=f(x_{t},h_{t-1})\\ c=q(h_{1},\ldots,h_{T_{x}})\end{gathered}$$

(11)

$h_{t}$ is the hidden state calculated in time step $t$ and is the context vector generated through the encoder and nonlinear function. Next, in decoder, the joint probability can be calculated in the process of predicting words. This follows (12).

$$p(\textsc{y})=\prod^{T_{y}}_{i=1}p(y_{i}|\{y_{1},y_{2},\ldots,y_{i-1}\},c)$$

(12)

The conditional probability at each time used in the (12) can be calculated through the following equation.

$$p(y_{i}|y_{1},y_{2},\ldots,y_{i-1},\textsc{x})=softmax(g(s_{i}))$$

(13)

And in (13), $s_{i}$ is the hidden state calculated from the time $i$ of the decoder and follows the following equation.

$$s_{i}=f(y_{i-1},s_{i-1},c)$$

(14)

The context vector $c_{i}$ is calculated using a weight for $h_{i}$ and follows (15).

$$c_{i}=\sum^{T_{j}}_{j=1}=a_{ij}h_{j}$$

(15)

where,

$$a_{ij}=\frac{exp(score(s_{i-1},h_{k}))}{\sum^{T_{x}}_{k=1}exp(score(s_{i-1},h_{k})}$$

(16)

Here, score is an alignment function that calculates the match score between the hidden state of the encoder and the decoder.

This experiment proves that our method is powerful compared to various normalization methods. The results of this are shown in Table 2. We fixed and experimented with all elements except normalization method. We compared our method with the method called ”Standard Normalization($X=(x-\mu)/\sigma$)” using the standard deviation($\sigma$) and average value($\mu$) of points, the method called ”Min-Max Normalization($X=(x-x_{max})/(x_{max}-x_{min})-0.5$)” using the minimum($x_{m}in$) and maximum values($x_{max}$) of points and the method called ”Robust Normalization($X=(x-x_{2/4})/(x_{3/4}-x_{1/4})$)” using the median($x_{2/4}$) and interquartile range (IQR). where, IQR is $x_{3/4}-x_{1/4}$. In this case, standard normalization is the method proposed in Ko et al.

In addition, we experimented by adding two normalization methods that did not exist before. First, ”all reference” is a normalization method in which (3) is applied to the hand. Second, ”center reference”, is a method using only each keypoint vector $v$ and center vector $c$ without a separate reference point. This divides the difference between v and c by the distance. Finally, the reference point was compared with the Kim et al[11] method of normalization by fixing it with the right shoulder.

Our method proved to be the most powerful method through the experiment. It can be seen that the performance has improved a lot compared to the methods of Kim et al and Ko et al, which are previous studies. In addition, it has proven that it is a new powerful method that can improve performance by performing better not only in the dataset (KETI) with a relatively large scale frame but also in PHOENIX with a relatively small image.

In this section, a difference in performance according to a frame selection method is experimented. Keypoints were normalized with our experimental normalization method, and everything else was fixed except the frame selection method. We demonstrate that our proposed method is excellent with a total of three comparative experiments. We conduct comparison experiments using only sampling, comparison experiments using only augmented, and comparison experiments using both sampling and augmentation methods.

First, we compare the experimental results when only sampling is performed. As the sampling method, the stochastic frame sampling and skip sampling methods proposed in this paper are compared. It is also compared with a random sampling technique that randomly selects a simple equal distribution without such additional processing.

In this case, the number of sampling was set to $N=50$ for the KETI data set and compared to the Ko et al method, and in the PHOENIX data set, the minimum frame length $N=16$ of the video was set. In the video sampling comparison experiment, the skip sampling technique showed the best results for the two datasets. In addition, probability sampling also differs significantly compared to random sampling following an even distribution.

Second, we experimented with the performance change when only frames were augmented. We set the maximum frame value of the training video in the dataset to $N$, and set the $N$ of KETI and PHOENIX to 426 and 475, respectively. We compared and experimented with a randomly augmented method following an even distribution and a stochastic augmented method following our method. This is shown in Table 3.

At KETI, the random augmentation was good in every way. However, in PHEONIX, the main evaluation index, BLEU-4, was the best in stochastic augmentation. The overall performance was improved when the frame augmentation was performed compared to when only sampling was performed.

Finally, we experimented with a combination of Sampling and Augmentation. We experiment with a total of 4 cases. In the previous experiment, since the random sampling method among the sampling methods has poor results, the experiment was conducted by combining the number of cases except random sampling. This is shown in Table 4. The SASS method, which combines our methods of skip sampling and stochastic augmentation, performed well in both KETI, and BLEU-4 was overwhelmingly the highest in PHOENIX.

In this section, we experiment on several factors affecting the model. We compare the performance according to the change in the probability set $Pr_{n}$ and compare according to $N$.

And experiments are conducted on how to invert the order of input values, a method of enhancing the translation performance proposed in the study of Sutskever et al [6]. For the first time, we experiment with the performance change according to $l_{p}$, the number of probability sets $Pr_{n}$.

We experiment only with $l_{p}\in[3,5,7,9,11,13,15,17]$ and evaluate only with BLEU-4. This is shown in Table 5. We achieve the best performance on both datasets when $l_{p}$ is 17. Even when compared to the average BLEU value, in the PHOENIX dataset, all of our methods are superior to other methodologies.

Next, an experiment was conducted to compare according to the set criteria of $N$. We set and compare $N$ according to the mean and median frame length of the training video. The median number of frames of training videos in these two datasets are 148 and 112 frames, respectively. The comparison accordingly is shown in Table 6.

Finally, we experiment with the effect of encoding by inverting the order of input $X$. This method takes into account that in a linguistic system, there is a higher association between the words located in the front, so that the match with the target sentence can be better if the input order of the source sentence is reversed. Therefore, we reverse the order of frames and encode each video by changing the order of frames to $(x_{T-1},x_{T-2},...,x_{0})$. This is shown in 7. This result shows that setting $N$ as the average has the best effect, and also that entering the original order of the frame has the best effect. It is analyzed that the change in the order of the input sentences is meaningless because our input value is the keypoint value of the frame, not the sentence.

In this section, we prove the excellence of our model by comparing existing studies with our model. KETI dataset was implemented directly and conducted a comparative experiment because we randomly divided it into the training, validation, and test sets. In Ko et al’s study, additional data was used, and we reproduced and compared the proposed model without adding data. This is shown in Table 8.

In addition, since we are a robust method of experimenting with a gloss-less dataset, PHOENIX dataset only compares video to text translation, not gloss-based methods. This is shown in Table 9.

Our model also performs well compared to previous studies in the dataset. In particular, it can be seen that KETI-dataset exhibits the best performance among keypoint-based SLT models. In addition, the BLEU score was the highest in the PHOENIX dataset. BLEU-4 increased significantly (about 4%p) compared to Sign2Text (Bahdanadu Attention), indicating that the ideal performance was improved through the Our video processing method even though the same Bahadanadu Attention was used.