Image-free Domain Generalization via CLIP for 3D Hand Pose Estimation

Hand pose estimation is the task of estimating the x, y and z coordinates of $21$ hand joints from either depth maps [22, 38] or RGB images [4, 26]. The depth map is invariant to the lighting conditions and shadows. It also has the advantage of being strong against clutter [2, 38]; while it has the disadvantage of not being able to capture various features such as textures and colors of the scene. The RGB image has the advantage of being able to capture detailed hand attributes such as unique colors, textures, and outlines. Besides, RGB cameras are more ubiquitous than depth sensors in our daily life, as we are holding RGB cameras in our smart phones; while we are not holding depth sensors in our pocket. However, unlike the depth map, the RGB image completely loses the 3D depth information. Therefore, compared to the depth map, there is much difficulty in achieving 3D hand pose estimation from RGB images [5, 26]. The useful yet challenging setting of RGB-based 3D hand pose estimation recently accelerates the development of many algorithms [40, 35, 64, 32]. More recently, RGB-based 3D hand mesh estimation has been established as well [26, 4]. The 3D hand model (ie. MANO) [4, 5, 34] has been exploited when constructing these pipelines. The graph convolutional network (GCN) is further exploited to better capture the relationship between vertices in the mesh topology [61, 18].

In the aspect of data, hands require data collection with much diverse camera perspectives, poses and shapes compared to other domains such as body pose estimation [9, 10, 6, 3, 48]. This is due to the fact that hands exhibit severe self-occlusions and diverse camera perspectives.

Contrastive learning is a way to achieving the self-supervised learning. It has been utilized in various works [28, 42, 12, 27, 51] for extracting the view-invariant representation from multiple views by exploiting the different views of the same content as positive samples and other content as negative samples. Chen et al. [12] proposed the contrastive learning for the data augmentation to learn better representation. They used data-augmented samples as positive; while other data samples as negative. The performance of these approaches could be improved by increasing the number of negative samples. However, the number of negative samples is limited by the GPU memory size. So, [13, 27] proposed a method to increase the number of negative samples with the momentum encoder. Zhu et al. [62] proposed a method for controlling the margin term according to the number of negative samples. Caron et al. [8] addressed the limitation by clustering the augmented data instead of comparing features. Contrastive learning has been also used in various tasks such as image and video classification [12, 27, 51] and object detection [28].

Recently, Spurr et al. [50] proposed the contrastive learning framework for the 3D hand pose estimation task. They extended the SimCLR [12] framework to be applicable to structured regression tasks such as the hand pose estimation. They used the geometric and appearance transformed hand images as positive and other images as negative samples for achieving the contrastive learning.

Most contrastive learning methods have been dependent on the data augmentation techniques such as scale manipulation, cut out, noise and rotation transformation. In our paper, we use the CLIP model, which is robust to the zero-shot learning and often used for the domain generalization tasks. We augment the data space in terms of style by manipulating the text prompts in CLIP and perform the contrastive learning between features of original and augmented data to make the feature of original data robust to domain generalization.

One of the important problems in computer vision is the dataset bias and domain shift between different datasets. To resolve this problem, domain adaptation (DA) /domain generalization (DG) approaches have recently attracted a lot of attention.

Domain adaptation (DA) is a methodology used in situations where the label of the target domain does not exist or is insufficient. DA utilizes source domain data, sparsely labeled, unlabeled target domain data as the training dataset to effectively learn the target domain distribution. DA is largely divided into three types: supervised domain adaptation (SDA) [21, 29, 37, 47], semi-supervised domain adaptation (SSDA) [46, 31, 33] and unsupervised domain adaptation (UDA) [57, 56, 59, 66]. SDA is a method that uses both labeled source and target domain data, SSDA is a method that exploits source domain data whose label is completely annotated and target domain data whose label is partially annotated. Finally, UDA is a method that uses source domain and target domain data without any labels.

Above mentioned methods (ie. SDA, SSDA and UDA) all require the target domain data to train the model. However, it is non-trivial to obtain the target data and also collecting the target data with labels is even harder. Domain generalization (DG) [19, 49, 58, 60] has emerged to relieve the challenge from collecting target data. DG differs from DA in that it makes features generalized in the target domain by using the source and additional unseen domain data which might be similar to the target domain data. Recently, Zhang et al. [60] proposed the DG methodology of the pose estimation using unconstrained dataset and could improve the generalization ability. However, it is limited by the fact that it requires to collect additional unseen domain data which might be similar to the target domain data. In our paper, we propose the domain generalization method which does not require additional dataset collection.

We constitute our baseline hand pose estimator $f^{\text{HPE}}$ samely as that of [63] which consists of 1) a 2D heatmap net $f^{\text{H2D}}$ that estimates the 2D heatmap $\mathbf{h}\in H$ from input RGB image $\mathbf{x}\in X$, and (2) a Poseprior net $f^{\text{PP}}$ that uses the estimated heatmap $\mathbf{h}\in H$ to predict 3D joint coordinates $\mathbf{c}\in C$. In the remainder of this subsection, we will explain details for two sub-networks (ie. $f^{\text{H2D}}$, $f^{\text{PP}}$).

The CLIP network $f^{\text{CLIP}}$ receives a 256$\times$256$\times$3-sized single RGB image $\mathbf{x}\in X$ and the text prompt $\mathbf{t}\in T$. The text encoder $f^{\text{CT}}$ and image encoder $f^{\text{CI}}$ of the CLIP model $f^{\text{CLIP}}$ generates a $512$-dimensional encoder features $f^{\text{CI}}(\mathbf{x})\subset\mathbb{R}^{512\times 1}$ and $f^{\text{CT}}(\mathbf{t})\subset\mathbb{R}^{512\times 1}$, respectively. In the original CLIP, ResNet-50 or ViT (VisionTransformer) is used as the baseline structure for the image encoder; also Transformer based on the byte pair encoding is used for the text encoder. In this paper, we used the ViT-B/32 as the image encoder and the Transformer network as the text encoder, which are the pre-trained models provided by OpenAI ¹¹1https://openai.com/ (We freeze the weights of CLIP model during the training).

Our CLIP-augmentation method is described in this section. We actually have two stages for this: 1) CLIP feature generation and 2) encoding vector generation stages.

The generated text prompt $\mathbf{t}$ is used as the input to the CLIP text encoder $f^{\text{CT}}$ to extract the feature vector $f^{\text{CT}}(\mathbf{t})$. To mix up the information, we defined $f^{\text{CLIP}}(\mathbf{x},\mathbf{t})$ as the weight-summed vector of $f^{\text{CI}}(\mathbf{x})$ and $f^{\text{CT}}(\mathbf{t})$. We chose the ratio between image encoder and text encoder as 6:4 and 9:1 for STB and RHD, respectively via the $10$-fold cross-validation. Ablation experiments on the ratio for weight-summation are shown in Table 4(a). Via this process, we enforce the CLIP feature $f^{\text{CLIP}}(\mathbf{x},\mathbf{t})$ to contain the extra information in addition to the source domain information.

Afterwards, the encoding vectors $\mathbf{e}^{\prime}_{1}$ and $\mathbf{e}^{\prime}_{2}$ are further exploited for the contrastive learning mechanism using the Eq. 4. The entire pipeline is trained in the end-to-end manner through 2D heatmap loss, 3D pose loss and contrastive loss. Via the contrastive loss, our pipeline becomes robust to images from various domains.

Our domain generalization framework is composed of end-to-end trainable networks based on 1) input image $\mathbf{x}$ from the source domain and 2) custom-created text prompt $\mathbf{t}$. We used only the source dataset (ie. FreiHAND [65]) for training and did not involve additional images or labels from the target domain or other unconstrained datasets. We trained the overall framework using three losses (ie. $L^{\text{Heat}}$, $L^{\text{Pose}}$ and $L^{\text{Con}}$) as follows:

$\displaystyle L(f^{\text{HPE}})$

$\displaystyle=$

$\displaystyle\lambda_{1}L^{\text{Heat}}+\lambda_{2}L^{\text{Pose}}+\lambda_{3}L^{\text{Con}}$

(1)

where $\lambda_{1}$, $\lambda_{2}$ and $\lambda_{3}$ are balance parameter controlling the weight of each loss function. From the $10$-random fold cross validation, we set $\lambda_{1}$ and $\lambda_{2}$ as $1$ and set $\lambda_{3}$ as $0.1$. Also, we used the Adam optimizer with $\beta=(0.9,0.999)$ and a learning rate of $10^{-4}$. In the remainder of this section, we will explain about three losses.

Only the hand pose estimation net $f^{\text{HPE}}$ disregarding the ‘Branch2’ of 2D heatmap net $f^{\text{H2D}}$ is exploited during the testing stage. The test RGB image $\mathbf{x}$ is input to the 2D heatmap net $f^{\text{H2D}}$ and it takes the ‘Branch1’ route (see Fig. 2) to generate 21 $32\times 32$-dimensional heatmap $\mathbf{h}$. Then the poseprior net $f^{\text{PP}}$ is further applied to map it towards the $21\times 3$-dimensional 3D joint coordinates $\mathbf{c}$.

We conduct our experiments using three types of RGB-based 3D hand pose benchmark datasets which have RGB images and corresponding ground-truth 3D pose annotations.

This paper deals with the domain generalization (DG) problem in the 3D hand pose estimation task. We mainly compared our method with the existing DG method: Zhang et al. [60] that solves the same problem with ours and achieves the best performance before us. We involved their results [60] for involving only ‘source domain images’. In Table 3, we compared ours with existing methods and confirm that our method showed the superior performance based only on the source images, without involving target images (ie. STB, RHD datasets) or additional unconstrained images. Compared to Zhang et al. [60], via our method, error rates are decreased by 3.33% and 3.11% in STB and RHD datasets, repectively.

Figure 3 shows the visualization of T-SNE distribution for the CLIP image and text encoder features (ie. $0.5\times(f^{\text{CI}}(\mathbf{x})+f^{\text{CT}}(\mathbf{t}))$) obtained from samples in the source dataset (FreiHAND) and the CLIP image encoder feature (ie. $f^{\text{CI}}(\mathbf{x})$) obtained from samples in the existing datasets (ie. STB, RHD and FreiHAND). The ‘aug’ denotes the samples obtained from the CLIP image and text encoder features (ie. $0.5\times(f^{\text{CI}}(\mathbf{x})+f^{\text{CT}}(\mathbf{t}))$) for the source dataset (FreiHAND); while ‘stb’, ‘rhd’ and ‘frei’ denote samples obtained from the CLIP image encoder feature (ie. $f^{\text{CI}}(\mathbf{x})$) for STB, RHD and FreiHAND datasets, respectively. Figure 3 shows that the distribution of augmented features (ie. ‘aug’ samples) cover the distribution of STB and RHD datasets, which are target domain datasets. Through this experiment, we qualitatively visualize the effect of our proposed method.

The validity of the augmented distribution is proven in Figure 4 which shows that our method maintains the context information of source domain and style of text prompt. In Figure 4, the image of the source domain dataset (FreiHAND) and text associated with the image are given to the CLIP image encoder (IE) $f^{\text{CI}}$ and CLIP text encoder (TE) $f^{\text{CT}}$, respectively and transformed to output features. And then, the two output features (ie. image feature and text feature) are merged by the weight summation (a). The rest of the images in the source domain dataset, except for the input image are given to the image encoder to extract image features (b). After extracting features, the cosine similarities between (a) and (b) are calculated. In the right side of the Fig. 4, we visualized text prompts with source domain images ranked by their similarity. The samples having the highest cosine similarity might be most relevant samples to the combined input image $\mathbf{x}$ and text prompt $\mathbf{t}$. From the visualization, we could observe that such an assumption is valid.

We assess the contribution of the CLIP model applied to hand pose estimation in our method: We configure four variants in our framework: 1) We prove the effectiveness of the CLIP model on the domain generalization problem through comparisons with and without the CLIP network, and study the optimal ratio between $f^{\text{CI}}(\mathbf{x})$ and $f^{\text{CT}}(\mathbf{t})$ when generating the combined features $f^{\text{CLIP}}(\mathbf{x},\mathbf{t})$ from them in Table 4(a). 2) For the optimal combination of hand pose estimator features and CLIP features, we study which operation is effective to combine CLIP features to the original features between ‘concatenation’ and ‘summation’. We also obtained the results for the case when we only used the CLIP text encoder for applying the style augmentation in Table 4(b). 3) We experimented to find the optimal weight ratio (ie. $\lambda_{3}$) between contrastive loss and other loss functions in Table 5. 4) Finally, we compared the performance of our method and the method applied with normal data augmentation in Table 6. In Table 4(a), we compared the performance of several variants: baseline, our method using only CLIP image encoder (IE) and our method using CLIP image encoder + CLIP text encoder (IE+TE). The error rate is reduced by $3.08\%$ in STB even when only CLIP image encoder is further involved. However, there is an significant improvement in both STB and RHD when both CLIP image encoder and CLIP text encoder are involved. In Table 4(a), the optimal weight ratio between $f^{\text{CI}}(\textbf{x})$ and $f^{\text{CT}}(\textbf{t})$ is seemingly $0.6$ and $0.9$ in STB and RHD, respectively. We obtained the values from the $10$-fold cross-validation. In Table 4(b), we study the effectiveness of IE+TE method on the style augmentation. From our results, we can observe that the sole CLIP text encoder (TE) is not enough to achieve the domain generalization to other domains since it does not have the context information from the source domain. On the other hand, ‘IE+TE’ can maintain context information of source domain and augmented style information via both CLIP image encoder and CLIP text encoder, thereby achieving the complete domain generalization. We also study the effect of two operations when combining hand pose estimator features and CLIP features: concatenation and summation. Also, the results showed that the concatenation is more effective operation than the summation to incorporate CLIP features into the hand pose estimation features. In Table 5, we showed results for the optimal weight between the contrastive loss and other loss functions (ie. $\lambda_{3}$). The best performance is obtained when $\lambda_{3}=0.001$ and $\lambda_{3}=0.1$ for STB and RHD, respectively in the table; while we set $\lambda_{3}=0.1$ for both datasets as the value is obtained from $10$-fold cross-validation. Thus, we report $\lambda_{3}=0.1$ results both for STB and RHD datasets. Finally, we compare the normal data augmentation method with ours in Table 6. We showed that our augmentation method consistently and significantly performs better than the normal data augmentation method. For the normal data augmentation, we involved six types of augmentation: color jitter, cut out, gaussian noise, sobel filter, color drop, and gaussian blur.