Self-Constrained Inference Optimization on Structural Groups for Human Pose Estimation

Human pose estimation, as a keypoint detection task, aims to detect the locations of body keypoints from the input image. Specifically, let $I$ be the image of size $W\times H\times 3$. Our task is to locate $K$ keypoints $X=\{X_{1},X_{2},...,X_{K}\}$ from $I$ precisely. Heatmap-based methods transform this problem to estimate $K$ heatmaps $\{H_{1},H_{2},...,H_{K}\}$ of size $W’\times H’$. Given a heatmap, the keypoint location can be determined using different grouping or peak finding methods [21, 25]. For example, the pixel with the highest heatmap value can be designated as the location of the corresponding keypoint. Meanwhile, given a keypoint at location $(p_{x},p_{y})$, the corresponding heatmap can be generated using the Gaussian kernel

In this work, the ground-truth heatmaps are denoted by ${\bar{H}_{1},\bar{H}_{2},...,\bar{H}_{K}}$.

Fig. 3 shows the overall framework of our proposed SCIO method for pose estimation. We first partition the detected human body keypoints into 6 structural groups, which correspond to different body parts, including lower and upper limbs, as well as two groups for the head part, as illustrated in Fig. 4. Each group contains four keypoints. We observe that these structural groups of four keypoints are the basic units for human pose and body motion. They are constrained by the biological structure of the human body. There are significant freedom and variations between structural groups. For example, the left arm and the right arm could move and pose in totally different ways. In the meantime, within each group, the set of keypoints are constraining each other with strong structural correlation between them.

As discussed in Section 1, we further partition each of these 6 structural groups into base keypoints and terminal keypoints. The base keypoints are near the body torso while the terminal keypoints are at the end or tip locations of the corresponding body part. Fig. 2 shows that the terminal keypoints are having much lower estimation confidence scores than those base keypoints during pose estimation. In this work, we denote these 4 keypoints within each group by

where $X_{D}$ is the terminal keypoint and the rest three $\{X_{A},X_{B},X_{C}\}$ are the base keypoints near the torso. The corresponding heatmap are denoted by $\mathbf{H}=\{H_{A},H_{B},H_{C}\ |\ H_{D}\}$. To characterize the structural correlation within each structural group $\mathbf{H}$, we propose to develop a self-constrained prediction-verification network. As illustrated in Fig. 3, the prediction network $\mathbf{\Phi}$ predicts the heatmap of the terminal keypoint $H_{D}$ from the base keypoints $\{H_{A},H_{B},H_{C}\}$ with feature map $\mathbf{f}$ as the visual context:

We observe that the feature map $\mathbf{f}$ provides important visual context for keypoint estimation. The verification network $\mathbf{\Gamma}$ shares the same structure as the prediction network. It performs the backward prediction of keypoint $H_{A}$ from the rest three:

Coupling the prediction and verification network together by passing the prediction output $\hat{H}_{D}$ of the prediction network into the verification network as input, we have the following prediction loop

This leads to the following self-constraint loss

This prediction-verification network with a forward-backward prediction loop learns the internal structural correlation between the base keypoints and the terminal keypoint. The learning process is guided by the self-constraint loss. If the internal structural correlation is successfully learned, then the self-constraint loss $\mathcal{L}_{A}^{s}$ generated by the forward and backward prediction loop should be small. This step is referred to as self-constrained learning.

Once successfully learned, the verification network $\mathbf{\Gamma}$ can be used to verify if the prediction $\hat{X}_{D}$ is accurate or not. In this case, the self-constraint loss is used as an objective function to optimize the prediction $\hat{X}_{D}$ based on local search, which can be formulated as

where $\mathbb{H}(\hat{X}_{D})$ represents the heatmap generated from keypoint $\hat{X}_{D}$ using the Gaussian kernel. This provides an effective mechanism for us to iteratively refine the prediction result based on the specific statistics of the test sample. This adaptive prediction and optimization is not available in traditional network prediction which is purely forward without any feedback or adaptation. This feedback-based adaptive prediction will result in better generalization capability on the test sample. This step is referred to as self-constrained optimization. In the following sections, we present more details about the proposed self-constrained learning (SCL) and self-constrained optimization (SCO) methods.

In this section, we explain the self-constrained learning in more details. As illustrated in Fig. 3, the input to the prediction and verification networks, namely, $\{H_{A},H_{B},H_{C}\}$ and $\{H_{B},H_{C},H_{D}\}$, are all heatmaps generated by the baseline pose estimation network. In this work, we use the HRNet [27] as our baseline, on top of which our proposed SCIO method is implemented. We observe that the visual context surrounding the keypoint location provides important visual cues for refining the locations of the keypoints. For example, the correct location of the knee keypoint should be at the center of the knee image region. Motivated by this, we also pass the feature map $\mathbf{f}$ generated by the backbone network to the prediction and verification network as inputs.

In our proposed scheme of self-constrained learning, the prediction and verification networks are jointly trained. Specifically, as illustrated in Fig. 3, the top branch shows the training process of the prediction network. Its input includes heatmaps $\{H_{A},H_{B},H_{C}\}$ and the visual feature map $\mathbf{f}$. The output of the prediction network is the predicted heatmap for keypoint $X_{D}$, denoted by $\hat{H}_{D}$. During the training stage, this prediction is compared to its ground-truth $\bar{H}_{D}$ and form the prediction loss $\mathcal{L}_{P}^{O}$ which is given by

The predicted heatmap $\hat{H}_{D}$, combined with the heatmaps $H_{B}$ and $H_{C}$ and the visual feature map $\mathbf{f}$, is passed to the verification network $\mathbf{\Gamma}$ as input. The output of $\mathbf{\Gamma}$ will be the predicted heatmap for keypoint $X_{A}$, denoted by $\hat{H}_{A}$. We then compare it with the ground-truth heatmap $\bar{H}_{A}$ and define the following self-constraint loss for the prediction network

These two losses are combined as $\mathcal{L}_{P}=\mathcal{L}_{P}^{O}+\mathcal{L}_{P}^{S}$ to train the prediction network $\mathbf{\Phi}$.

Similarly, for the verification network, the inputs are heatmaps $\{H_{B},H_{C},H_{D}\}$ and visual feature map $\mathbf{f}$. It predicts the heatmap $\hat{H}_{A}$ for keypoint $X_{A}$. It is then, combined with $\{H_{B},H_{C}\}$ and $\mathbf{f}$ to form the input to the prediction network $\mathbf{\Phi}$ which predicts the heatmap $\hat{H}_{D}$. Therefore, the overall loss function for the verification network is given by

The prediction and verification network are jointly trained in an iterative manner. Specifically, during the training epochs for the prediction network, the verification network is fixed and used to compute the self-constraint loss for the prediction network. Similarly, during the training epochs for the verification network, the prediction network is fixed and used to compute the self-constraint loss for the verification network.

As discussed in Section 1, one of the major challenges in pose estimation is to improve the accuracy of hard keypoints, for example, those terminal keypoints. In existing approaches for network prediction, the inference process is purely forward. The knowledge learned from the training set is directly applied to the test set. There is no effective mechanism to verify if the prediction result is accurate or not since the ground-truth is not available. This forward inference process often suffers from generalization problems since there is no feedback process to adjust the prediction results based on the actual test samples.

The proposed self-constrained inference optimization aims to address the above issue. The verification network $\mathbf{\Gamma}$, once successfully learned, can be used as a feedback module to evaluate the accuracy of the prediction result. This is achieved by mapping the prediction result $\hat{H}_{D}$ for the low-confidence keypoint back to the high-confidence keypoint $\hat{H}_{A}$. Using the self-constraint loss as an objective function, we can perform local search or refinement of the prediction result $\hat{X}_{D}$ to minimize the objective function, as formulated in (8). Here, the basic idea is that: if the prediction $\hat{X}_{D}$ becomes accurate during local search, then, using it as the input, the verification network should be able to accurately predict the high-confidence keypoint $\hat{H}_{A}$, which implies that the self-constraint loss $||{H}_{A}-\hat{H}_{A}||_{2}$ on the high-confidence keypoint ${X}_{A}$ should be small.

Motivated by this, we propose to perform local search and refinement of the low-confidence keypoint. Specifically, we add a small perturbation $\Delta_{D}$ onto the predicted result $\hat{X}_{D}$ and search its small neighborhood to minimize the self-constraint loss:

Here, $\delta$ controls the search range and direction of the keypoint, and the direction will be dynamically adjusted with the loss. $\mathbb{H}(\hat{X}_{D}+\Delta_{D})$ represents the heatmap generated from the keypoint location $\hat{X}_{D}+\Delta_{D}$ using the Gaussian kernel. In the Supplemental Material section, we provide further discussion on the extra computational complexity of the proposed SCIO method.

The comparison and ablation experiments are performed on MS COCO dataset [17] and CrowdPose [15] dataset, both of which contain very challenging scenes for pose estimation.

MS COCO Dataset: The COCO dataset contains challenging images with multi-person poses of various body scales and occlusion patterns in unconstrained environments. It contains 64K images and 270K persons labeled with 17 keypoints. We train our models on train2017 with 57K images including 150K persons and conduct ablation studies on val2017. We test our models on test-dev for performance comparisons with the state-of-the-art methods. In evaluation, we use the metric of Object Keypoint Similarity (OKS) score to evaluate the performance.

CrowdPose Dataset: The CrowdPose dataset contains 20K images and 80K persons labeled with 14 keypoints. Note that, for this dataset, we partition the keypoints into 4 groups, instead of 6 groups as in the COCO dataset. CrowdPose has more crowded scenes. For training, we use the train set which has 10K images and 35.4K persons. For evaluation, we use the validation set which has 2K images and 8K persons, and the test set which has 8K images and 29K persons.

For fair comparisons, we use HRNet and ResNet as our backbone and follow the same training configuration as [32] and [27] for ResNet and HRNet, respectively. For the prediction and verification networks, we choose the FCN network [19]. The networks are trained with the Adam optimizer. We choose a batch size of 36 and an initial learning rate of 0.001. The whole model is trained for 210 epochs. During inference, we set the number of search steps to be 50.

Following existing papers [27], we use the standard Object Keypoint Similarity (OKS) metric which is defined as:

Here $d_{i}$ is the Euclidean distance between the detected keypoint and the corresponding ground truth, $v_{i}$ is the visibility flag of the ground truth, $s$ is the object scale, and $k_{i}$ is a per-keypoint constant that controls falloff. $\delta(*)$ means if * holds, $\delta(*)$ equals to 1, otherwise, $\delta(*)$ equals to 0. We report standard average precision and recall scores: $AP^{50}$, $AP^{75}$, $AP$, $AP^{M}$, $AP^{L}$, $AR$, $AP^{easy}$, $AP^{med}$, $AP^{hard}$ at various OKS [8, 27].

We compare our SCIO method with other top-performing methods on the COCO test-dev and CrowdPose datasets. Table 1 shows the performance comparisons with state-of-the-art methods on the MS COCO dataset. It should be noted that the best performance is reported here for each method. We can see that our SCIO method outperforms the current best by a large margin, up to 2.5%, which is quite significant. Table 2 shows the results on challenging CrowdPose. In the literature, there are only few methods have reported results on this challenging dataset. Compared to the current best method MIPNet [14], our SCIO method has improved the pose estimation accuracy by up to 1.5%, which is quite significantly.

In Table 3, we compare our SCIO with state-of-the-art methods using different backbone networks, including R152, HR32, and HR48 backbone networks. We can see that our SCIO method consistently outperforms existing methods. Table 4 shows the performance comparison on pose estimation with different input image size, for example 128$\times$96 instead of 384$\times$288. We have only found two methods that reported results on small input images. We can see that our SCIO method also outperforms these two methods on small input images.

To systematically evaluate our method and study the contribution of each algorithm component, we use the HRNet-W48 backbone to perform a number of ablation experiments on the COCO val2017 dataset. Our algorithm has two major new components, the Self-Constrained Learning (SCL) and the Self-Constrained optimization (SCO). In the first row of Table 5, we report the baseline (HRNet-W48) results. The second row shows the results with the SCL. The third row shows results with the SCL and SCO of the prediction results. We can clearly see that each algorithm component is contributing significantly to the overall performance. In Table 6, We also use normalization and sigmoid functions to evaluate the loss of terminal keypoints, and the results show that the confidence of each keypoint from HRNet has been greatly improved after using SCIO.

Fig. 5 shows three examples of how the estimation keypoints have been refined by the self-constrained inference optimization method. The top row shows the original estimation of the keypoints. The bottom row shows the refined estimation of the keypoints. Besides each result image, we show the enlarged image of those keypoints whose estimation errors are large in the original method. However, using our self-constrained optimization method, these errors have been successfully corrected. Fig. 6 shows how the self-constraint loss decreases during the search process. We can see that the loss drops quickly and the keypoints have been refined to the correct locations. In the Supplemental Materials, we provide additional experiments and algorithm details for further understanding of the proposed SCIO method.