Body-Part Joint Detection and Association via Extended Object Representation

In our proposed BPJDet, we train a dense single-stage anchor-based detector to directly predict a set of objects $\{\widehat{\mathcal{O}}\!\in\!\widehat{\mathbf{O}}\!\parallel\!\widehat{\mathcal{O}}\!=\!cat(\widehat{\mathcal{O}}^{box},\widehat{\mathcal{O}}^{dis}),\widehat{\mathbf{O}}\!=\!\widehat{\mathbf{O}}^{b}\cup\widehat{\mathbf{O}}^{p}\}$, which contains human body set $\widehat{\mathbf{O}}^{b}$ and body-part set $\widehat{\mathbf{O}}^{p}$ concurrently. A typical prediction $\widehat{\mathcal{O}}$ is concatenated of the bounding box $\widehat{\mathcal{O}}^{box}$ and corresponding center point displacement $\widehat{\mathcal{O}}^{dis}$. It locates an object with a tight bounding box $\mathbf{\hat{b}}\!=\!(\hat{b}_{x},\hat{b}_{y},\hat{b}_{w},\hat{b}_{h})$ where coordinates $(\hat{b}_{x},\hat{b}_{y})$ are the center position, $\hat{b}_{w}$ and $\hat{b}_{h}$ are the width and height of $\mathbf{\hat{b}}$, respectively. It also records the relative displacement $\mathbf{\hat{d}}\!=\!(\hat{d}_{x_{i}},\hat{d}_{y_{i}})|^{k}_{i=1}$ of center point of body-part ($\widehat{\mathcal{O}}\!\in\!\widehat{\mathbf{O}}^{b}$) or affiliated body ($\widehat{\mathcal{O}}\!\in\!\widehat{\mathbf{O}}^{p}$) to each $\mathbf{\hat{b}}$. Considering that body-part may be more than one component (e.g., both hands part with $k=2$), we allow $\mathbf{\hat{d}}$ to be compatible with $k$-dimensional 2D offsets. For these regressed offsets, we will explain how to decode them for the body-part association with fusing detected body-part set $\widehat{\mathbf{O}}^{p}$ in Section 3.4.

Intuitively, we can benefit a lot from this extended representation. On one hand, an appropriate large body bounding box $\widehat{\mathcal{O}}^{box}$ possesses both strong local characteristics and weak global features (such as surrounding background and anatomical position) for its body part $\widehat{\mathcal{O}}^{dis}$ offset regression. This enables the network to learn their intrinsic relationships. On the other hand, compared to methods [17, 18, 9] training multiple subnetworks or stages, mixing $\widehat{\mathcal{O}}^{box}$ and $\widehat{\mathcal{O}}^{dis}$ up can be leveraged directly in BPJDet without the need of complicated post-processing. By designing a one-stage network that uses shared heads to jointly predict $\widehat{\mathcal{O}}^{box}$ and $\widehat{\mathcal{O}}^{dis}$, our approach can achieve high accuracy with minimal computational burden during training and inference.

Our network structure is shown in Fig. 2 left. We choose the recently most cost-effective one-stage YOLOv5 [21] as the basic backbone $\mathcal{N}$. Specifically, following YOLOv5, we feed $\mathcal{N}$ one RGB image $\mathbf{I}\!\in\!\mathbb{R}^{h\times w\times 3}$ as the input, keep its beneficial data augmentation strategies (e.g., Mosaic and MixUp), and output four grids $\widehat{\mathit{G}}\!=\!\{\widehat{\mathcal{G}}^{s}\!\parallel\!s\!\in\!\{8,16,32,64\}\}$ from four multi-scale heads. Each grid $\widehat{\mathcal{G}}\!\in\!\mathbb{R}^{A_{a}\times A_{o}\times\frac{h}{s}\times\frac{w}{s}}$ contains dense object outputs $\widehat{\mathbf{O}}$ produced from $A_{a}$ anchor channels and $A_{o}$ output channels. Supposing that one target object $\mathcal{O}$ is centered at $(b_{x},b_{y})$ in the feature map $\mathbf{F}^{s}$, the corresponding grid $\widehat{\mathcal{G}}^{s}$ at cell $(\frac{b_{x}}{s},\frac{b_{y}}{s})$ should be highly confident. When having defined $A_{a}$ anchor boxes $\mathcal{B}^{s}\!=\!\{(B^{w}_{i},B^{h}_{i})|^{A_{a}}_{i}\}$ for the grid $\widehat{\mathcal{G}}^{s}$, we will generate $A_{a}$ anchor channels at each cell $(\frac{b_{x}}{s},\frac{b_{y}}{s})$. Furthermore, to obtain robust capability, YOLOv5 allows detection redundancy of multiple objects and four surrounding cells matching for each cell. This redundancy makes sense to the detection of body or part position $\mathcal{O}^{box}$, but it is not completely facilitative to the regression of offsets $\mathcal{O}^{dis}$. We interpret this in Section 3.3.

Then, we explain the arrangement of one prediction $\widehat{\mathcal{O}}$ from $A_{o}$ output channels of $\widehat{\mathcal{G}}^{s}_{i,x,y}$ which is related to $i$-th anchor box at grid cell $(x,y)$. As shown in the example of grid cells in Fig. 2 right, one typical predicted $\widehat{\mathcal{O}}$ embedding consists of four parts: the objectness or probability $\hat{o}$ that an object exists, the candidate bounding box $\mathbf{\hat{b}}^{\prime}=({\hat{b}_{x}}^{\prime},{\hat{b}_{y}}^{\prime},{\hat{b}_{w}}^{\prime},{\hat{b}_{h}}^{\prime})$, the object classification score $\hat{c}_{k+1}$, and the candidate of body-part offsets $\mathbf{\hat{d}}^{\prime}\!=\!(\hat{d}^{\prime}_{x_{i}},\hat{d}^{\prime}_{y_{i}})|^{k}_{i=1}$. Thus, $A_{o}\!=\!3k+6$. To transform the candidate $\mathbf{\hat{b}}^{\prime}$ into coordinates $\mathbf{\hat{b}}$ relative to the grid cell $\widehat{\mathcal{G}}^{s}_{i,x,y}$, we apply conversions as below:

where $\phi$ is the sigmoid function that limits model predictions in the range $(0,1)$. Similarly, this detection strategy can be extended to the offset of body-part. A body-part’s intermediate offsets $\mathbf{\hat{d}}^{\prime}$ are predicted in the grid coordinates and relative to the grid cell origin $(x,y)$ using:

In the way, $\hat{d}_{x_{i}}$ and $\hat{d}_{y_{i}}$ are constrained to $\pm 2\frac{B^{w}_{i}}{s}$ and $\pm 2\frac{B^{h}_{i}}{s}$, respectively. To learn $\mathbf{\hat{b}}$ and $\mathbf{\hat{d}}$, losses are applied in the grid space. Sample targets of $\mathbf{b}$ and $\mathbf{d}$ are shown in Fig. 2 right.

For a set of predicted grids $\mathit{\widehat{G}}$, we firstly build target grids set $\mathit{G}$ following formats introduced in Section 3.2. Then, we mainly compute following four loss components:

For the bounding box regression loss $\mathcal{L}_{box}$, we adopt the complete intersection over union (CIoU) across four grids $\mathcal{G}^{s}$. $\mathsf{BCE}$ in the objectness loss $\mathcal{L}_{obj}$ and classification loss $\mathcal{L}_{cls}$ is the binary cross-entropy. The multiplier $o$ in $\mathcal{L}_{obj}$ is used for penalizing candidates without hitting target grid cells ($o=0$), and encouraging candidates around target anchor ground-truths ($o=1$). The $w_{s}$ is a balance weight for different grid level. Finally, we utilize the mean squared error (MSE) to measure offset outputs $\mathbf{\hat{d}^{\prime}}$ and normalized targets $\mathbf{d^{\prime}}$ in body-part displacement loss $\mathcal{L}_{bpd}$. Before that, we apply a filter $\varphi(\cdot)$ with the visibility label of body-part to remove out those false-positive offset predictions from $\widehat{\mathcal{O}}$. Finally, we calculate the total training loss $\mathcal{L}=(\mathit{\widehat{G}},\mathit{G})$ as follows:

where $N_{bs}$ is the batch size. The $\alpha$, $\beta$, $\gamma$ and $\lambda$ are weights of losses $\mathcal{L}_{box}$, $\mathcal{L}_{obj}$, $\mathcal{L}_{cls}$ and $\mathcal{L}_{bpd}$, respectively.

In inference, we need to process the predicted objects set $\widehat{\mathbf{O}}$ to get final results. First of all, we apply the conventional Non-Maximum Suppression (NMS) to filter out false-positive and redundant bounding boxes of both body and part objects:

where $\tau_{conf}$ and $\tau_{iou}$ are thresholds for object confidence and IoU overlap, respectively. We fetch confidence of each predicted object $\widehat{\mathcal{O}}$ by $\hat{o}\cdot\hat{c}_{i}$, where $i\!=\!1$ for body object and $i\!>\!1$ for part object. Then, we rescale $\widehat{\mathcal{O}}^{box^{\prime}}$ and $\widehat{\mathcal{O}}^{dis^{\prime}}$ in $\widehat{\mathbf{O}}^{b^{\prime}}$ and $\widehat{\mathbf{O}}^{p^{\prime}}$ to obtain real $\widetilde{\mathbf{O}}^{b}$ and $\widetilde{\mathbf{O}}^{p}$ by below transformations:

where $x_{o}$ and $y_{o}$ are offsets from grid cell centers. The $s$ maps the box and offset size back to the original image shape.

Finally, we update associated body parts of each left body object in $\widetilde{\mathbf{O}}^{b}$ by fusing its regressed center point offsets ($\widetilde{\mathcal{O}}^{dis}\!\in\!\widehat{\mathbf{O}}^{b^{\prime}}$) with the remaining candidate part objects $\widetilde{\mathbf{O}}^{p}$. Specifically, we search each box $\widetilde{\mathcal{O}}^{box}$ in part objects with its nearest body-part offset $\widetilde{\mathcal{O}}^{dis}$ belonging to body object. The part box that has a large inner IoU ($>\tau^{inner}_{iou}$) with the body box will be selected. The updated body objects set is denoted as $\widetilde{\mathbf{O}}^{b^{\prime}}$. We report all evaluation results on $\widetilde{\mathbf{O}}^{b^{\prime}}$ and $\widetilde{\mathbf{O}}^{p}$.

Datasets: We mainly expect to evaluate the association quality of BPJDet, while maintaining high object detection accuracy. Three public datasets including CityPersons [15], CrowdHuman [16] and BodyHands [9] are chosen. The former two are for pedestrian detection tasks. In CityPersons, it has 2,975 and 500 images for training and validation, respectively. In CrowdHuman, there are 15,000 images for training and 4,375 images for validation. Following BFJDet [18], we use its re-annotated box labels of visible faces for conducting corresponding experiments. The last dataset BodyHands is for hand-body associations tasks. It has 18,861 and 1,629 images in train-set and test-set with annotations for hand and body locations and correspondences. We implement body-hand joint detection task in it for comparing.

Evaluation Metric: For evaluation metrics, we report the standard VOC Average Precision (AP) metric with IoU=0.5 for object detection of body, face and hands. We also present the log-average miss rate (MR^-2) of body and its parts. For the association quality of BPJDet, we report the log-average miss matching rate (mMR^-2) as proposed by BFJDet [18] of body-face pairs, and present Conditional Accuracy and Joint AP defined by BodyHands [9] of body-hand pairs.

Implementation Details: We adopt the PyTorch 1.10 and 4 RTX-3090 GPUs for training. Depending on the dataset complexity and scale, we train on the CityPersons, CrowdHuman and BodyHands datasets for 100, 150 and 100 epochs using the SGD optimizer, respectively. Following the original YOLOv5 [21] architecture, we train three kinds of models including BFJDet-S/M/L by controlling the depth and width of bottlenecks in $\mathcal{N}$. The shape of input images is resized and zero-padded to $1536\times 1536\times 3$ following settings in JointDet [17] and BFJDet [18]. We adopt the data training augmentation but leave out test time augmentation (TTA). As for many hyperparameters, we keep most of them unchanged, including adaptive anchors boxes $\mathcal{B}^{s}$, the grid balance weight $w_{s}$, and the loss weights $\alpha\!=\!0.05$, $\beta\!=\!0.7$ and $\gamma\!=\!0.3$. We set $\lambda\!=\!0.015$ based on ablation studies. When testing, we use thresholds $\tau^{b}_{conf}\!=\!0.05$, $\tau^{b}_{iou}\!=\!0.6$, $\tau^{p}_{conf}\!=\!0.1$, $\tau^{p}_{iou}\!=\!0.3$ and $\tau^{inner}_{iou}\!=\!0.6$ for applying $\mathsf{NMS}$ on $\widehat{\mathbf{O}}$.

Ablation Studies: We mainly investigate the hyperparameter $\lambda$. For simplicity, we conduct all ablation experiments on the CityPersons dataset using BPJDet-S and training about joint body-face detection task. The input shape is $1280\times 1280\times 3$. Total epoch is expanded to 150 for searching the best result. We uniformly sample $\lambda$ from 0.005 to 0.030 with step 0.005 for training, and report metrics including MR^-2s and average mMR^-2 of each best model. As in Fig. 3, we can clearly find that the model performance is optimal when $\lambda\!=\!0.015$. A larger or smaller $\lambda$ will lead to inferior results. More ablation studies are provided in our supplementary material.

Results on CityPersons and CrowdHuman: We compare joint body-face detection performance of BPJDet with method BFJDet [18] on these two benchmarks. Table 1 shows results on the val-set of CityPersons. Following [2, 18], results are reported on four subsets of Reasonable (occlusion $<$ 35%), Partial (10% $<$ occlusion $\leqslant$ 35%), Bare (occlusion $\leqslant$ 10%) and Heavy (occlusion $>$ 35%). Comparing with RetinaNet+BFJ which obtains similar body AP but far inferior face AP to ours, BPJDet-L exceeds it of mMR^-2 by 13.1%, 13.8%, 13.0% and 16.9% in four subsets, respectively. Comparing with FPN+BFJ that has similar face AP but higher body AP to ours, BPJDet-L also surpasses it of mMR^-2 by 6.3%, 2.9%, 7.5% and 7.3% in four subsets. Especially, in the Heavy subset, our association superiority is the most prominent, which reveals that our BPJDet has an advantage for addressing miss-matchings in crowded scenes.

Table 2 shows results on the more challenging CityPersons. Our method BPJDet achieves considerable gains in all metrics comparing with BFJDet based on one-stage RetinaNet [12] or two-stage FPN [11] and CrowdDet [3]. Remarkably, BPJDet-L has achieved the lowest mMR^-2 value 50.1%, which is 2.2% lower than the previous best method CrowdDet+BFJ. These again demonstrate the superiority of our method. More persuasive visual results of BPJDet-L trained on CrowdHuman and tested on complicated overcrowded scenes are shown in Fig. 4.

Results on BodyHands: We conduct joint body-hand detection experiments on BodyHands [9], and compare our BPJDet with several methods proposed in BodyHands. As shown in Table 3, our BPJDet outperforms all other methods by a significant margin. With achieving highest hand AP 85.9% and conditional accuracy 86.91% of body, BPJDet-L largely improves the previous best Joint AP of body and hands by 20.52%. We give more qualitative comparisons of joint body-hand detection trained on BodyHands in Fig. 5. The compared masked images are fetched from BodyHands paper. Our BPJDet can detect many unlabeled objects, and associate hands that method proposed in BodyHands fails to match. This illustrates why we have an overwhelming Joint AP advantage. All these results further validate the generalizability of our extended object representation and its impressive strength on body and part relationship discovery.