RepParser: End-to-End Multiple Human Parsing with Representative Parts

To date, methods of multiple human parsing are based on the two-stage pipeline. Most of them can be divided into two categories: 1) bottom-up paradigm and 2) top-down paradigm. As mentioned above, the bottom-up methods (Gong et al. 2018, 2019; Li et al. 2018; Zhao et al. 2018) regard multiple human parsing as a segment-then-grouping pipeline. The series of the bottom-up methods usually generate redundant human parsing results, leading to high computational costs during post-processing. Compared with bottom-up series, top-down approaches (Ji et al. 2020; Yang et al. 2019; He et al. 2017; Yang et al. 2020; He et al. 2021; Liu et al. 2021) focus on the single-person human parsing problem as they employ person detector to solve the issue of person separation. Furthermore, recent works have developed two versions of top-down framework: the unified top-down model (Yang et al. 2019; He et al. 2017; Yang et al. 2020; Qin et al. 2019) and the separated top-down model (Ji et al. 2020; Ruan et al. 2019; Liu et al. 2019). The difference between the two versions is whether unifying the person detector into a single-person parsing model. For example, Mask-RCNN (He et al. 2017) can be regarded as the first unified top-down approach, where it adopts Faster R-CNN (Ren et al. 2017) to predict a bounding box for each person and extracts region-of-person from detector’s features for performing instance-specific part segmentation. Following this idea, Parsing R-CNN (Yang et al. 2019) and RP R-CNN (Yang et al. 2020) devote to solving the problem of the single-person human parsing and propose new variants of Mask R-CNN via contextual modeling or part re-scoring.

Different from two-stage methods, our work is devoted to designing a novel single-stage pipeline and focuses on instance-aware body part segmentation with representative parts.

Traditional solutions try to build an instance-specific model for instance-level recognition. For example, Tian et al. (Tian, Shen, and Chen 2020) adopt conditional convolutions for one-stage instance segmentation, where each convolution kernel is dynamically generated from a center point of person instance. This design improves the instance segmentation performance while maintaining high efficiency. Moreover, Li et al. (Yanwei Li and Jia 2021) propose a location-aware kernel generation for panoptic scene understanding. Mao et al. (Mao et al. 2021) propose to dynamically generate a keypoint-aware estimator for multi-person pose estimation. Although these different approaches vary in tasks, they all share a characteristic: they focus on the instance-specific convolution kernel generation. However, each convolution kernel generated through existing methods encodes sparse content of an instance (i.e., object center only). Therefore, the generated kernels severely ignore the person-part context which is essential for accurate human parsing, thus leading to suboptimal results as demonstrated in the experimental results.

As a supplement to them, we extend instance-specific modeling to solve the multiple human parsing. Instead of directly deriving from single-stage frameworks that are used in other instance recognition tasks, we propose to parse multiple human instances through representative parts, and encode person-part context into each instance-aware convolution kernels as well as part representation. As a result, the proposed method significantly outperforms variants of single-stage methods applied in other instance recognition tasks.

As discussed before, bounding boxes and object centers are often used to represent person instances in two-stage methods and single-stage methods. Due to the rectangular shape, a bounding box has a rough global context of a person but cannot account for the semantically important local areas. Instead, the object center only accounts for small local areas, thus ignoring the interrelation among body parts and the global context of an instance. To overcome these limitations, our core idea is that each person instance in an image is represented by representative parts. It is expected that the representative parts can encode the characteristics of each person instance and only focus on the pixels of corresponding body parts. Motivated by this, we propose to dynamically construct representative parts of an instance through keypoints of body parts, as they can reflect the global context of a person (e.g., posture or shape) and semantically salient part areas. Formally, let $\mathcal{R}=\{(x_{k},y_{k})\}_{k=1}^{C}$ denotes representative parts of a person, where $(x_{k},y_{k})$ is the keypoint of the $k$-th part (e.g., face, left-arm, right-arm, and so on), $C$ is the number of part categories (e.g., $C$=20 for CIHP dataset). Thus, we parse person instances conditioning on their representative parts, as they not only present characteristics of pose and shape but also reflect person-part relations.

To construct representations of representative parts, we need to localize them by object centers. As shown in Fig. 2, we firstly adopt a feature pyramid network (Lin et al. 2017) to produce multi-scale feature maps from levels 3 to 7. Following FCOS (Tian et al. 2019), we treat each location on the feature maps as a potential instance. Thus, for each location $(x_{h},y_{h})$ on the feature maps, we estimate the confidential score being a person center and the offsets to representative parts. Based on this, representative parts $\mathcal{R}$ is calculated by Eq. 1.

$$\mathcal{R}=\{(x_{h}+\Delta x_{k},y_{h}+\Delta y_{k})\}_{k=1}^{C},$$

(1)

where $\{(\Delta x_{k},\Delta y_{k})\}_{k=1}^{C}$ are the normalized offsets from the center of a person instance to the center of the representative parts. After that, we construct an initial representation of representative parts by sampling pixel points from the image-level feature $F$. Formally, we denote $f_{h}$ as the feature of sampled instance point and $\{f_{h}^{k}\}_{k=1}^{C}$ as the features of representative parts. Next, we employ these sampled representative parts for instance-aware kernel generation and part-aware human parsing.

To obtain high-quality instance representations for accurately human parsing, it is expected that the instance-aware convolution kernels are dynamically generated by relying on the characteristics of instances. To achieve this, we propose to generate instance-aware kernels by representative parts, since they encode potential contexts about person-part relations. Instead of directly applying initial representative parts to predict instance-aware kernels, we first re-adjust the representation of the representative parts according to the person-part relations, aiming to dynamically encode the person-part context into the corresponding kernel. Specifically, the re-adjusted representative parts are obtained through Eq. 2:

$$\begin{array}[]{lll}\alpha&=\{\sigma(W_{a}[f_{h}\oplus f_{h}^{k}])\}_{k=1}^{C},&\\ f_{p}&=W_{m}[(\alpha_{1}\cdot f_{h}^{1})\oplus...\oplus(\alpha_{C}\cdot f_{h}^{C})],&\\ \end{array}$$

(2)

where $W_{a}$ and $W_{m}$ are learnable parameters that are respectively responsible for relation estimation and feature re-adjustment. $\sigma(\cdot)$ is the standard sigmoid function and $\oplus$ means a concatenation operation. The $\alpha$ is the estimated relation matrix, where each element in $\alpha$ denotes a part being relevant to a person with a confidential score. The $f_{p}$ denotes the representation of representative parts, which is dynamically generated via person-part interaction. Note that the estimated relational score $\alpha$ is continually updated by directly supervised training so that it becomes more accurate from time to time.

Given re-adjusted representative parts $f_{p}$ and instance representation $f_{h}$, we generate two types of convolution kernels through Eq. 3.

$$\begin{array}[]{lll}W_{f}&=V_{1}(f_{h}\oplus f_{p}),&\\ W_{o}&=V_{2}(f_{h}\oplus f_{p}),&\\ \end{array}$$

(3)

where $V_{1}$ and $V_{2}$ are linear matrices for kernel generation. $W_{f}$ is used to project an image feature to an instance-aware feature without resolution reduction, while the $W_{o}$ is responsible for predicting part masks from many part-aware features. Notably, the generated kernels are very compact, as they are designed in a $1\times 1$ convolution layer with less channels (e.g., 32 for $W_{f}$ and C for $W_{o}$). We project the image-level feature $F$ to the instance-aware feature for human parsing by Eq. 4.

$$\begin{array}[]{lll}F_{p}&=W_{f}\circledast F,&\\ \end{array}$$

(4)

where $\circledast$ is the convolution operation. Compared with top-down methods, such as Parsing-RCNN consisting of eight $3\times 3$ convolution layers with 256 channels for instance feature extraction, the generated kernels are much more lightweight.

After handling the issue of person separation by representative parts, we would like to predict an accurate mask for each part. Instead of predicting masks from the instance-aware feature, we first build multiple branches for separate parsing. In each branch, we construct a part-aware representation that can be used to only fire on the pixels of the corresponding part. Specifically, we first divide the instance-aware feature $F_{p}$ into $C$ groups, each of which is responsible for part-specific segmentation. For each group, we construct a geometry map that records relative distances from all pixels to the corresponding representative part, suggesting the salient area of the corresponding part. Formally, we denote all geometry maps as $F_{s}$ and compute the part-aware representation by Eq. 5.

$$\begin{array}[]{lll}\hat{F_{p}}&=\{W_{p}^{k}[F_{p}^{k}\oplus F_{s}^{k}]\}_{k=1}^{C},&\\ \end{array}$$

(5)

where $W_{p}$ is the learnable transformation matrix. Next, we utilized dynamically generated kernel $W_{o}$ to predict mask for each part, which is formalized by Eq. 6.

$$\begin{array}[]{lll}M&=\{\phi(W_{o}^{k}\circledast\hat{F_{p}^{k}})\}_{k=1}^{C},&\\ \end{array}$$

(6)

where $M$ is the predicted parsing maps and $\phi(\cdot)$ is the standard softmax function. Although it is possible to predict masks by instance-aware feature, we empirically find that using part-aware representation performs better.

In this section, we compare proposed RepParser with state-of-the-art multiple human parsing methods and report evaluation results summarized from two datasets. In addition to existing two-stage methods that are either based on top-down paradigm or bottom-up paradigm, we also compare RepParser with other representative single-stage methods that are applied for other instance recognition tasks, including DETR (Carion et al. 2020), Deformable DETR (Zhu et al. 2021) and condInst (Tian, Shen, and Chen 2020). Moreover, we re-implement these single-stage methods and train them under the same settings for a fair comparison, since these methods are not evaluated for multiple human parsing in original papers. In addition to standard metrics, we also measure inference time per image for each method on the same hardware if possible. Furthermore, we also report the relative proportion of time costs reduction for investigation of the efficiency of each single-stage model.

As for the comparison with top-down methods, RepParser achieves competitive parsing performance with fewer time costs. For example, RepParser with ResNet-50 significantly outperforms the Mask-RCNN under the same settings, e.g., 35.9% mIoU vs 33.4% mIoU, 39.4% ${\rm{AP}}^{p}_{vol}$ vs 33.9% ${\rm{AP}}^{p}_{vol}$ and 36.8% ${\rm{PCP}}_{50}$ vs 26.8% ${\rm{PCP}}_{50}$. On two stronger methods using M-CE2P and RP-RCNN, RepParser shows comparable parsing results under the same setting, but has much lower time costs (e.g., 208ms vs 1107ms, 193ms vs 394ms). It is worth noting that many non-unified top-down methods such as SemaTree (Ji et al. 2020) adopt an isolated object detector to detect persons and then crop the RoIs on the original images, thus leading to high inference costs (i.e., from the input image to the parsing results). As illustrated in Fig. 3, the inference time of top-down methods, such as Mask RCNN, Parsing RCNN, and PR-RCNN, dramatically increases as the number of persons linearly increases. However, the RepParser keeps almost constant inference time, since it eliminates the prior detection and each generated kernel is very compact (i.e., $32\times 32\times 1\times 1$). This suggests that the RepParser could be applied to many complex real-world scenes, such as crowded scene, while keeping stable yet high efficiency. Moreover, another major merit of the RepParser is that it does not rely on bounding boxes. As illustrated in Fig. 4(left), top-down methods only perform human parsing inside the predicted bounding boxes. As a result, some body parts cannot be parsed if the detector yield inaccurate bounding boxes. However, RepParser can well handle the body parts even outside the box (see Fig. 4(right)).

In terms of the comparison with single-stage parsing methods, the RepParser achieves higher parsing performance than that of other single-stage methods, while maintaining competitive efficiency. This indicates that directly deriving single-stage methods from other instance tasks leads to suboptimal results since they severely ignore the instance-part contexts that are essential for accurate multiple human parsing.

Many works (Sun et al. 2019; Wang et al. 2021b) have demonstrated that higher resolution representation brings better performance for dense prediction tasks. Inspired by this, we investigate which level of the image feature is beneficial for human parsing. Hence, we separately apply generated instance-aware kernels on three different feature maps, which separately are 1/16, 1/8, and 1/4 smaller than the size of the image. Table 5 indicates that the performance drops dramatically if the resolution of the input feature map is downsampled to 1/16 of the input image. We conjecture the possible reason behind this is that the human parsing task requires pixel-level understanding and high-resolution feature maps preserve more visual contents. However, larger resolution leads to a high computation burden. Besides, generating parsing results from the image feature at the 1/4 scale of the image size brings minor gains, when compared it with the counterpart at 1/8 scale of the image size. Thus, we choose 1/8 as the default setting for a better trade-off between accuracy and speed.

In general, a good initialization of models leads to better performance. Thus we explore the impact of initialization methods on multiple human parsing. As shown in Tab 6, human parsing methods pre-trained on the COCO keypoint dataset can improve ${\rm{AP}}_{vol}^{p}$ by 1%. It indicates that a good initialization will lead to better parsing results.