Learning Semantic Person Image Generation by Region-Adaptive Normalization

The task of human pose transfer has achieved great development during these years, especially with the unprecedented success of deep learning. Originally, these methods regard this task as a conditional image generation by taking the source appearance image as a condition to guide the generation of the target image, which mainly comes from the conditional generative adversarial networks (cGANs) [24]. Ma et al. [22] first present a two-stage model, which directly concatenated the target pose and the source image as input to generate the target image in a coarse-to-fine manner. Pumarola et al. [27] suggest an unsupervised manner by taking the concatenation of the generated results and the source pose as input to reconstruct the source image. Similarly, Tang et al. [36] propose a cycle-in-cycle way to constrain the learning of pose and appearance translation. However, directly concatenating the target pose and source image usually brings limited improvements due to the inconsistent poses. To solve the above issues that hinder the appearance translation, Essner et al. [5] adopt the combination of VAE [17] and U-Net [13] to disentangle the appearance and the pose of person images. Men et al. [23] also adopt the disentanglement by extracting the region styles to perform on the whole target pose. Siarohin et al. [31] proposed a Deformable GAN, which decomposes the overall deformation by a set of local affine transformations to deal with the misalignments caused by different poses. Subsequently, in order to enhance the spatial rearrangement ability, there are also some works  [18, 19, 29] that use flow-based deformation to align the source appearance. Both [18] and [19] adopt additional 3D human models to calculate the flow fields between the source and target image, while Ren et al. [29] obtain the global flow fields in an unsupervised manner and further propose a local neural texture render. Recently, Zhu et al. [47] propose to progressively transform the source image by a sequence of pose-attentional transfer blocks (PATBs), which is flexible but useful information may be lost during multiple transfers. Based on these observations of this work, Tang et al. [35] further propose a XingGAN model, which consists of shape-guided appearance-based generation branch, appearance-guided shape-based generation branch as well as co-attention fusion module to effectively transfer and update person shape and appearance features in a crossing and progressive manner. However, nearly all these methods directly utilize the global appearance features on the target one, and seldom explore the benefits of per-region appearance translation that may bring to the final reconstruction.

On the other hand, most of these aforementioned methods use human keypoints as pose representation due to its cheapness. However, it ignores the body skeleton that is useful to build the human body. There are also some other works that use DensePose[25] or semantic parsing maps [4, 33] as pose representation, which can provide more information about body depth or part segmentation. However, they directly take the semantic maps as a condition that is concatenated to the pose image along the channel dimension. Therefore, they can achieve only limited improvements on the target results especially the semantic appearance of each region due to the misalignment. Here, we also use the target semantic parsing map to assist the image generation but utilize it through the region adaptive normalization on each body region to separately guide the semantic appearance generation.

Semantic image generation aims at synthesizing photo-realistic images from the given semantic layout. Several methods [13, 38, 26, 20, 46, 37] have been suggested for solving this task. SPADE [26] adopts semantic label maps to predict the spatially-varying affine transformation parameters for incorporating the class prior to the target image, which controls the image generation process more precisely and obtains visual fidelity results. upSimilarly, CC-FPSE [20] proposes to predict the spatially-varying conditional convolutional kernels based on the input semantic layout. LGGAN [37] takes global and local contexts into account and gets promising performance. The image-level global generator learns a global appearance distribution and the class-specific local generator generates different object classes separately. Furthermore, SEAN [46] improves SPADE by introducing per-region style encoding, which is better suited to encode, transfer, and synthesize style in terms of visual quality. Inspired by the benefit brought by SEAN [46], in this work, we handle the person image generation by separate per-region style translation. To be specific, we adopt the region adaptive normalization on each semantic region to retain and transfer the style code between each corresponding semantic part of the source and target images.

Instead of directly predicting the target person image $\hat{I}_{p_{t}}$, the prediction of the target parsing map tends to be much easier due to the lack of texture translation. However, it still suffers from the large pose changes between the source and the target parsing maps. Inspired by the PATN [47], which proposed a progressive manner to handle the pose image transfer, here we adopt a semantic pose-attentional transfer network (SPATN) to model the semantic translation. The framework is shown in the left part of Fig. 1 (a). It takes the source pose $p_{s}$, target pose $p_{t}$ and source semantic map $S_{p_{s}}$ as input to predict the target semantic map $\hat{S}_{p_{t}}$, which can be formulated as:

where $\mathcal{F}_{SN}$ and $\Theta_{SN}$ denote the proposed SPATN model and its learnable parameters, respectively.

We observe that using sparse keypoint to represent the pose can only provide limited body structure information and it ignores the correspondence of each part, which usually fails to deal with some complex poses (\egcrossed arms). In order to better model the pose structure and further benefit the generation of target semantic parsing, we introduce the distance map as a complementary representation of pose structure. Originally, the pose $p$ is constructed by 18-channels heat maps that each one encodes one joint of a human body. With these 18 points, we generate 12 lines {$L_{m}|_{m=1}^{12}$} to represent the body skeleton. In this work, each skeleton generates one channel distance map. When this skeleton is invisible or occluded, the distance map is set to 0. Thus we can generate a 12-channels distance map $\{M_{m}|_{m=1}^{12}\}$, which has the same width and height as the source image. The values in $(x,y)$ of each $M_{m}$ is calculated by the smallest distance between the point $(x,y)$ and the skeleton $L_{m}$. The m-th distance map can be obtained by:

where $(x^{\prime},y^{\prime})$ denotes the point on the skeleton $L_{m}$. Here, we further normalize these values by introducing a negative parameter $\kappa$. The final distance map is then defined as:

where $m\in\{1,2,3...12\}$ represents the m-th skeleton. In this way, the closer the point to the skeleton, the larger the value is. Thus it can well model the body structure. In this work, $\kappa$ is set to $-0.1$. Finally, the poses $p_{s}$ and $p_{t}$ can be represented by 30-channels features, respectively, which consist of 18-channels joint heat maps and the extra 12-channels distance maps. upThe illustration and analyses of the generated distance map $M$ can be found in our suppl.

Fig. 1 (b) illustrates the framework of our proposed semantic person generation network (SPGNet). It takes the target pose $p_{t}$, source image $I_{p_{s}}$, as well as target semantic parsing map $\hat{S}_{p_{t}}$ (predicted through SPATN in the first stage) as input to generate a photo-realistic target image $\hat{I}_{p_{t}}$, which has the same appearance with $I_{p_{s}}$ and the same pose with $p_{t}$. The whole model can be formulated as:

where $\Theta_{SPG}$ denotes the SPGNet model parameters.

Following [31, 44, 18], we adopt a dual-path UNet [30] to encode the appearance and pose features separately. The pose features are extracted by the suggested PEBlocks, while the appearance features are obtained through the AEBlocks (see Fig. 1 (b)). The PEBlock and AEBlock have the same architectures, except the channel numbers in the first block (30 for PEBlock and 3 for AEBlock). Each block is constructed by two residual blocks [9]. Inspired by [18], here we adopt a feature deformation operation that is performed on the appearance features $f_{a}$ to solve the inconsistent poses. With the warped appearance feature $f_{a}^{w}$ and target pose feature $f_{p}$, we further propose a semantic person generation block (SPGBlock) by the region-adaptive normalization layers. The final result is generated by several stacks of UpBlock and SPGBlock in multi-scale feature spaces.

With the pose feature $f_{p}$, the warped appearance feature $f_{a}^{w}$, the decoder feature $f_{t}$, the predicted target semantic map $\hat{S}_{p_{t}}$ and the target style map $\mathcal{C}$, we suggest a semantic person generation block (SPGBlock) by incorporating the semantic region normalization to generate the target person feature. As is shown in Fig. 1 (b), each block contains two SEAN operations. The first one takes the $f_{p}$ and $f_{a}^{w}$ as input to generate the preliminary target feature. Then the second one takes the former output and the decoder feature $f_{t}$ to generate the residual features of the $f_{t}$. The output of the SPGBlock is expected to fuse the appearance to the pose features on each semantic region, respectively. The details of SEAN are shown in Fig. 2.

Denote by the input feature as $h$, the output of SEAN in the position $(n,c,y,x)$ is given by :

where $\mu_{c}$ and $\sigma_{c}$ are the mean and standard deviation of $h$ in the channel $c$. $\alpha$ and $\beta$ are the weighted sum of these features from the convolution output of the target semantic map $\hat{S}_{p_{t}}$ and the target style map $\mathcal{C}$, which are defined as:

where $\theta_{\alpha}$ and $\theta_{\beta}$ are the learnable parameters. It should be noted that the target style map $\mathcal{C}$ is shared in all the SPGBlock and the parameters in the style map generation are optimized by the gradient from the final objective $\mathcal{L}_{full}$.

Finally, after several stacks of the UpBlock and the SPGBlock on different feature scales, we can generate the target results $\hat{I}_{p_{t}}$ in a coarse-to-fine manner.

To train the proposed SPATN network, which is suggested in the first stage to predict the target semantic map $\hat{S}_{p_{t}}$, we adopt a cross-entropy loss to constrain the predicted $\hat{S}_{p_{t}}$ close to its ground-truth label $S_{p_{t}}$, which is defined as:

where $i$, $j$ and $c$ denote the positions of each semantic maps.

For the second stage, our proposed SPGNet is trained to generate a photo-realistic person image with the target pose $p_{t}$. The predicted $\hat{I}_{p_{t}}$ is expected to approximate the ground-truth image $I_{p_{t}}$ in both pixel and perceptual levels, and more importantly, should follow the real person image distribution. Thus, the learning objective to train the SPGNet contains two parts, \ie, reconstruction loss and photo-realistic loss.

We compare our proposed model with several recent state of the art methods, including Def-GAN [31], Intr-Flow [18], PATN [47], GFLA [29], XingGAN [35], and BiGraphGAN [34]. The quantitative result on DeepFashion is shown in Table 1. We can observe that our proposed method achieves comparable performance on this dataset, which means that results of ours own high visual quality and are realistic. In addition, we also conduct the evaluation on the challenging dataset Market-1501. The comparison is shown in the left part of Table 2. Mask-SSIM (M-SSIM) and Mask-LPIPS (M-LPIPS) are also considered to exclude the effect of irrelevant regions, \ie, background. We can see that our method can also have a superior performance to these competing methods, indicating the great generalization of our method. The superior performance can be attributed to the proposed two-stage model in which the region adaptive normalization is utilized for per-region style translation.

Figure 3 gives the qualitative comparison on both DeepFashion and Market-1501 datasets. We can have the following observations, (i) though the source image $I_{p_{s}}$ is under an extreme pose (1-st and 3-rd rows in the left part), results of our method are more realistic, and especially, have the consistent shape with the ground-truth $I_{p_{t}}$, indicating the benefits of the SPATN that bring to the generation process. (ii) In terms of complex texture (4$\sim$6-th rows in the left part), ours are obviously superior to the competing methods in retaining the appearance textures, which is mainly attributed to the utilization of per-region style translation. (3) Within the complex background, the results of ours on the Market-1501 (right part of Figure 3) still perform favorably in generating the clear and photo-realistic results as well as a consistent appearance with the source image.

Although the above evaluation indicators can evaluate the performance of the generated results from different aspects, the person image generation task is likely to be user-oriented. Therefore, we conduct a user study on the two test sets to evaluate the performance from real human perception. To be specific, it mainly contains two groups that are conducted from different aspects. (i) Comparison with ground-truth. Following [22, 31], we randomly select 55 real and 55 generated images from the test set and then shuffle them. The first 10 of the 110 images are used for practice and the remaining 100 images are used to assess the performance. 30 volunteers that cover the bachelor and master students with computer vision background are required to give a choice about whether this one is real or generated within a second. (ii) Comparison with the state of the arts. To achieve this goal, we randomly select 55 image pairs, including source image, target pose, ground-truth and images generated by these competing methods. Volunteers are required to select the image that is the closest to the ground-truth in terms of the visual quality, and more importantly, the pose and appearance consistency with ground-truth. Results are shown in the right part of Table 2. Here we adopt three metrics, \ie, R2G: the percentage of the real image that is judged as the generated one, G2R: the percentage of the generated image that is judged as real one, Jab: the percentage that the image is judged as the best one. Higher values in these three metrics indicate better performance. We can see that results of ours outperform the state of the arts with a large margin (\eg, 9.40% and 8.06% higher than the 2-nd best one in Jab), which further indicates the superiority of our method.

upIn order to explore the effectiveness of our per-region style translation in the human generation task, in this paper, we follow [18] and conduct the ablation study on the DeepFashion datasets with the following variants, (i) Baseline: our baseline model is a dual-path U-Net, which mainly comes from Intr-Flow[18]. There are two encoders, which encode appearance and pose, respectively. The decoder combines the target pose and the warped appearance features to generate the final image. It should be noted that there is not any conditional normalization in this model. (ii) Baseline-S: based on the Baseline model, the pose encoder of Baseline-S takes the additional features (\ie, the target semantic parsing map that is predicted by our SPATN model) as input. (iii) Ours-SPADE: by replacing the per-region adaptive instance normalization (SEAN) with the spatial adaptive normalization (SPADE) [26]. (iv) Ours-CC [20]: by replacing SEAN with CC-FPSE in the decoder module, in which ours-CC takes the target semantic parsing map as input and predicts the conditional convolution kernels to guide image generation. (v) PATN-S: by taking the semantic parsing maps as additional input data to retrain PATN. (vi) SEAN: by using the SEAN generator [46] to directly broadcast the style information to the target semantic layout to generate an image. (vii) Ours-Full. The quantitative results are shown in Table 3. We can see that (1) compared with Baseline and Baseline-S, PATN and PATN-S, though slight improvements can be obtained, Ours-Full obviously outperforms them with a large margin, especially the FID metric (\eg, Baseline-S is 13.1% higher than Baseline), indicating the benefits that the per-region adaptive normalization brings to this task. (2) Compared with the Baseline model, Ours-SPADE and Ours-CC achieve an obvious improvement by incorporating the spatial adaptive normalization and the conditional convolution, respectively, but they are still inferior to Ours-Full. (3) Directly taking SEAN to handle this task, the poor performance may be caused by the inconsistent poses, indicating the necessity of feature deformation in Ours-Full. (4) With the per-region adaptive normalization, Ours-Full performs superior to others, indicating the effectiveness of our SPGNet in human person generation. The visual comparison is shown in Fig. 9. We can see that compared with other variants, Ours-Full obviously outperforms others in generating photo-realistic and consistent appearance results, which further indicates the benefits of our SPGNet by adopting the per-region style translation in handling this task.

In order to explore the training manners of the two-stage model that may bring benefits to the final results, in this supplemental material, we conduct three schemes, (i) $S_{1}\triangleright S_{2}$: pre-training the SPATN model in stage one, then fixing its parameters and training SPGNet in stage two; (ii) $S_{1}\!\Join\!S_{2}$: pre-training the SPATN model in stage one, then jointly training with the SPGNet in stage two; (iii) $S_{1}\!\parallel\!S_{2}$: training the SPATN and SPGNet simultaneously, in which the SPGNet takes the ground-truth target parsing map ${S}_{p_{t}}$ as input. During the inference, the model takes the predicted target parsing map $\hat{S}_{p_{t}}$ from the SPATN model as input to generate the final results. The comparison results are shown in Table 4. We find that though the joint training manner ($S_{1}\!\Join\!S_{2}$) usually obtains great performance in many fields, it shows inferior results in this task. We analyze that 1) due to the long pathway of SPGNet, the gradient from the $\mathcal{L}_{full}$ to update the SPATN model tends to be zero, making the SPATN model seldomly benefits from the end-to-end training, 2) more importantly, there is a gap between the predicted semantic parsing map and the ground-truth. The inaccurate prediction may confuse the learning of the SPGNet, thus has a negative impact on the generation process. Therefore, we adopt the $S_{1}\!\parallel\!S_{2}$ scheme to train our two-stage model and use the predicted target semantic maps in the first stage to guide the final generation of the target image in the inference.

Our SPGNet consists of a pose encoder, an appearance encoder and a decoder which is composed of several SPGBlocks. Table 5 shows the details of SPGNet. Table 6 and Table 7 give the detailed network architecture of SPGBlock and Feature Deformation Module, respectively. Conv. ($d,k,s$) and ConvT.($d,k,s$) denote convolution and transposed convolution layer, where $d$, $k$ and $s$ are output dimention, convolution kernel size and stride, respectively. And $dim({f_{t-1}})$ is the dimention of feature map $f_{t-1}$. LReLU is leaky ReLU with negative slope $c$. BN and IN represent batch normalization and instance normalization, respectively. The denotation of input is the same as that in the paper. $I_{p_{s}}$ denotes the source appearance image. $S_{p_{s}}$ and $\hat{S}_{p_{t}}$ are source semantic parsing map and predicted target semantic parsing map. $p$ is keypoint heat map of the target pose. $\Phi_{2D}$ and $V$ denote the projection of the predicted 3D flow and visibility map from Intr-Flow [18], respectively.

It may be not enough to use only keypoints skeleton as pose representations to generate semantic maps, especially when the poses are complex or rare in the dataset. The introduction of distance maps makes pose representation more robust to various poses. We generate 12 lines {$L_{m}|_{m=1}^{12}$} distance map with 18-channels keypoint heat maps to represent the body skeleton. Each skeleton generates one channel distance map, thus we can generate a 12-channels distance map $\{M_{m}|_{m=1}^{12}\}$, which has the same width and height of the source image. The values in $(x,y)$ of each $M_{m}$ is calculated by the smallest distance between the point $(x,y)$ and the skeleton $L_{m}$. The m-th distance map can be obtained by:

where $(x^{\prime},y^{\prime})$ denotes the point on the skeleton $L_{m}$. Here, we further normalize these values by introducing a negative parameter $\kappa$. The final distance map is then defined as:

where $m\in\{1,2,3...12\}$ represents the m-th skeleton. In this way, the closer the point to the skeleton, the larger the value is. Thus it can well model the body structure. The process of distance map generation is shown in Fig. 5. The experimental results on DeepFashion show that without using the distance map, the mIOU of the predicted semantic maps is 0.520, while when using the distance map, the mIOU is 0.539. By using the distance map, our method can generate plausible semantic parsing maps even though the poses are complex or rare (\eg, crossed hand in Figure 6), which will further benefit the later target image generation.

In this section, we show more visual comparison with the competing methods (\iePATN [47], Intr-Flow [18], GFLA [29], XingGAN [35], ADGAN [23]) on DeepFashion and Market-1501 in Figs. 7 and 8, respectively. It can be seen that our method can generate more semantic, consistent, and photo-realistic results. Besides, we also show more visual comparison of different SPGNet variants in Fig. 9.