Template-Free Try-on Image Synthesis via Semantic-guided Optimization

Existing virtual try-on approaches can be roughly categorized into 3D-based methods (e.g., 3D body shape) and 2D-based methods (e.g., clothing warping). We first introduce these two approaches and then compare them with TF-TIS.

Research on human pose transfer [17, 18, 19, 20, 21, 22, 23] has trended recently, as copious applications are planned in the future. The process of human pose transfer comprises two stages: pose estimation and image generation. The first stage can be divided into two categories (i.e., keypoints estimation [24, 25, 26] and human semantic parsing [27, 28, 29]). For example, Hidalgo et al.[26] trained a single-stage network through multi-task learning to determine the keypoints of the whole body simultaneously. Gong et al.[28] used the part grouping network (PGN) to reformulate instance-level human parsing as two twinned sub-tasks that can be jointly learned and mutually refined.

For the second stage, with the advances of GANs, image generation has received considerable attentions and has been widely adopted [18, 21, 23] to generate realistic images. Among the existing pose transfer research, most constructed novel architectures and successfully transferred the pose of the given human image based on the human joint points. For instance, Ma et al.[17] separated this task into two stages: pose integration, which generates initial but blurry images, and image refinement, which refines images by training a refinement network in an adversarial way. Siarohin et al.[20] used geometric affine transformation to mitigate the misalignment problem between different poses. However, most of the previous works did not extend the application of pose transfer to explore virtual try-on. By infusing pose transfer, virtual try-on services provide consumers with more chances to realize their appearance in trying on new clothes in multiple aspects and induce them to buy clothes. Hence, by converting semantic segmentation, TF-TIS seamlessly integrates virtual try-on with pose transfer to generate multi-view try-on images for customers.

The association between different fields has been studied and exploited recently [30, 31, 32, 33, 34, 35, 36, 37] (e.g., the cross-modal matching between audio and visual signals [30, 31, 32], image and text [33, 35, 36]). Castrejón et al. [38] used the identical network architecture with different weights as encoders to extract low-level features from different modalities (e.g., sketches and natural images) and then inputted them into the shared cross-modal representation network to learn the representation for scenes. Tae-Hyun et al. [39] attempted to learn voice-face correlation in a self-supervised manner (i.e., directly capturing the dominant facial traits of the person correlated with the input speech instead of synthesizing the face from the attributes). Inspired by the cross-modal learning, which can find hidden details from an inconspicuous part of data or align the embedding from one domain to another, we used a similar concept to learn the correlation between clothes and human poses to synthesize the image of the virtual try-on with a suitable pose for the user from the corresponding clothing.

A virtual try-on service usually requires three inputs [11, 9, 12]: 1) a user image, 2) an in-shop clothing image, and 3) a target pose. One potential improvement is to automatically generate the target pose according to the in-shop clothing because it reduces the users’ efforts. Moreover, a suitable pose better demonstrates the in-shop clothing, which may stimulate consumption. For example, plain T-shirts in a sideways pose can mostly show muscle lines. To synthesize the target pose directly from in-shop clothing, cloth2pose uses pairs of in-shop clothes and mannequin photos on the online shopping site for training. Specifically, cloth2pose first derives keypoints of mannequin photos by existing models, e.g., [24, 40, 41].²²2We use OpenPose model [24], a 2D pose estimation model pre-trained on large-scale human pose datasets (COCO [42] and MPII [43]) in our experiment. The following keypoints are used: nose, eyes, ears, neck, shoulders, elbows, wrists, hips, knees, and ankles. Let $x_{k}$ denote the 2D position of the $k^{th}$ keypoint on the image ($I_{t}$). Because it is difficult to regress the clothing features to a single point, we converted the keypoint position $x_{k}$ into the pose map $P_{k}$ by applying a 2D Gaussian distribution for each keypoint. The values at position $p\in{R}^{2}$ in $P_{k}$ are defined as follows:

where $\sigma$ determines the spread of the peak. After constructing the 2D keypoint map for each keypoint, we stack all the 2D keypoint maps together as a keypoint tensor, denoted as $P$.

Afterward, cloth2pose extracts features of the in-shop clothes by using the first 10 layers of VGG-19 [44], denoted as $\phi_{0}$. Let $C_{t}$ denote the image of in-shop clothing. The clothing feature map $F$ is obtained as $\phi_{0}(C_{t})$. Here, cloth2pose exploits a progressive refinement architecture as illustrated in Fig. 2. Specifically, at the first block, the network produces a set of keypoint information only from the clothing feature map: $P^{1}=\phi_{1}(F)$, where $\phi_{1}$ refers to the first convolutional block. For the succeeding convolutional blocks, we employed five convolutional layers with a $7\times 7$ kernel and two with a $1\times 1$ kernel to generate the keypoint tensor, and each layer is followed by a ReLU. The convolutional block takes the concatenation of $F$ and the prediction from the previous block as input to predict the refined keypoint tensor:

where $\phi_{i}$ represents the $i^{th}$ convolutional block and $N_{c2p}$ is the number of total convolutional blocks in cloth2pose.

An intuitive choice for the loss function is the $L_{2}$ distance between the keypoint tensors extracted from the pose estimation model ($P$) and that estimated from cloth2pose ($P^{N_{c2p}}$), i.e., $\lVert P^{N_{c2p}}-P\lVert^{2}_{2}$. However, only using the $L_{2}$ loss is likely to generate many responses in various locations for one joint. Given this condition, we employed the sparsity constraint to limit the number of candidates. Therefore, if the model predicts several candidates for one keypoint, the nonjoint area is penalized less by L2 loss than by L1 loss. The final loss is as follows:

where $\lambda=0.00008$ is the hyperparameter for striking a balance between multiple candidates and keypoint vanishing. If the value of $\lambda$ is too high (e.g., $\lambda=0.001$), then the output $P^{N_{c2p}}$ is without any candidates. Conversely, if the value is too low (e.g., $\lambda=0.00001$), then the sparsity constraint becomes ineffective, and the output still has more than one candidate.

Showing the corresponding area of each body part explicitly, the human body segmentation are employed to synthesize realistic human images. Accordingly, the goal of the pose-guided parsing translator is to translate the source body semantic segmentation $M_{s}$ to the target body semantic segmentation $M_{t}$ according to the target pose $P$. We first used the PGN[28], which is pretrained on the Crowd Instance-level Human Parsing dataset, to produce semantic parsing labels. The labels contain 20 categories, including left-hand, top clothes, and face. Afterward, to precisely map each item to the new position according to the pose $P$, we used one-hot encoding to constitute a 20-channel tensor $M\in{R}^{{20}\times{W}\times{H}}$, where each channel is a binary mask representing one category. Due to the unnecessity of the clothing channel of $M_{s}$, we replace it with the original in-shop clothing mask $M_{c}$. This replacement facilitates offering the in-shop clothing shape to realize the virtual try-on service.

Adapted from pix2pix[45], the pose-guided parsing translator consists of two downsampling layers, nine residual blocks, and two upsampling layers. Convolutional layers and highway connections, concatenating the input and the output of the corresponding block, are composed in each residual block. The objective of the translator $G_{t}$ adopts a CGAN as follows:

where $G_{t}$ minimizes the objective against $D_{t}$ that maximizes it (i.e., $\arg\min_{G_{t}}\max_{D_{t}}\mathcal{L}^{G_{t}}_{GAN}(G_{t},D_{t})$) and $M_{in}$ represents the concatenation of $M^{{}^{\prime}}_{s}$, $P$, and $M_{c}$.

To accurately differentiate each pixel as the corresponding channel, we integrate a pixel-wise binary cross-entropy loss of the $G_{t}$, denoted as $\mathcal{L}^{G_{t}}_{BCE}$, with our CGAN objective, and the discriminator stays the same:

where $n_{c}$ denotes the total number of channels of human parsing masks. In summary, the objective of the pose-guided parsing translator is derived as follows:

Having obtained the target semantic segmentation from the previous stage, the segmentation region coloring aims to synthesize a coarse try-on result by rendering information into the segmentation regions, denoted as $M_{g}=G_{t}(M_{in})$. Given the great success of applying GANs in various image generation tasks, we adopte the architecture of CGAN[46] to synthesize results. Specifically, we propose a coloring generator $G_{c}$ rendering the personal information into the body semantic segmentation $M_{g}$ according to $I_{s}$ and $C_{t}$ (i.e., the appearance of the source person and in-shop clothing texture). Because it is difficult to derive a significant number of training images, we traine our network to change the source person. To avoid supplying $G_{c}$ with the clothing information, we remove the clothing information from $I_{s}$. In other words, we take as input 1) the in-shop clothing $C_{t}\in{R}^{{3}\times{W}\times{H}}$, 2) the source person image without clothing information $I^{{}^{\prime}}_{s}\in{R}^{{3}\times{W}\times{H}}$, and 3) the target semantic segmentation $M_{g}\in{R}^{20\times{W}\times{H}}$ for $G_{c}$.

Fig. 2 illustrates the architecture of TF-TIS. We adopted the UNet architecture with highway connections, combining the input and processed information. Highway connections were employed to avoid the vanishing gradient [47]. Six residual blocks were implemented between the encoder and the decoder of $G_{c}$. For each residual block, two convolutional layers and ReLU were stacked to integrate $M_{g}$, $I^{{}^{\prime}}_{s}$, and $C_{t}$ from small local regions to broader regions so that the appearance information of $I^{{}^{\prime}}_{s}$ and $C_{t}$ can be extracted.

Because the background information is less important and easily distracts the generator from synthesizing try-on images, we filtered out it to force $G_{c}$ to concentrate on generating the correct human part of the image rather than the whole image. Specifically, the background information of the generation result $I_{g}=G_{c}(C_{t},I^{{}^{\prime}}_{s},M_{g})$ is filtered out with $M^{fg}_{g}$ and so is the ground truth $I_{t}$ with $M^{fg}_{t}$, where $M^{fg}_{g}$ and $M^{fg}_{t}$ represent $M_{g}$ and $M_{t}$ without the background channel, respectively. Afterward, a global structural information and other low-frequency features are obtained from calculating the L1 distance function:

where $\otimes$ represents the pixel-wise multiplication.

For the discriminator, we constructed the coloring discriminator $D_{c}$ against $G_{c}$ to distinguish two pairs: one including $I_{t}$ and $I_{s}$, and the other including $I_{g}$ and $I_{s}$. With the additional real image $I_{s}$, $D_{c}$ impels $G_{c}$ to generate more realistic images. Moreover, because this is a binary classification problem (i.e., the image is real or fake), we employed the binary cross-entropy loss as the GAN loss to compare the generated images:

where $G_{c}$ attempts to deceive $D_{c}$ to recognize the synthesized image as a real image; thus, the goal of $\mathcal{L}_{BCE}$ in $\mathcal{L}^{G_{c}}_{GAN}$ is equal to $1$. In contrast, because $D_{c}$ must classify the generated or real images correctly, the goals of $\mathcal{L}_{BCE}$ in $\mathcal{L}^{D_{c}}_{GAN}$ are equal to $0$ and $1$, respectively. In summary, the overall loss function of segmentation region coloring is as follows:

Because users care most about the characteristics of products, the performance of the virtual try-on service is highly dependent on the saliency of the synthesized image, for example, users (e.g., facial details or body shape), clothing features (e.g., button or bow tie), and 3D physics (e.g., pleat and shadows). Hence, in the fourth stage, we proposed two networks to refine the facial and clothing regions separately.

To train and evaluate the proposed TF-TIS, a dataset containing two different poses and one clothing image for each person is required. Still, most of the existing datasets provide either only one pose for each person with the corresponding clothing image [1, 9] or multiple poses for each person but without clothing images[53]. Therefore, we collected a new large-scale dataset containing $10,895$ in-shop clothes with the corresponding images of mannequins wearing in-shop clothes in two different poses.³³3Please refer to the images in https://github.com/fashion-on/FashionOn.github.io. In addition, the DeepFashion dataset [53], with a size of $288\times 192$, is also adopted to broaden the diversity of the data. After removing the incomplete image pairs and wrapping one in-shop clothing and two human images into each triplet, $11,283$ triplets were created. Finally, we randomly split the dataset into the training set and the testing set with $9,590$ and $1,693$ triplets, respectively.

Cloth2pose. We initialize the first 10 layers with that of the VGG-19 [44] and fine-tune them to generate a set of clothing feature maps $F$ from the information on the in-shop clothing. For the following convolutional blocks, each contains five convolutional layers with a $7\times 7$ kernel and two with a $1\times 1$ kernel. Each layer is followed by a ReLU. In this stage, we apply $N_{c2p}=4$ to the number of the convolutional blocks.

Pose-guided Parsing Translator. Based on the framework of ResNet, we implement two downsampling layers, nine residual blocks, and followed by two upsampling layers. Specifically, we construct two single-stride convolutional layers with a $3\times 3$ kernel and one highway connection, combining the input and the output of each corresponding residual block.

Segmentation Region Coloring. The architecture is composed of the encoder and decoder with six residual blocks between them. Except for the last residual block and one fully connected layer, each block contains two single-stride convolutional layers with a $3\times 3$ kernel, one downsampling two-stride convolutional layer with a $3\times 3$ kernel. The number of filters of all convolutional layers linearly increases and decreases, respectively, for the encoder and decoder.

Salient Region Refinement. The generator of FacialGAN ($G_{rf}$) is similar to $G_{c}$ but without the fully connected layer. In addition, $G_{rf}$ has four residual blocks containing two convolutional layers and one downsampling convolutional layer. For ClothingGAN, the generator ($G_{rc}$) comprises two different encoders and one decoder. The detail encoder ($E_{D}$) consists of four downsampling convolutional layers and three convolutional layers, and the warped-clothing encoder ($E_{W}$) consists of four downsampling convolutional layers and one convolutional layer. All downsampling convolutional layers have a $4\times 4$ kernel and a $2\times 2$ stride, and other convolutional layers have a $3\times 3$ kernel and a $1\times 1$ stride. Both kinds of convolutional layers are followed by the IN layer and LeakyReLU. The decoder ($Dec$) consists of five $3\times 3$ convolutional layers, and each layer is followed by one upsampling layer, one IN layer, and one ReLU.

For the context discriminator ($D_{rc}$), we adopt two discriminators: (1) the global discriminator, which consists of four downsampling convolutional layers and outputs a 1024-dimensional vector representing the global consistency, and (2) the local discriminator, which consists of three downsampling convolutional layers and outputs a 1024-dimensional vector representing the local consistency. A fully connected layer and sigmoid function are applied to the concatenation of the two vectors to differentiate whether the image is real or generated.

We used Adam [54] with $\beta_{1}=0.5$ and $\beta_{2}=0.999$ as the optimizer for all stages. The learning rates of the pose-guided parsing translator and the other stages are 2e-4 and 2e-5, respectively.

Several try-on results are depicted in Fig 3, 5, 6, and 7.

Because the structural similarity (SSIM) [57] and inception score (IS) [58] are fairly standard metrics that focus on the overall quality of the generated image instead of the pixel-wise comparison, we calculated them for the reconstruction of the try-on results in our dataset. The SSIM measures the similarity by comparing the generated images against the original images in the structural information, whereas IS provides scores to indicate whether the generated results are visually diverse and semantically meaningful.

Compared with the other virtual try-on systems (i.e., VTNCAP, CP-VTON, GFLA+CP-VTON, and FashionOn), our method outperforms them in terms of SSIM and IS, as revealed in Table II. Moreover, TF-TIS outperforms VTNCAP and CP-VTON in terms of IS by $18.9\%$ and $8.14\%$, respectively. Additionally, the comparison in term of SSIM indicates that TF-TIS exceeds VTNCAP and CP-VTON by $19.8\%$ and $11.5\%$, respectively. Although TF-TIS only surpasses the results of FashionOn within 1% in both metrics, the result complements the important details and the local and global consistency that FashionOn lacks, as demonstrated in Fig 3.

Runtime. We evaluated the efficiency of the proposed TF-TIS by separately reporting the running time of the four modules. The results of the runtime were conducted on a NVIDIA 1080-Ti GPU and were averaged with 2000 randomly selected image sets. The runtime of each module is as follows: cloth2pose (1.3 ms), pose-guided parsing translator (2.6 ms), segmentation region coloring (3.1 ms), and salient region refinement ($G_{rf}$: 1.9 ms, $G_{rc}$: 2.6 ms). The results indicate that the proposed TF-TIS not only reduces the cost of hiring photographers but also provides a real-time try-on service for fashion e-commerce platforms.