StableGarment: Garment-Centric Generation via Stable Diffusion

Diffusion Model (DM). DM [26, 49] belongs to the category of generative models that denoise from a Gaussian prior $\mathbf{x_{T}}$ to target data distribution $\mathbf{x_{0}}$ by means of an iterative denoising procedure. The common loss used in DM is:

where $\mathbf{x_{t}}$ is an noisy image constructed by adding noise $\bm{\epsilon}\in\mathcal{N}(\mathbf{0},\mathbf{1})$ to the natural image $\mathbf{x_{0}}$ and the network $\bm{\epsilon_{\theta}(\cdot)}$ is trained to predict the added noise. At inference time, data samples can be generated from Gaussian noise $\bm{\epsilon}\in\mathcal{N}(\mathbf{0},\mathbf{1})$ using the predicted noise $\bm{\epsilon_{\theta}}(\mathbf{x_{t}},t)$ at each timestep $t$ with samplers like DDPM [26] or DDIM [48].

Latent Diffusion Model (LDM). LDM [43] is proposed to model image representations in autoencoder’s latent space. LDM significantly speeds up the sampling process and facilitates text-to-image generation by incorporating additional text conditions. The LDM loss is:

where $z_{0}$ represents image latents and $\bm{\tau_{\theta}(\cdot)}$ refers to the BERT text encoder [15] used to encodes text description $\mathbf{c_{t}}$.

Stable Diffusion (SD). SD is a widely adopted text-to-image diffusion model based on LDM. Compared to LDM, SD is trained on the large LAION [46] dataset and replaces BERT with the pre-trained CLIP [42] text encoder.

Overview. The overall architecture of StableGarment is depicted in Fig. 2. The model is segmented into three primary components: a garment encoder, a try-on ControlNet, and a pre-trained stable diffusion model. The parameters of the pre-trained stable diffusion, including the VAE and UNet, are fixed. A trainable duplicate of the UNet is developed on top of the original, specifically designed to process garment input. Additionally, a try-on ControlNet accepts input from both the pose image and the image context. Both the garment encoder and the try-on ControlNet are integrated with the denoising UNet in an additive fashion, thereby offering the flexibility for seamless model switching.

Garment Encoder. The garment encoder, integral to our model, processes the garment input with a critical objective: to capture the intricate details of the garment. Drawing inspiration from recent studies [39, 58, 59, 27], we have devised our encoder composed of garment UNet and a pretrained VAE encoder. The garment UNet is a trainable copy of the denoising UNet and interacts with the denoising UNet via additive self-attention (ASA) layers. The underlying rationale for this configuration is to infuse reference information across multiple scales, thereby extracting the finest details from the reference image while simultaneously preserving its capacity for model switching.

Try-on ControlNet. To enable the StableGarment framework for virtual try-on tasks, we have developed a try-on ControlNet. This network is engineered to integrate the target body shape and image context into the denoising UNet workflow. The target body shape is precisely captured using DensePose[55], while the image context includes elements such as the background, skin tone, and attributes. A garment mask $m$ is employed to highlight areas in need of inpainting. Acknowledging the tight correlation between image context, inpainting area, and the pose, these conditions are concatenated into a unified input to the try-on Controlnet. Starting with an image context $I$, a garment mask $m$ is produced following the methodology outlined in [14]. Consequently, the inputs for the try-on ControlNet comprise the garment mask $m$, the masked image $I_{m}=I\cdot m$, alongside pose guidance $p$. The control signals of the ControlNet are directly infused into the denoising UNet as feature residuals. This setup enables the try-on ControlNet to deliver precise, pixel-aligned control signals for virtual try-ons, drawing from the provided image context to ensure accurate garment alignment.

Text Prompt. To preserve the model’s ability to follow prompts, we dispatch distinct prompts to both the garment UNet and the denoising UNet. Specifically, the garment UNet receives prompts of the garment category, facilitating its understanding of the input garment’s dynamic nature. Concurrently, the denoising UNet is prompted with descriptions of the target image, maintaining its proficiency in generating images responsive to a diverse range of textual cues.

Dataset Preparation. For the preparation of our dataset, we employ a pre-trained model to generate high-quality synthesized garment data, which is crucial for a precise and controllable garment-centric learning process. The primary objective of the data engine is to generate garment-centric images across a wide range of text prompts, thereby preserving the model’s ability to follow text prompts throughout the learning phase. A detailed demonstration can be found in the supplementary material. Our data engine is divided into three modules:

• •

  Parser: The parser module is tasked with creating a parse map and dense map from an input image. It incorporates a pre-trained parsing segmentation network alongside Detectron2 [55], facilitating accurate segmentation of the garment.

• •

  Tagger: The tagger module generates detailed text descriptions for target garment images. It utilizes CogVLM-chat [52] for generating text descriptions, and GPT4 to create inpainting templates, which are subsequently used by the drawer module.

• •

  Drawer: The drawer module functions as the synthesized data generator. It uses ControlNet [59] in conjunction with a pre-trained inpainting model, epiCRealism [4], to produce the final synthetic garment images.

Learning Objectives. The training for the StableGarment model incorporates a two-stage strategy. Initially, in the first phase, we utilize synthetic data to train our garment encoder. This training focuses on accurately capturing the nuances of clothing details while maintaining the ability for textual modifications. In the second stage, we freeze our garment encoder and only train our try-on ControlNet. We persist with the same learning objective, incorporating tryon-specific conditions. The loss functions for the two stages are expressed as Eq. (5):

where $I_{g}$ and $I$ are the garment reference image and the image context respectively, $p$ is the pose guidance from the image context, and $m$ denotes the corresponding garment mask.

Inference. As illustrated in Table 1, StableGarment is capable of dealing with various tasks. In addition to using different components for different tasks, the inference pipelines are different. In the context of garment-centric generation tasks, our method is flexible and generalizable to other third-party plugins, e.g., stylized base models and ControlNets. Specifically, for virtual try-on tasks, we employ an inpainting pipeline to achieve both virtual try-on consistency and cloth-agnostic region preservation. Concretely, each updating denoising step can be rewritten as:

where $z_{t-1}^{src}$ represents the noised image latents from the source model input and $\epsilon_{t-1}$ represents predicted noise. This adjustment ensures the seamless preservation of background elements and model details.

Subject-driven Generation Results. Fig. 3 displays our StableGarment can generate reasonable outputs following the different prompts while preserving clothing texture under different diffusion base models. BLIP-Diffusion and IP-Adapter can only preserve partial texture features of the garment, making them prone to losing details such as shape and text. In contrast, our method not only achieves better fidelity in texture, shape, and detail preservation but also offers adaptability in style transfer when switching base models.

Virtual Try-on Results. Fig. 4 shows a visual comparison of our outputs to other baselines on VITON-HD. Although previous methods synthesize reasonable warping garments and preserve the background well, they tend to generate blurry try-on results with manifest artifacts. Diffusion-based methods greatly improve the garment generation quality, but most of these methods fail to capture accurate clothing texture, e.g. text and tiny patterns. Due to the meaningful feature spaces of our proposed garment encoder, the try-on cloth successfully preserves the slight details. For example, our method accurately captures the ‘ADIDAS’ on the cloths in the last row. The quantitative results on VTION-HD convincingly verify that our method can achieve a more realistic try-on effect for these garments. More examples of our virtual try-on results will be reported in the supplementary materials.

Comparison with Subject-driven Generation Methods. We propose a garment-centric benchmark as described above to evaluate our method and other baselines, which includes 14 garments and 6 different prompts. Table 2 provides a detailed quantitative evaluation of these methods. Overall, both IP-Adapter and our method demonstrate the ability to generate accurate outputs that adhere to the given text descriptions. Moreover, our method clearly outperforms previous state-of-the-art fine-tuning-free methods on the CLIP-I and Aesthetic score metrics, showcasing the capability of our garment encoder to capture comprehensive and fine-grained features from reference garments.

Comparison with Visual Try-on Methods. We compare our method with existing baselines on the VITON-HD dataset and report the results in Table 3. Our method demonstrates competitive performance across all metrics compared to the baselines, particularly in the unpaired metrics(i.e., FID and KID). The success of our approach can be attributed to the garment encoder and DensePose-based try-on ControlNet, which effectively capture fine-grained garment features and generate high-quality try-on images. Furthermore, we utilize DINO-M to evaluate the recovery of cloth texture in the unpaired setting. Under this metric, our method displays competitive performance, further demonstrating the superiority of our method in the virtual try-on task.

Additive Self-Attention. To study the effectiveness of our ASA operator, we built a model with concatenated self-attention and trained with the same setup. The qualitative and quantitative comparison can be found in Fig. 6 and Table 5 . When testing under standard setup, the quantitative performance of both models is comparable. However, when switching the base models into stylized ones, it is visually obvious that the images generated via the CSA model often suffer from unavoidable artifacts. Especially when the text prompt becomes complex, such as a detailed background description, the CSA model often misinterprets the attributes of the garment. Instead, the ASA counterpart generates stylized images with better quality. Both details of the garment and style of the base model can be well preserved by the ASA model.

Effects of Synthesized Data. We also ablate the benefits of using synthesized data. The comparing setups are: 1) training with VITON-HD and 2) training with our synthesized data. From Table 5, we found that on the GC genearation benchmark, their performances are close under the standard settings. However, when testing the prompt following capacity on other stylized base mdoels, the model trained with synthesized data significantly outperforms its counterpart. From Fig. 6, the images generated by the model trained without synthesized data would display monotonous backgrounds, which reflects a characteristic of the VITON-HD dataset.

Comparison with Different Try-on ControlNet. We evaluate three types of try-on ControlNet, with three different control inputs: (1) garment-agnostic input and mask, (2) garment-agnostic input, mask and OpenPose, and (3) garment-agnostic input, mask, and DensePose. Fig. 6 shows a visual comparison of these three different setups of try-on ControlNets. All of these ControlNets enable to preserve detailed clothing texture due to our proposed garment encoder. However, without the aid of geometric information, it is difficult for our model to generate accurate human body, including arms and hands. In comparison to OpenPose control, DensePose control offers more geometric information. As shown in Fig. 6, replacing OpenPose with DensePose as a condition results in the disappearance of artifacts caused by occulusion in the generated images, with the thickness of the arms also better matching source model. We also conduct a comprehensive quantitative experiment to verify the effectiveness of our proposed DensePose-based try-on ControlNet. As presented in Table 6, when adding the OpenPose condition on top of the base inpainting ControlNet, all metrics improve, as the model gains more information about the human body structure. Replacing OpenPose with DensePose leads to improvements in the detailed morphology of the human body, and all metrics also show enhancements.