C3Net: Compound Conditioned ControlNet for Multimodal Content Generation

Diffusion models (DMs) consist of a class of probabilistic generative models capable of understanding the desired data distribution and synthesizing new samples, through a continuous application of denoising autoencoders in output generation. For the three dominant formulations, Denoising diffusion probabilistic models (DDPMs) [12] utilize two Markov Chains for image generation: a forward chain that injects random noise to the data and transforms the data distribution into an unstructured simple prior, and a reverse chain that denoises and recovers the original data by understanding the learnable transition kernels. Score-based generative models (SGMs) [41, 42] introduce score functions defined as the gradient of log probability density, adding a series of escalating Gaussian noise into the data and jointly calculating the scores for all noisy data distributions. Stochastic Differential Equations (SDEs) [42] can further be leveraged in the injection and denoising processes, allowing for the scenario of unlimited time steps or noise levels in DDPMs and SGMs. Latent diffusion models (LDMs) [36] first train a Variational autoencoder (VAE) [19, 35] to encode inputs into a low-dimensional and efficient latent space, and then apply a diffusion model to further generate latent codes. By abstracting negligible details and reducing modeling dimension, the motivation is to focus on the semantic aspects of the data to achieve higher computational efficiency. The diffusion models have achieved state-of-the-art synthesis quality in image inpainting, image superresolution, and audio generation from text.

Composable Diffusion (CoDi) [44] is a joint-modality generative model capable of producing a combination of output modalities in parallel based on a combination of input modalities, such as text, audio, image and video. CoDi first trains a latent diffusion model for each modality independently, adds a cross-attention module to each diffuser, and further apply an environment encoder to project the latent variables of different LDMs into a shared latent space. CoDi’s design enables multimodal generation without training on all possible combinations of modalities, reducing the size of training from one of exponential to linear.

Unimodal Models trained on large single-modality datasets can achieve a broader and more diverse coverage of real-world data distribution, without being constrained by the presence of cross-modality data pairs. Specifically, using unimodal models as pre-training can achieve better zero-shot performance compared with the jointly-trained multimodal models, with MAE [9] and T5 [33] outperforming the state-of-the-art CLIP-based model under similar model capacities. Moreover, as an effective unimodal pre-training technique for audio processing tasks, Self-Supervised Audio Spectrogram Transformer (SSAST) [7] enables models to learn the underlying patterns and features of large, unlabeled audio datasets and further improve their performance on the fine-tuning datasets. In the case of C3Net, we first apply unimodal pre-trained encoders for each modality, and then fine-tune the encoders on smaller-scale high-quality datasets based on contrastive learning.

Contrastive Language-Image Pre-Training (CLIP) [32, 34] is a neural network that aligns the text and image modalities by pre-training on a large dataset of text-image pairs with a contrastive loss function. Given a sample size of $N$ text-image pairs, CLIP learns to map the two modalities into a common embedding space by jointly training a text encoder and image encoder to maximize the cosine similarity of $N$ matched pairs and minimize the cosine similarity of ($N^{2}-N$) unmatched pairs using a contrastive loss function. Similar with CLIP, Contrastive Language-Audio Pre-Training (CLAP) [4] aligns the text and audio modalities via contrastive learning paradigm between the audio and text embeddings in pair, also following the same loss function. CoDi [44] proposes the “Bridging Alignment” technique to align conditional encoders for multimodal generation. CoDi leverages CLIP as the pretrained text-image paired encoder, and trains audio and video prompt encoders on audio-text and video-text paired datasets using contrastive learning, with text and image encoder weights frozen.

The above alignment techniques can be applied to align the latent space of LDMs with multiple modalities to achieve joint multimodal generation. In comparison, C3Net also utilizes the modality-specific encoders to align the conditions from multi-modalities to the same latent space, while it takes a step further by adding neural architecture similar with ControlNet [47] to facilitate better understanding of multimodal conditions and joint-modality generation.

ControlNet [47] learns and adds spatial conditioning to control the large pre-trained diffusion models. By freezing the original weights for the pre-trained diffusion model, ControlNet leverages a trainable copy of its deep-and-robust encoding layers to learn the diverse set of conditional controls and avoid overfitting. The original locked model and the trainable copy are then connected with a zero-initialized convolution layer called “zero-convolution”, where the convolution weights are first initialized to zero and progressively learned throughout training.

This architecture provides an effective solution for controlling large diffusion models, while ensuring that no new and harmful noises would be added to the deep features of the diffusion models. In the design of C3Net, we independently apply a ControlNet-like architecture to each input modality, further enabling our model to learn to coordinate multimodal conditions and synthesize more optimized results in the cross-modality generation.

C3Net (Compound Conditioned ControlNet) takes inspiration of the general architecture from [44] to enable multimodal generation conditioned on compound information, such as audio, text and image. C3Net first aligns multimodal conditions to a shared latent space. We consider a compound multimodal condition $c_{a}$, $c_{i}$, $c_{t}$, respectively denoting audio, image, and text conditions, and project them to a shared latent space $\xi$ using an encoder $C(\cdot)$.

We observed that text captioning is ubiquitously adapted in large-scale multimodal datasets as one of the ground truth labels, and that as mentioned in [44], certain dual modalities datasets (e.g., audio and image) are either harshly-aligned or scarce in quantity. To address this issue, choosing a shared latent space $\xi$ in which the text encoder is well-established is advisable. Following the practical implementation of [44], we adopt CLIP [32] latent space as our shared latent space $\xi$. Thus, an instance of the compound condition alignment stage yields a tuple of latent

$$C(x_{a}),C(x_{i}),C(x_{t})\in\xi$$

(1)

denoting the aligned latent from audio, image, and text conditions, respectively.

After acquiring latent conditions, we generate multimodal contents using latent diffusion models with C3-UNet and Control C3-UNet as the backbone. Specifically, we can sample feature maps $z_{0}$ of any modality from a diffusion model sampling process, which is conditioned on $C(x_{a}),C(x_{i}),C(x_{t})$. Note that the synthesis of different modalities utilizes different diffusion models. Then, $z_{0}$ is decoded on respective decoders to generate the content of its modality.

Our encoders $C(\cdot)$ are multi-modal encoders aligning conditions in different modalities to the shared space $\xi$. The encoders are first pre-trained on unimodal datasets and then fine-tuned using contrastive learning proposed in [32]. In contrast, the original settings of [44] is an alignment model trained from scratch on multi-modal datasets.

We propose to pre-train encoders on unimodal data because high-quality paired datasets are scarce for some modalities (e.g., audio and text pair). On the other hand, many high-quality unimodal datasets are readily available. In the following, we use the audio encoder as an example. As shown in Figure 3, we use the pre-trained neural net from [8] which is a masked auto-encoder. The MAE has been trained to extract audio features during the unsupervised training stage, which makes it easier for the following contrastive learning for the audio encoder. We then fine-tune the audio encoder using high-quality datasets, which are available on relatively small scales. During the fine-tuning stage of the audio encoder, we utilize an established frozen text encoder from [32]. In detail, we use contrastive objective to fine-tune the pre-trained audio encoder, so that the audio encoder learns to align audio to latent in $\xi$ as similar as possible to the latent that its ground truth caption is aligned to.

Similar to the findings in [49, 23, 22, 21, 28, 43], our encoder networks can primarily learn the data pattern for respective modalities with unsupervised pre-training. Trained with fewer but high-quality multi-modal data, our unimodal pre-trained encoder is on par with or outperformed encoders trained on only paired data on downstream generation tasks.

Figure 4 shows the multimodal diffusion model of our C3Net, which consists of the C3-UNet and Control C3-UNet. C3-UNet is a trained UNet $\mathcal{F}(\cdot,\Theta)$ employed in a latent diffusion model, which generates feature maps conditioned on instances in $\xi$. Control C3-UNet, similar to the ControlNet setting in [47], is a trainable copy $\mathcal{F}(\cdot,\Theta_{c})$ of the C3-UNet, where $\Theta_{c}$ denotes the trainable copy of parameters $\Theta$. In the implementation of C3Net, we use the trained UNet of Composable Diffusion [44] as C3-UNet $\mathcal{F}(\cdot,\Theta)$. Figure 4 shows the detailed architecture¹¹1 The trained $\mathcal{F}(\cdot,\Theta)$ is a U-Net with an encoder, a middle block, and a skip-connected decoder. The encoder and decoder contain 12 blocks, and the full model contains 25 blocks, including the middle block. Of the 25 blocks, there are 4 down-sampling and 4 up-sampling blocks. Refer to Figure 4. In C3-UNet, the “Encoder Blocks” contains 12 encoder blocks in 4 resolutions, while the “×3” indicates the block of the same resolution repeats three times. Condition latent is encoded using the Latent Encoder, and diffusion time steps are encoded with a time encoder using positional encoding. Similar to [47], the Control C3-UNet is a trainable copy of the 12 encoder block and 1 middle block of the C3-UNet. The feature maps are added to the 12 skip-connections and 1 middle block of the C3-UNet after a “zero convolution” layer..

The Control C3-UNet can provide additional information lost during the latent interpolation. Notably, our Control C3Net takes the aligned latent $C(x_{a}),C(x_{i}),$ and $C(x_{t})$ separately in each modality. The Control C3-UNet is trained to provide extra information in each condition by modifying the feature maps of the C3-UNet. Thanks to the shared latent space $\xi$, it is sufficient to train one trainable copy of parameters $\Theta_{c}$ for the Control C3-UNet. This is because conditions from all modalities have been aligned to the shared $\xi$, and a single set of trained parameters $\Theta_{c}$ is sufficient for additional control by taking condition latent already aligned in $\xi$ regardless of the original modality.

Our diffusion model follows a similar setting in [44] and [32]. The C3-UNet takes a linear interpolation of the aligned latent $C(x_{a}),C(x_{i}),C(x_{t})$ as the condition. The Control C3-UNet takes conditions in each modality separately and connects to the C3-UNet at each level of skip-connection after multiplying a constant, which we empirically set to be 0.1, but it varies depending on the generation task. Constant multiplied adjusts the additional control scale the Control C3-UNet provides when using different combinations of compound conditions. However, when only a single condition is provided, the Control C3-UNet can not provide additional information and therefore we set the constant to zero.

During training, we use the text-image dataset to train C3Net’s image and text generation, and the text-audio dataset for audio generation (training and validation datasets will be described in Section 4.1). Specifically, we train Control C3-UNet on each modality separately. Take image generation as an example, for each image-text pair, denoted as $I$ and $x$ respectively, in the dataset. The ground truth text $x$ is aligned to the shared latent space as $C(x)$, and a masked text $x_{m}$ is generated by randomly selecting 50% of the text prompt to replace with empty strings and it is aligned as $C(x_{m})$. The C3-UNet takes $C(x_{m})$, and Control C3-UNet takes $C(x)$ as condition latent, respectively. The ground truth image $I$ is used to generate $z_{0}$ in a typical latent diffusion model [36]. We train the C3-UNet to predict noise in a timestep $t$ during the image diffusion. Therefore, the objective function of each modality can be denoted as

$$\mathcal{L}_{c}=\mathop{\mathbb{E}_{z_{0},t,c,\epsilon\in\mathcal{N}(0,1)}}\left[\|\epsilon-\epsilon_{\theta}(z_{t},t,c)\|^{2}_{2}\right]$$

(2)

where $\epsilon$ is the ground truth noise, $\epsilon_{\theta}(\cdot)$ is the network, and $c$ is the tuple of $C(x_{m}),C(x)$.

We collected our training datasets for fine-tuning the alignment encoder as well as the respective Control C3-UNet for image, audio, and text synthesis. Major effort was made to clean up flawed data in some datasets through data prepossessing and relation scores, including CLIP score [10], CLAP [4] similarity score, and the data quality metrics.

For the fine-tuning of audio encoder, we used AudioCap [18], a dataset of sounds with event descriptions for audio captioning. We also added the sound effects from Epidemic Sound as provided in [46]. We selected 1 million ten-second sound clips from AudioSet [6] with optimal quality and CLAP similarity score to their text captions.

For the fine-tuning of image and text Control C3-UNets, we utilized the COCO [25] dataset and part of the LAION-400M [39] dataset, with both consisting of images and corresponding text captions. For the fine-tuning of audio Control C3-UNet, we utilized a combination of AudioCap [18] and AudioSet [6] for training.

In the absence of $k$-modality datasets, where $k>2$, for multimodal synthesis evaluation, it is crucial to construct a high-quality evaluation set with three modalities (i.e., image, text, and audio) for evaluating C3Net.

Observing that the AudioCap dataset [18] contains high-quality audios and text captions, we curated a tri-modal test set based on AudioCap. Specifically, we first generated the third modality (i.e., image) using Stable Diffusion [37] prompted on the AudioCap text captions. We further selected 2,000 data tuples based on image quality and CLIP score to ensure that the content for each tuple is highly correlated. As a result, a total of 2,000 tri-modal ground-truth tuples, each with highly relevant audio, image and text caption, are available for evaluating C3Net and CoDi [44], which is to date the most representative (and only) work on diffusion-based tri-modality content generation. We show some examples in Figure 5.

Figure 6 shows qualitative comparison on compound condition image synthesis between C3Net and our baseline. We will show quantitative comparison in the following.