Diffusion Model with Clustering-based
Conditioning for Food Image Generation

With recent advances in machine learning and modern computer vision, image-based dietary assessment methods have been introduced to automatically record and analyze participants’ food intake. The most important aspect of image-based dietary assessment is to detect and classify the food and beverages in the images captured by participants. DiaWear(shroff2008wearable, 72) is one of the pioneering works that uses a neural network to perform context-aware food recognition on food images captured by mobile devices. Deep learning-based detection methods, such as Faster R-CNN(girshick2017faster, 64) and YOLO(redmon2016you, 62) are frequently used for food detection and have shown good results when trained on a large number of ground truth images(han2021improving, 24, 11, 40). The most recent work focuses on estimating food portion size or energy directly from the eating occasion images. In (fang2016comparison, 15), a geometric model is introduced to estimate the food portion size. Conditional GAN (mirza2014conditional, 56) is used in (fang-icip2018, 14) to generate the energy distribution map with an end-to-end regression framework for food energy estimation (fang2019end, 13), which is further improved in (shao2021towards, 68) by integrating information from RGB images and in (shao2023endtoend, 70) by leveraging the 3D reconstruction. By combining these components, systems like goFOOD(lu2020gofoodtm, 50) and Technology-Assisted Dietary Assessment (TADA)(shao2021_ibdasystem, 69) can provide an end-to-end automatic nutrient estimation for a meal.

Despite the promising advancements in image-based dietary assessment methods, the performance of deep learning-based methods relies heavily on the quality and quantity of the datasets. Training a model for food recognition or portion estimation requires a large, diverse, and accurately annotated dataset. Nevertheless, the food images we collect from real-world scenarios often exhibit stark differences from established food image datasets such as Food2K (Food2K, 55), Food-101 (bossard2014food, 3), and UEC-256 (uec-256, 43). This discrepancy primarily stems from inherent limitations associated with real-world data, such as variability, inconsistent sizes, and potential inaccuracies in annotations. While various food recognition systems have been developed to target real-world scenarios, such as continual learning (ILIO, 28, 31, 26, 61, 30), few-shot learning (food_fewshot, 39), and long-tailed classification (gao2022dynamic_LTingredient, 18, 25, 32), they usually necessitate more sophisticated training regimes. This amplification in complexity invariably leads to a surge in computational requirements, which are typically constrained in real-world situations. A potential solution to mitigate the problem of data scarcity in the real world without requiring any changes to the existing image-based dietary assessment system is to use synthetic food images for the training process. However, generating synthetic food images could be challenging due to inter-class similarity and intra-class variability. In this work, we investigate the application of the stable diffusion model(rombach2022high, 65) for synthetic food image generation. We propose a novel framework for training a stable diffusion model to generate more diverse synthetic food images and demonstrate that these images can be used to address the data imbalance issue in tasks involving long-tailed food classification.

One of the main challenges of supervised machine learning or deep learning methods is the requirement for a large amount of annotated training samples, and the performance heavily relies on the quality of the dataset. One way to address this issue is by generating high-quality image data to augment the training dataset. For a long time, Generative Adversarial Networks (GANs) have been the most impactful methods and have shown remarkable results for synthetic image generation. These synthetic images have found applications in various fields, from video games to forensic scenarios. Many applications have utilized GANs to create synthetic images that can be used for generating ground truth information used in training machine learning networks (deepsynth, 10, 23).

There are few studies on applying GANs in the field of food image synthesis, such as RamenGAN (ito2018food, 37), Multi-ingredient Pizza Image Generator (MPG) (han2020mpg, 22), and CookGAN (han2020cookgan, 21). RamenGAN (ito2018food, 37) and its related work (horita2019unseen, 36) generate synthetic food images of noodles and rice based on a conditional GAN network. MPG (han2020mpg, 22) uses conditional StyleGAN2 (karras2020analyzing, 42) to generate realistic pizza images with various ingredients. CookGAN (han2020cookgan, 21) proposed a method to generate a meal image conditioned on a list of ingredients, with the types of foods limited to salad, a cookie, and a muffin.

In recent years, diffusion models have shown the ability to produce high-quality images while maintaining ease of use. The diffusion model defines a Markov chain to gradually add random noise to the data, and the model learns to reverse the diffusion process. As a result, the trained model is capable of constructing the desired data samples from the noise (ho2020denoising, 35). In (dhariwal2021diffusion, 9), the authors demonstrate that the diffusion model outperforms GANs and achieves better image generation results on multiple datasets. Additionally, in (rombach2022high, 65), the authors propose latent diffusion models, which apply the diffusion process in the latent space of the data, resulting in a computational reduction and a boost in the visual fidelity of the generated images. Despite diffusion models being applied to many downstream tasks (pinaya2022brain, 60, 8, 52), the study of applying them to food image generation is still lacking. In this paper, our work represents the first investigation and application of the diffusion model to food image generation.

There are two processes in diffusion models (ho2020denoising, 35): (i) a forward diffusion process to diffuse an image into random noise, and (ii) a reverse diffusion process that converts the noise into the image from a learned data distribution during the iterative denoising process.

In the forward diffusion process, a Markov chain is employed to gradually add noise to transform the training data into Gaussian noise. This discrete Markov chain is constructed with a predefined sequence of variance values $0<\beta_{1},...,\beta_{T}<1$. Given the input data distribution $q(x_{0})$, a forward diffusing process at time $t$ is defined as Equation 1:

where $\epsilon\sim\mathcal{N}(0,\mathbf{I})$ denotes the injected noise, and the forward process can be interpreted as sampling $x_{t}$ at any time step $t$ in a closed form by using the notation: $\alpha_{t}:=1-\beta_{t}$ and $\bar{\alpha_{t}}:=\prod_{n}^{s=1}\alpha_{s}$, which represent the noise level at each time step. At the final time step $T$, $x_{T}$ approximately becomes an isotropic Gaussian noise.

In the reverse diffusion process, we can recover the input data distribution $q(x_{0})$ from $p(x_{T})\sim\mathcal{N}(x_{T};0,\mathbf{I})$. The equations for this process are defined as follows:

The training object of the diffusion model is to find a denoising autoencoder $\epsilon_{\theta}(x_{t},t)$ to predict the injected noise $\epsilon$. The corresponding objective can be simplified as follows:

where $t$ is uniformly sampled from $\{1,...,T\}$.

The latent diffusion(rombach2022high, 65) follows similar processes as the diffusion model, except that it uses the latent space of images as input samples to the diffusion model. It utilizes a perceptual compression model(esser2021taming, 12) consisting of an encoder $\mathcal{E}$ and a decoder $D$. The encoder is used to map the image $x$ to the latent space $z=\epsilon(x)$ and the decoder is used to reconstruct the image $\bar{x}=D(z)$ from the latent space. The latent space representation reduces the dimension of the input data, therefore focusing on the critical features of the input data and reducing the computational complexity of the diffusion model.

We utilize Affinity Propagation (dueck2009affinity, 16) to cluster the training images from one class into sub-classes. This helps in addressing the high intra-class variance problem of the food dataset. A pre-trained ResNet-18 (he2016deep, 33), fine-tuned on the Food-101 dataset for food classification, is used to map the images into feature vectors. These feature vectors represent the visual features of the input food images and are used for clustering. Affinity propagation creates clusters by sending messages between samples until convergence, and it does not require a pre-defined number of clusters. A small number of most representative samples are referred to as exemplars to describe the dataset. Messages sent between pairs of samples indicate how well one sample can serve as the exemplar of the other, and these messages are iteratively updated in response to the values from other pairs. The updating process continues until convergence, and the final exemplars are used to form the clusters.

Suppose $z_{i}$ and $z_{j}$ are features vectors of two food images, we define $s(i,j)=1-\frac{{z_{i}\cdot z_{j}}}{{\|z_{i}\|\cdot\|z_{j}\|}}$ as the similarity between $z_{i}$ and $z_{j}$, which is the cosine distance between two vectors. In Affinity Propagation, two messages sent between the sample are responsibility $r$ and availability $a$. Responsibility $r(i,j)$ describes how well $z_{j}$ to be the exemplar for $x_{i}$, relative to other candidate exemplars for $z_{i}$, and it is defined as:

Availability $a(i,j)$ describes suitability for $z_{i}$ to pick $z_{j}$ as its exemplar, considering other samples’ preference for $z_{j}$ as an exemplar, and it is defined as:

Both responsibility $r$ and availability $a$ are set to zero at the beginning and updated during each iteration until convergence. The damping factor $\lambda$ is used during the process to prevent numerical oscillations. The updated functions for both responsibility and availability during each iteration are shown in Equation 6:

where $n$ indicates the iteration step.

After the input food images are clustered into sub-classes, we denote the new class label of each food image by its food name followed by the cluster ID (e.g., burger_1 for an image of a burger in cluster 1 of its class). A diffusion model is then trained to model the conditional distributions $p(x|y)$, where $x$ is the input image and $y$ is the class label of the input image. To achieve this, we train a conditional denoising autoencoder $\epsilon_{\theta}(x_{t},t,y)$, as mentioned in section 3.1, which allows us to control the image generation process through the class label $y$. By conditioning the diffusion model on the new class label, we can generate food images that belong to a specific cluster of a food class, and this helps in generating more diverse food images within each food class.

Similar to the latent diffusion model(rombach2022high, 65), we use a U-Net with an attention mechanism as our conditional denoising autoencoder. Figure 2 shows how the sub-class labels $y_{sub}$ control the denoising process through U-Net. To incorporate the sub-class information into the denoising process, we use a pre-trained CLIP text encoder $\tau_{\theta}$ to project $y_{sub}$ into an embedding vector $\tau_{\theta}(y_{sub})$, which is then added to the intermediate layers of the U-Net using a cross-attention mechanism. The implementation of the cross-attention mechanism is shown in the following equation:

where $q$, $k$, and $v$ are the query, key, and value for the attention mechanism, respectively. $l$ denotes the $l-th$ intermediate layer of U-Net, and $\pi_{l}(z_{t})$ is the intermediate vector representation of U-Net. $W_{q}$, $W_{k}$, and $W_{v}$ are learnable projection matrices, $\mathbf{A}$ is the attention map of cross-attention mechanism.

The new training objective of the conditional latent diffusion model is shown below:

where both denonising autoencoder $\epsilon_{\theta}$ and text encoder $\tau_{\theta}$ are jointly optimized together.

When generating images for a specific class of food, the latent diffusion model is conditioned with its sub-class labels to control the generation process, where the occurrence of the labels will follow the distribution of sub-classes within the class.

Compared methods. We compare our proposed ClusDiff with two other techniques: one of the latest GAN-based methods, StyleGAN3, and the baseline latent diffusion method. StyleGAN architectures have shown an ability to generate highly realistic images across multiple applications, including generating food images (han2020mpg, 22). Therefore, we choose the latest variant, StyleGAN3 (karras2021alias, 41), as the GAN-based method used for comparison. As for the baseline diffusion method, we select the latent diffusion model(rombach2022high, 65) to showcase the effectiveness of our proposed clustering-based training framework. This will allow us to demonstrate the improvements achieved by our modifications compared to the baseline.

We generate 100 synthetic images for each type of food in the Food-101 dataset using each image generation method and compare the FID scores. Fig. 3 shows the sample results of synthetic food images generated by each method. We manually selected three classes of food for demonstration, and the generated food images were randomly chosen. Table 1 shows the comparison results for different food image generation methods.

From Fig. 3, we can see that compared to the synthetic food images generated by StyleGAN3, the images generated by the latent diffusion method show significant improvement in the background of food images. Furthermore, the latent diffusion method generates more refined details in the food region compared to StyleGAN3, as evident by the intricate textures of the sushi fillings and the distinct presentation of hamburger ingredients. The FID scores from Table 1 further confirm our visual inspections, showing that the baseline diffusion method outperforms StyleGAN3 by approximately 10 FID points.

As shown in Fig. 3, the visual difference between the synthetic images generated by the fine-tuned latent diffusion method and our proposed ClusDiff, is minimal. This is because ClusDiff maintains the original network structure of the latent diffusion model.

The comparison between ClusDiff and the baseline latent diffusion model is shown in Table 1. Since both methods are fine-tuned on the same pre-trained weight, this comparison demonstrates that ClusDiff significantly enhances the FID results of the latent diffusion model. This improvement indicates that our proposed modifications in ClusDiff effectively assist the latent diffusion model in capturing the underlying distribution from the training images, leading to better diversity and fidelity in the generated images.

We demonstrate the effective use of synthetic food images in addressing the challenges of class imbalance associated with the long-tailed classification problem. Specifically, we generate food images for each of the 74 food classes in the VFN-LT dataset to construct a balanced training set. The food images are generated by using a pre-trained stable diffusion model with our clustering-based conditioning on the Food-101 dataset for a fair evaluation.

Compared methods. We compare with existing long-tailed image classification methods, including random over-sampling (ROS)(ros, 76), random under-sampling (RUS) (rus_ros, 5), context-rich oversampling (CMO)(cmo, 58) and visual-aware hybrid sampling (Food2stage)(he2022long, 25) as data augmentation-based approaches. In addition, several loss functions have been commonly used to address class imbalance including label-distribution-aware margin loss (LDAM) (ldam, 6), balanced Softmax (BS) (bsloss, 63), influence-balanced loss (IB) (ibloss, 59) and Focal loss (focalloss, 49). We use vanilla training as the baseline and our method as Baseline + ClusDiff by using a balanced training set containing original and synthetic food images generated by our proposed ClusDiff.

Table 2 summarizes the results on the VFN-LT dataset. Our method achieves the best performance in terms of head, tail, and overall classification accuracy. Compared with existing data augmentation based approaches, using synthetic images is more effective, which improves the generalization ability and avoids the over-fitting issue for both instance-rich and instance-rare classes. Moreover, the use of synthetic data streamlines the training process, eliminating the need for any new loss functions, thus offering an advantage over existing methods and showing great potential for real-life applications.