DAE-GAN: Dynamic Aspect-aware GAN for Text-to-Image Synthesis

Comprehensive understanding of text semantics plays a vital role in text-to-image synthesis. Previous methods mainly extract text features from sentence-level and word-level. However, they overlook aspect information contained in the text, which refers to several words rather than a word that depicts a particular part or feature of something, e.g., ‘red eyes’ in ‘the black bird is medium sized and has red eyes’. The granularity of aspect-level information is between those of sentence-level and word-level information. It could be helpful for the refinement of image details and should gain more attention. As shown in Figure 2, we represent text features from multiple granularities, i.e., sentence-level, word-level, and aspect-level. We use a Long Short-Term Memory (LSTM) network to extract the semantic embedding of text description $T$, which is formulated as follows:

$$\bm{s},\bm{W}=\text{LSTM}(T),$$

(1)

where $T=\{T_{j}|j=0,1,...,l-1\}$ consists of $l$ words. $\bm{W}=\{\bm{W}_{j}|j=0,1,...,l-1\}\in\mathbb{R}^{l\times{d_{w}}}$ represents word-level features that are obtained from the hidden state of LSTM at each time step. Here, $d_{w}$ means the dimension of text embedding. $\bm{s}\in\mathbb{R}^{d_{w}}$ stands for sentence-level semantic features from the last hidden state of LSTM.

We further employ the Conditioning Augmentation (CA) [38] to augment the training data and avoid overfitting by resampling the input sentence vector from an independent Gaussian distribution. Specifically, we enhance sentence features with CA, which is represented as follows:

$$\bm{s}_{ca}=F^{ca}(\bm{s}),$$

(2)

where $F^{ca}(\bm{\cdot})$ stands for the CA function, and $\bm{s}_{ca}$ is the augmented sentence semantic representations with CA.

As mentioned before, aspect information is very crucial for the details of generated images. However, since the focus and description of different sentences are different, it is hard to identify and extract the proper aspect information for each sentence. To this end, we employ the syntactic structures to tackle this problem. Specifically, we first adopt NLTK to make the POS Tagging for each sentence. Then, we manually design different rules to extract aspect information for different datasets. After that, we can obtain the aspect information $\{\bm{asp}_{i}|i=0,1,...,n-1\}$. Next, we use an LSTM to integrate this information and extract the aspect-level features, which is formulated as follows:

$$\bm{A}=\text{LSTM}(\{\bm{asp}_{i}|i=0,1,...,n-1\}),$$

(3)

where $\bm{A}$ denotes the aspect-level feature representation for text description and $n$ is the the number of extracted aspects.

Following the common practice, we first generate a low-resolution image at the initial stage. As illustrated in Figure 2, we utilize the augmented sentence embeddings $\bm{s}_{ca}$ and a random noise vector $\bm{z}$ to generate an initial image $I_{0}$. $\bm{z}\sim N(0,1)$ is sampled from a normal distribution. Mathematically, we use $\bm{R}_{0}$ to denote the corresponding image features at the initial stage:

$$\bm{R}_{0}=F_{0}(\bm{s}_{ca},\bm{z}),$$

(4)

where $F_{0}$ is the image generator at the initial generation stage. As depicted in Figure 2, it consists of one fully connected layer and four upsampling layers.

To the best of our knowledge, we are the first to introduce aspect information contained in the given sentence into text-to-image synthesis. Therefore, how to integrate the aspect information into image refinement stage is the main challenge that we should tackle. Inspired by human learning behaviors, in this work, we develop a novel Aspect-aware Dynamic Re-drawer (ADR) to refine images with the consideration of aspect information in the sentence. Specifically, we design a novel Attended Global Refinement (AGR) module to employ fine-grained word-level features for global refinement, and a novel Aspect-aware Local Refinement (ALR) module to utilize aspect-level features for local enhancement. By alternately applying these two components in a dynamic way, we are able to refine image details from both global and local perspectives. In the following part, we will take the $i^{th}$ refinement operation for generated image as an example to introduce the technical details of AGR and ALR.

To generate a photo-realistic image and ensure the semantic consistency between text description and the corresponding image simultaneously, we carefully design the loss function. During each step, the generator $G$ (e.g., ADR) and the discriminator $D$ are trained in an alternative fashion. To start with common practice, the objective loss function of each generator at each step is defined as follows:

$\displaystyle L_{G_{i}}=-\frac{1}{2}[$

$\displaystyle\underbrace{\mathbb{E}_{I_{i}\sim{p_{G_{i}}}}\log{D_{i}(I_{i})}}_{\text{unconditional~{} loss}}+\underbrace{\mathbb{E}_{I_{i}\sim{p_{G_{i}}}}\log{D_{i}(I_{i},T)}}_{\text{conditional~{}loss}}],$

(7)

where the first unconditional loss term is derived from the discriminator in distinguishing between real and fake images. The second term is a conditional loss to make the synthesized image match the input sentence.

Traditionally, the conditional loss term consists of sentence-image and word-image pairs. Different from previous works, we introduce aspect information through the generation process. To ensure the generated images truly contain local fine-grained details that match the corresponding aspects, we also include an aspect-image matching pair in the conditional loss as follows:

$$D(I,T)=D(I,\bm{s})^{\beta_{1}}\cdot D(I,\bm{W})^{\beta_{2}}\cdot D(I,\bm{A})^{\beta_{3}},$$

(8)

where $D(I,\bm{s})$, $D(I,\bm{W})$, and $D(I,\bm{A})$ calculate the matching degrees between the image and the sentence, word, and aspect, respectively.

Following [45, 36], we further utilize the DAMSM loss [36] to compute the matching degree between images and text descriptions, mathematically denoted as $L_{\text{DAMSM}}$. And the CA loss is defined as the Kullback-Leibler divergence between the standard Gaussian distribution and the Gaussian distribution of training text, i.e.,

$$L_{CA}=D_{KL}(\mathcal{N}(\mu(\textbf{s}),{\sum}(\textbf{s}))||\mathcal{N}(0,I)).$$

(9)

The final objective function of the generator networks is composed of the aforementioned three terms:

$$L_{G}=\sum_{i}L_{G_{i}}+\lambda_{1}L_{CA}+\lambda_{2}L_{DAMSM}.$$

(10)

For adversarial learning, each discriminator $D_{i}$ is trained to precisely identify the input image as real or fake by minimizing the cross-entropy loss. The adversarial loss for each discriminator $D_{i}$ is defined as:

		$\displaystyle L_{D_{i}}=-\frac{1}{2}[\underbrace{\mathbb{E}_{I_{i}^{GT}\sim{p_{GT}}}\log{D_{i}(I_{i}^{GT})}+\mathbb{E}_{I_{i}\sim{p_{G_{i}}}}\log{D_{i}(I_{i})}}_{\text{unconditional~{}loss}}$		(11)
		$\displaystyle+\underbrace{\mathbb{E}_{I_{i}^{GT}\sim{p_{GT}}}\log{D}_{i}(I_{i}^{GT},T)+\mathbb{E}_{I_{i}\sim{p_{G_{i}}}}\log{D_{i}(I_{i},T)}}_{\text{conditional~{}loss}}],$

where the unconditional loss is responsible for distinguishing synthesized images from real ones and the conditional term determines whether the image matches the input sentence. $I_{i}^{GT}$ is sampled from the real image distribution $p_{GT}$ at the $i^{th}$ step. The final objective function of the discriminator networks is $L_{D}=\sum_{i}L_{D_{i}}$.

Datasets. To demonstrate the capability of our proposed method, we conduct extensive experiments on the CUB-200 [32] and COCO [12] datasets, following previous text-to-image synthesis works [36, 45, 18, 38]. The CUB-200 dataset contains $200$ bird categories with $8,855$ training images and $2,933$ test images. Each image in CUB-200 has 10 text captions. For the COCO dataset, it consists of a training set with $80k$ images and a test set with $40k$ images. There are 5 captions for each image in COCO.

Evaluation Metrics. Following [36, 45], for better comparison, we quantitatively measure the performance of DAE-GAN in terms of Inception Score (IS) [24], Fréchet Inception Distance (FID) [7], and R-precision [36].

We obtain IS by employing a pre-trained Inception-v3 network [27] to compute the KL-divergence between the conditional class distribution and the marginal class distribution. A large IS indicates generated images have a high diversity for all classes, and each of them could be clearly recognized as a specific class rather than an ambiguous one.

FID computes the Fréchet distance between the synthetic and real-world images based on the feature map output from the pre-trained Inception_v3 network. Lower FID score means a closer distance between the generated image distribution and real image distribution and therefore implies the model is capable of synthesizing photo-realistic images.

R-precision is used to evaluate the semantic consistency between the synthetic image and the given text description. Similarly, we calculate the cosine distance between the global image vector and 100 candidate global sentence vectors to measure the image-text semantic similarity. The lower R-precision means better semantic consistency between synthesized images and given text descriptions.

Implementation Details. ^***https://github.com/hiarsal/DAE-GAN For aspect rules, each (adjective, noun) pair is an aspect that describes an object or scene. For COCO that contains layout and location, if a preposition is before the pair that expresses a relative spacial relationship, we will also add it. Consistent with [36, 45], we adopt Inception-v3 [27] pre-trained on ImageNet [23] as image encoder, and use pre-trained LSTM [36] as text encoder. The size of the initial generated low-resolution image (i.e., $N_{0}$) is set to $64\times{64}$. The finally synthesized high-resolution image at the last step has the size ($N_{n}$) of $256\times{256}$. During intermediate steps, all the image size ($N_{i}$) is fixed to $128\times 128$. Empirically, we set $d_{w}=256$ and $d_{r}=64$ to be the dimensions of text and image feature vectors, respectively. For other related hyperparameters, we set $(\beta_{1},~{}\beta_{2},~{}\beta_{3})=(1,~{}1,~{}0.2)$. For CUB-200, we set $(\lambda_{1},~{}\lambda_{2},~{}n)=(1,~{}5,~{}2)$, and for COCO, $(\lambda_{1},~{}\lambda_{2},~{}n)=(1,~{}50,~{}3)$. During training, we use the Adam optimizer with the learning rate of $0.0002$ to train the networks on 8 NVIDIA Tesla V100 GPUs in parallel with the batch size of 32 on each one. DAE-GAN is trained with 600 epochs and 120 epochs on CUB-200 and COCO respectively.

Performance on CUB-200. We compare our methods with state-of-the-art methods on CUB-200. The overall results are summarized in Table 1, 2, and 3.

It is clear that our proposed DAE-GAN achieves highly comparable performance, especially for FID and R-precision scores that measure photo realism and semantic consistency respectively. Specifically, DAE-GAN first learns comprehensive text semantics from multiple granularities, i.e., sentence-level, word-level, as well as aspect-level. This is one of the reasons that DAE-GAN could improve FID and R-precision scores against other baselines by a large margin. Moreover, ADR, the core component of DAE-GAN, is developed to refine images by alternately applying AGR and ALR in a dynamic manner, in which AGR enhances images from the global perspective with word-level features and ALR refines images from the local perspective with aspect-level features. This is another vital reason to make our synthesized image more photo-realistic and keep semantic consistency between text and image.

CUB-200 is a dataset full of description details. Therefore, models with comprehensive text semantic understanding achieve much better results than coarse-grained ones. For example, GAN-INT-CLS and StackGAN only leverage sentence-level features as input. Based on them, AttnGAN and DM-GAN employ word-level features to refine images and achieve higher performance. RiFeGAN is particularly designed for datasets with fine-grained visual details, e.g., CUB-200. It synthesizes images from multiple captions, which on the one hand leads to very high IS due to more caption details, on the other hand causes low R-precision score. Differently, our DAE-GAN can synthesize images with high FID and R-precision scores only from one given caption. This is largely achieved by comprehensive representation and utilization of text information, including sentence-level, word-level as well as aspect-level.

Performance on COCO. We also evaluate our method on COCO that has multiple objects, complex layout and simple details. The relative results are reported in Table 1, 2 and 3. We also list the observations as follows:

DAE-GAN still achieves the best quantitative performance against baseline methods with regard to IS, FID and R-precision. The results demonstrate that DAE-GAN is also capable of synthesizing well semantic consistent images with multi-object and complex layout. Comprehensive understanding of conditional text description and the newly proposed refinement paradigm ADR make it possible for DAE-GAN to refine different objects with dynamically provided aspect information. It is also the main reason that DAE-GAN can generalize well on different datasets.

To evaluate the visual quality of generated images, we show some subjective comparisons among AttnGAN [36], DM-GAN [45] and our proposed DAE-GAN in Figure 4.

In CUB-200, we can obtain that DAE-GAN generates better results. For example, when synthesizing a bird with the detail of long narrow bill ($4^{th}$ column), only DAE-GAN achieves this goal. Also in the $1^{st}$ and $3^{rd}$ columns, only DAE-GAN synthesizes photo-realistic and semantic consistency images. The reason is that DAE-GAN obtains comprehensive text representations, especially aspect-level features. Moreover, ALR is developed to dynamically enhance image details with aspect information.

In COCO dataset, we can also observe that images generated by DAE-GAN are more vivid and realistic. Taking examples in $6^{th}$ and $7^{th}$ columns of Figure 4, AttnGAN and DM-GAN often generate one object multiple times and the spatial distribution is also chaotic [36, 45], while DAE-GAN could address the problems well. By alternately applying AGR and ALR, DAE-GAN will not only enhance local details, but also refine images from a global perspective. This mechanism allows DAE-GAN to avoid getting stuck in a few most important words like other methods.

The overall experiments have proved the superiority of our proposed DAE-GAN. However, which component is really important for performance improvement is still unclear. Therefore, we perform an ablation study on COCO to verify the effectiveness of each part in ADR, including AGR and ALR. Corresponding results are illustrated in Table 4. According to the results, we can observe varying degrees of model performance decline when removing AGR and ALR separately from DAE-GAN. Recalling the aspect, since ALR is dependent on aspect information, we further remove aspect from AGR. The model performance also declines. The ablation study demonstrates comprehensive utilization of text information is helpful for image synthesis. AGR and ALR could employ the information well for image refinement.

Visualization of Generation Process. To evaluate model rationality and interpretability, we study the synthesis process in Figure 5. In the left example, DAE-GAN initially generates a low-resolution image (i.e., $I_{0}$) with the whole sentence. Then, based on $I_{0}$, ADR further employs fine-grained information (i.e., word-level and aspect-level features) to refine the image. Specifically, in image $I_{1}$, ADR pours its attention to the refinement respecting the aspect information of ‘a metallic blue-black back’ and improves the image size to $128\times 128$. Finally in $I_{2}$, ADR focuses on the refinement of aspect ‘an orange throat’ and further improves the resolution to $256\times 256$. Moreover, we can observe that in $I_{1}$, almost the bird head is in orange, while in $I_{2}$, only the throat is correctly drawn in orange and the image looks more vivid. In the right example, we present a visualization study in COCO with three aspects. We could observe that the images are also well generated under the corresponding guide of aspects ‘a grassy field’, ‘with wild animals’ and ‘underneath a cloudy sky’.

Influence of Aspect Order. As aspect information is placed in a significant position in our model, and it is dynamically utilized to refine local details, we are curious whether the order of aspect input will affect the generation results. Thus, an example case is studied in Figure 6. To be specific, the given text description has three aspect features as shown in Figure 6. We exchange the input order of the last two aspect features to explore how the images generated at the related steps will correspondingly change with the variation of aspect input orders. Meanwhile, we will not change the input of sentence-level features and word-level features. Taking the look at the examples of upper part, when refining the aspect of ‘blue and black wing feathers’, we could find the bird feathers are vividly drawn in blue and black. Then in $I_{3}$, it is obvious that the color of bird beak is changed from light black in $I_{2}$ to light blue. In the examples of bottom part, ADR focuses on the aspect ‘a light blue beak’ in $I_{2}$. Visually compared with $I_{1}$, the bird beak color turned light blue from light black. Then in $I_{3}$, the feathers are refined more vividly in blue and black than the ones in $I_{2}$.

The experimental results again fully confirmed the importance of aspect information to image refinement. Meanwhile, DAE-GAN can make full use of the aspect information to achieve image refinement in a dynamic manner. Furtherly, these examples also illustrate that our proposed DAE-GAN has good interpretation.