ComicGAN: Text-to-Comic Generative Adversarial Network

AttnGAN (?) is a state-of-the-art GAN-based model that generates images from text, using images with multiple corresponding text captions as training data. This model was reproduced and tested on original text-image datasets (?). This model was used as a baseline for experimentation of generation of comics, and was fine tuned and augmented to improve results. AttnGAN uses multiple generators and discriminators, with encoded input text and encoded images. First, a Deep Attentional Multimodal Similarity Model is trained. This model is created by training a text encoder, and an image encoder. The text encoder is a Recurrent Neural Network that encodes the input text into a feature vector. The image encoder is a Convolutional Neural Network that encodes an image into feature vector. The DAMSM trains the image and text encoders to learn the semantic representation of the text description within ground truth images, e.g. that the word red in the input text represents the colour red within the image. The GAN is then trained. The first generator stage creates an image based on the entire encoded input text. Further generator stages use encodings of the most important words, and the results of the previous generator stage to upscale the image resolution, and add detail. Each generator stage output is judged by a discriminator. The DAMSM compares the final generated image to the text input, and resulting loss is used to improve generators. ComicGAN first enables the generation of comics with AttnGAN by introducing a method to create text descriptions from a labeled image dataset. It then attempts to further improve generation of comics by creating a CNN image encoder designed specifically for comics.

To create a feature extractor for images, the Inception v3 CNN is trained on the ImageNet (?) dataset for multi-class classification. This same base network (with different final dense layers) is then used in AttnGAN as the feature extractor. Using the same transfer learning methodology, a multi-label classifier was trained specifically on comics, with the objective that a specialised CNN architecture could perform better feature extraction on comics, therefore creating higher quality comics with this input to the GAN. This Comics Classifier was trained and tested on the labels from which the comic descriptions were generated: the colour of the background and the characters in the comic. As it was time intensive and expensive to judge the results of image encoders while integrated as part of AttnGAN (due to the long training times), feature extraction capability was judged based on accuracy of the classifier. A higher accuracy means that the appropriate features are being extracted by the model.

The image encoder built into AttnGAN is based on the Inception v3 (?) CNN architecture. It is used to extract 17x17 regional features (which should represent individual features within an image such as a specific object or colour), and a final image encoding. To better capture these encodings of the features, a CNN created specifically to encode comics was designed and implemented. The architecture creates the same 17x17 regional features and final image encoding, but is adjusted and simplified to suit the simpler and more pronounced visual features of comics (compared to the more complex visual features of photo realistic images).

The comics CNN is constructed as follows: The input (image of size 299x299 pixels with 3 channels of red, green and blue) is put through two convolutional blocks, consisting of two 2-d convolutional layers and a max pooling layer, which reduce the spatial size from 299 to 35, while increasing the depth to 128. An inception block is created by creating three convolutional branches from the input. Each branch processes the input further with convolutional layers. Different size features are extracted as the layers of each have different stride lengths. Average or max pooling can be applied to the oupute of each branch, before the outputs are concatenated to give the output of the block. These blocks can replace convolutions with larger spatial features, while being less computationally expensive (?). Three inception blocks increase depth, and reduce spatial size to 17. These blocks output 17x17 feature regions. This output is then processed through two more inception blocks, before final processing by average pooling, dropout, and flattening layers. A final fully connected layer outputs the final number of classes. As in inception v3, the comics CNN uses the feature regions while training to improve the loss of the model. The 17x17 features are used to predict the final class output as well. The final loss which is back-propagated through the model is: $loss+0.4\times featureloss$ Where $loss$ is the binary cross entropy loss between the final layer prediction and the actual output, and $featureloss$ is the binary cross entropy (BCE) loss combined with sigmoid activation ($\sigma$) (equation 1, 2) between the feature layer prediction ($x$) and the actual output ($y$). The addition of the fractional loss $featureloss$ ensures that the regional features capture representative characteristics of the image.

A text dataset corresponding to the comic panels was created. The steps and mechanisms used to label comics and subsequently create the text descriptions are described in this section.

1. 1.

   Comic panels were manually labelled with characters, and the background colour. Recurring characters had a unique number, while non-recurring characters were grouped together under one number. Background colour was chosen as the most dominant colour present.

2. 2.

   From these labels, descriptions were generated. To create a basic description, character numbers were replaced with the text names, and separated by ”and”. To complete the description, the background colour was added to the text in the form of ”with colour background”.

3. 3.

   To increase the number of descriptions per panel, augmentation was performed on the descriptions. The order of the characters could be permuted, providing $nfactorial$ unique descriptions, where $n$ is the number of characters present. Descriptions were multiplied further by placing the background colour at the start, or end of the character portion of the description, $doubling$ the number of descriptions.

Image synthesis from comics dialogue required the parsing of the comic dialogue. A resource for transcriptions of Dilbert comics (?) was downloaded, and a script was created to parse the dialogue into a dataset where each comic panel was associated with the text that appeared in it.

Initially it was thought that the comics dialogue could be used as input text for generating comics. However, problems with this approach were evident immediately. Text-to-image models are trained on images and text. Each image has corresponding textual descriptions which describe the subject/object within the image. There is a direct semantic connection between the text and the image. This is not at all what the comics dialogue and subtext is. There is no description of the characters or objects within the comic (see Fig. 4). The connection between the dialogue of comics and the illustration is abstract at best.

Automatic generation of descriptions was infeasible without the training of a custom ML model (see Fig. 5). This is due to the fact that most description generators are trained on real life photographs, as opposed to illustrations. Therefore a study on correlation between comic and text had to be carried out. Images cannot be generated from text without a text dataset. Text descriptions had to be manually created. This led to questions: What type annotations should be created? What should they contain? How should they be composed such that the text-to-image models can learn the representation of the text that is present in the images?

A direct semantic connection between the text and contents of each image is needed for the text-to-image model to learn to create images that represent text. Initial intuition led to emulation of the descriptions in other text-image datasets, which contained a description of the main subject of the image, using adjectives such as colours, shape, and size. This led to complicated descriptions, for which it was infeasible to create enough data to learn these descriptions (see Initial experiments Section for results). Final text datasets were created from labels of characters and background colour present in the comic (See section Text Description Formulation).

Comics are different in their makeup in comparison to real images. Computer vision applications trained specifically on photo realistic images are not necessarily effective when applied to comics. This was observed when descriptions generators did not work appropriately on comics, but did on real life images (see above section). Features of comics are different to those in images. Generally they are simpler and bolder: Black lines distinguish all characters and objects, characters each have unique hair, facial features and clothes. Backgrounds are often a single block colour, and do not contain the same level of variation and detail (see original Dilbert and PhD comics in figures 1, 5).

GANs are notoriously hard to evaluate. They have no objective loss function, and results such as illustrations are subjective (?). Due to the differences between photo images and comic illustrations, common evaluation metrics (e.g., IS) are not applicable for evaluating generated comics, as they are suitable only for photo-realistic images. Furthermore, Comic illustration synthesis from text has not been previously studied. Baseline algorithms are not available to provide comparisons on novel models created in this research.

Absence of comic datasets with text descriptions proved to be a major problem. Text descriptions had to be manually created for each comic panel, which is time consuming and infeasible to carry out for the number of comic panels needed for a training set. Methods to create descriptions from character and colour labels helped partially automate text dataset creations, but it did not solve the problem. A set of comics, labelled with traits suitable to create descriptions was still needed. Such datasets do not exist. Due to time limitations of this project, only 2500 Dilbert comics were labelled. Filtering of unusable comics, this reduced the dataset size to 1000 panels. Moreover, data-augmentation transformation applicable to images (e.g., brightness, rotation, cropping) are not applicable to comics, as they should have specific colours and orientations. Ideally, evaluation of GAN generation methods with more data, and multiple different comic datasets such as (Garfield, PhD comics) is desired to form a more complete judgement. Furthermore, more descriptive text, rather than text derived from labels could improve generated image quality, and would serve as a good comparison to what text-to-image models can learn.

Three distinct experiments regarding CNN models for comic feature extraction were conducted. Training and validation accuracy over epochs, as well as accuracy and F1 score on a test set was used to analyse the performance of the multi-label classifier models.

Three different CNN architectures were experimented with:

• •

  Custom comics CNN architecture.

• •

  Inception v3 pre-trained on ImageNet, final classification layer trained on comics.

• •

  Inception v3 trained from scratch on comics.

The inception v3 architecture pre-trained on imagenet performed best (see Section 1, although it was prone to overfitting on the training set. Inception v3 architecture trained from scratch took far longer to train and did not reach the same accuracy compared to its pre-trained version. The comics CNN outperformed the scratch version of Inception, showing a 6% improvement in accuracy on the test set, and taking far less time and epochs to converge (see Fig. 7). While the comics CNN was not able to exceed accuracy of the pre-trained version of inception, the improvement over the scratch version of inception indicates that the comics CNN is more suited to extracting features of comics. Therefore, a version of comics CNN pre-trained on a large dataset of comics should outperform the pre-trained version of inception.

In this section, text-to-image models based on AttnGAN and GAN generated comics are evaluated. FID of comics are compared, as well as extensive qualitative evaluation. First, ComicGAN results will be compared to GAN generation of comics based on preliminary research. Then, 6 ComicGAN are evaluated, based on 3 meaningful experiments: (1) Initial models using AttnGAN to create comics from text. (2) ComicGAN trained on comic descriptions. (3) The ComicGAN created from training data of comic dialogue only. Models for ComicGAN used hyper-parameters based on that of the CUB bird model of AttnGAN, with slight changes to batch size. All parameters can be found in the configuration files in the source code.

A multi-label conditional GAN could be created from a modified version of AttnGAN. While AttnGAN can produce high quality images from input text, it requires a large volume of detailed descriptions corresponding to images as training data. Multi-labeled image datasets are more commonly available and are easier to create by virtue of being simpler. AttnGAN uses a text-encoder (Recurrent Neural Network) to encode the input text as a feature vector. By simplifying the architecture of AttnGAN, label vectors could be passed in as input instead of text, foregoing the need for a text-encoder. Furthermore, the image-encoder could be simplified to a multi-label classifier, by reducing the output dimension of the CNN to that of the number of input classes with a linear layer. The DAMSM would compare the input label vector to the label vector predicted for the generated image. The 17x17 high-level regional features produced by the image-encoder would also have to be reshaped to the number of number of classes, and then compared to single encodings of the classes present in description. This modification of the model would remove the need to create the text-encoder, and remove any chance of misrepresentation or misinterpretation of the contents of the image that could be caused by the text encoder. This could improve image quality in two ways: images would have better correspondence to the input labels; images could be of better general quality. These improvements would occur because the DAMSM component could learn the labels and features on the images better, and therefore provide a more accurate loss to better train the generators. This modification could not be created due to time limitations of this research, but is proposed as future work.

Other improvements include: adapting StoryGAN (?) to the task of generating an entire comic strip, composed of multiple related panels that follow one after another; Further evaluation should include detailed analysis of various comic datasets, with multiple detailed text descriptions per image; Moreover, richer, more detailed text descriptions are necessary to fully test the capabilities of ComicGAN. Due to the lack of annotated comic datasets, the models created performed very similarly to a multi-label GAN, which is not ideal for the technology in its’ current state.