Prompt-based Learning for Unpaired Image Captioning

The target of PL-UIC is to train an unpaired image captioning model guided by prompts. As shown in Fig. 3, the overall framework consists of three parts, i.e., the semantic prompt extraction module, the adversarial learning based UIC model, and the metric prompt based generator refining module. Specifically, the semantic prompt extraction module contains the CLIP model and a feed-forward layer to provide the semantic prompt for each image in the image datasets. Then, the semantic prompt guided UIC model is trained to generate captions by adopting unpaired image-text samples. Later, the caption generator refining module is trained using pseudo labels. These pseudo labels are selected by the designed metric prompt, which can measure the correlations of image-text pairs. We combine the designed two prompt-related modules into the common adversarial learning UIC framework, which achieves promising results on two widely used captioning datasets.

Unpaired image captioning is to describe images without using any aligned vision-text pairs. The image dataset with $N_{i}$ images is represented by $\mathcal{I}=\{\textit{{I}}_{1},\textit{ {I}}_{2},...,\textit{{I}}_{N_{i}}\}$, the unpaired language dataset is represented as $\mathcal{S}=\{\textit{{S}}_{1},\textit{{S}}_{2},...,\textit{{S}}_{N_{s}}\}$ with $N_{s}$ sentences, and the captions generated by the UIC model are denoted as $\mathcal{C}=\{\textit{{C}}_{1},\textit{{C}}_{2},...,\textit{{C}}_{N_{c}}\}$, where $N_{c}$ means the number of generated captions. For simplicity, we utilize I, C, and S to represent an image, a virtual generated caption, and a real sentence, respectively.

The common pipeline of UIC is illustrated in Fig. 4, where a CNN-LSTM network is adopted for the encoder-decoder framework. Firstly, a CNN encoder is utilized to extract the image features $\textit{{f}}_{\textbf{I}}$. Then, an LSTM decoder network $LSTM_{g}$ is adopted to transform the image features into texts $\textit{{C}}=\{\textit{{c}}_{1},\textit{{c}}_{2},...,\textit{{c}}_{n_{c}}\}$, where $n_{c}$ is the number of words in one caption. And this procedure can be written as:

where $\textit{{x}}_{t}$, $\textit{{c}}_{t}$, $\textit{{o}}_{t}$, and $\textit{{h}}_{t}$ are the input of the $LSTM_{g}$ decoder layer, a one-hot vector representation of the outputted word, the probability of every word in the dictionary, and the hidden state of the LSTM at the $t$-th time step, separately. $\textit{ {x}}_{0}$ is the initial input of the decoder, and one feed-forward layer $FC$ is utilized to adjust the image features. $\textit{ {h}}_{0}$ is a zero vector for the initial hidden states of the decoder. $\textit{{W}}_{e}$ is for word embedding. Besides, an LSTM discriminator $LSTM_{d}$ is adopted to differentiate a real sentence S and a virtual sentence C, which is utilized to make the generated captions as real sentence as possible. Formally,

where $\textit{q}_{t}$ represents the probability that the outputted sequential words $\{\textit{{c}}_{1},...,\textit{ {c}}_{t}\}$ of one caption C or $\{\textit{{s}}_{1},...,\textit{{s}}_{t}\}$ from a real sentence S are regarded as a real sentence S.

Although the adversarial learning based UIC models work well in some scenarios, however, their overall performance is still limited due to the less cross-domain knowledge. We think this situation can be alleviated by introducing vision-language prior knowledge.

The motivation of the semantic prompt and metric prompt is the vision-language prior knowledge fully reflected in the feature representation of the VL-PTMs. For example, the CLIP model used in this paper performs well in zero/few-shot learning since its abundant language-aware visual features. The CLIP model includes one visual encoder and one sentence encoder. The visual encoder is used to extract the deep image features, which can be implemented with ResNet-101 [53] or ViT [54]. The text encoder is used to learn the sentence features using a Transformer network [55]. Based on a large amount of image-text dataset (400 million) crawled from the Internet, the CLIP is optimized to align features of these two domains using contrastive learning [56]. Specifically, its target is to maximize the cosine similarity between the matched image-caption pairs and minimize the unmatched ones. Therefore, the cross-domain vision-language prior knowledge can be mastered by the CLIP.

To intuitively exhibit the prior knowledge of the CLIP model, two types of examples are visualized in Fig. 5. Fig. 5 (a) is the zero-shot image classification, which reflects the prior knowledge from the vision domain to the language domain. Fig. 5 (b) is about highlighting the most related regions in an image for the given texts without training, which illustrates the prior knowledge from the language domain to the vision domain.

The main target of this paper is to extract the learned knowledge from CLIP for the guidance of unpaired image captioning from three aspects, i.e., the visual encoder, the semantic prompt, and the metric prompt. The visual encoder of the CLIP model is adopted to extract the image features for UIC directly, where abundant language-aware image features are provided to guide the caption generation. The details of the two prompts are introduced in the following sections.

For unpaired image captioning, the performance of existing methods is restrained by the limited cross-domain knowledge. To alleviate the issue, a semantic prompt extraction module is designed to explore more cross-domain prior knowledge from the VL-PTMs. The prompt extracted from the module will be taken as guidance for the caption generation.

To obtain the semantic prompt $\hat{\textit{{P}}}$ for each image, the “SOS”, “CLS”, and “EOS” embeddings are concatenated at the start and end of the FC layer, which represents the start and end of the prompts, respectively. Moreover, the positional pointers of the CLIP model are added to the former embeddings, and the results $\hat{\textit{{P}}}$ are taken as the learned prompt for each image. Formally,

where $[\cdot]$ means the concatenation operation, and E represents various types of embeddings in CLIP. The $\textit{{E}}_{pos}$ indicates the positional embeddings in the CLIP model. And we utilize ${[V]_{1},[V]_{2},...,[V]_{n}}$ to represent a semantic prompt $\hat{\textit{{P}}}$, where $[V]_{1}$ denotes the features for a textual word. Note that only the image dataset of UIC is utilized for training this module, and the architecture of the trainable layer in the semantic prompt extraction module can be replaced by other networks for better performance, which we left in the future work.

After the training of semantic prompt-based UIC is done, we can get better results than the prompt-free UIC model. Due to the limitation of unsupervised learning of captioning generator, the unpaired image-text samples are still hard to optimize, which bring us sub-optimal performance only. Therefore, we are inspired to transform the unpaired image-text samples into paired ones using generated captions for the training images. Then, these data can be utilized as pseudo labels to train a fully-supervised caption generator.

Considering that the UIC model is still weak in the initial stage, indiscriminate leveraging these data may harm our model. In this section, a pseudo label filtering scheme is proposed to filter out the low-quality image-caption pairs and leave the high-quality ones, which enables us to polish our model in a more stable way. Based on the proposed scheme, the model refining and label filtering processes can be executed iteratively. The whole process is illustrated in Fig. 6. In the filtering process, the metric prompt is designed to select the high-matched image-caption pairs. Unlike the prior knowledge in the semantic prompt, the metric prompt contains different cross-domain knowledge that indicates whether an image and a caption are matched.

As illustrated in Fig. 2, the high-quality image-caption pair has a high metric value, and the low-quality image-caption pair has a low metric value. Suppose we have a metric gate $G$ with threshold $T$, if the metric is lower than the threshold $T$, the raw image caption $\{\textit{{I}},\textit{{C}}\}$ will be filtered out. Otherwise, the raw image caption pair will be preserved to train the semi-supervised caption generator.

Stage-I: The semantic prompt extraction module, i.e., a single FC layer embedded on the CLIP model, is firstly trained on an image dataset. As a result, the semantic prompt of each image will be obtained, which represents the contextual features of images and texts, and will be utilized as a part of the input of the UIC model.

Stage-II: Aided by the aforementioned semantic prompts, the UIC model is trained by the image features and semantic prompts, which is optimized by adversarial learning and visual concept alignments. The trained model can be adopted to generate the captions of images.

Stage-III: With the generated image-text pairs, the captioning model is re-trained to enhance the captioning performance. In this stage, metric prompt-based pseudo label filtering is implemented to select the high-quality image-text pairs as the pseudo labels. These labels can be utilized to train a better semi-supervised model. The filtering and re-training strategies loop multiple times to enhance the captioning performance.

A brief introduction to these datasets is given below. 1). COCO [57] is one of the popular benchmarks of the image captioning tasks, which contains 123,287 images. The common Karpathy split [60] is utilized in the experiments, with 113,287 training images, 5,000 validation images, and 5,000 testing images. 2). Flickr30K dataset consists of 31,783 images with 29,783 training images, 1,000 validation images, and 1,000 testing images, which is the common split [58] for the experiments on image captioning. 3). COCO caption dataset is annotated manually. Each image is annotated with 5 captions. For training, the COCO images and captions are utilized in an unpaired way. A word vocabulary is built by words presented no less than 4 times in captions of the training images. 4). Shutterstock sentence dataset is crawled from the Shutterstock website by [21], which contains 2,282,444 different image captions. 5). Conceptual sentences [59] are automatically collected from webs, which include 3.3 million image-caption pairs. The captions are utilized as a sentence dataset.

In UIC, we utilize the pretrained ViT-b16 CLIP model as the image encoder. The object concepts are extracted by Faster-RCNN [66] related object detection model. The LSTM with 512 hidden states is utilized for the sentence decoder and discriminator. The learning rate of the caption generator is set as 0.001 at the training Stage II in Section III-F. At the training Stage III, the learning rate is set as 0.00001. The architecture of the semi-supervised image captioning model is the same as the generator of the UIC model. The whole PL-UIC model is trained on 2 V100 GPUs.

For the semantic prompt extraction and pseudo label filtering, the ViT-b16 visual encoder [23] is utilized in the CLIP model. For pseudo label filtering of COCO image datasets, the threshold of metric prompt is set as 30. For the Flickr30k image dataset, the threshold of the metric prompt is set as 26 due to the smaller scale of the image dataset.

In this section, two kinds of experimental comparisons are carried out to validate the effectiveness of the proposed approach. The first one is to compare the PL-UIC with other approaches on the COCO dataset using unpaired image and caption samples. The second setting is to compare PL-UIC with algorithms adopting independent datasets, i.e., COCO images with Shuttershock sentences, and Flickr30k images with Conceptual captions. We compare the PL-UIC with the related methods Feat2sen [21], Recons-Align of [21], and WS-UIC [43]. Feat2sen relies on pseudo labels, outputted by an object-to-sentence model [21], to train a fully-supervised caption generator, where only sparse knowledge of the text domain and the image domain is concerned. Recons-Align depends on adversarial learning and the visual concept reward, both of which cannot be implemented to explore the contextual vision-language information thoroughly. WS-UIC relies on one more unrecognized object loss to improve the alignment between the objects and images, which is still weak to explore plentiful of cross-domain knowledge. As a result, the experimental performance of these methods is severely constrained. Different from all these works, we adopt the semantic prompt and metric prompt of each image, containing full of vision-language prior knowledge, as additional guidance for caption generation.

For the first setting, we report the experimental results of ours and the related UIC models in Table I. * means the re-trained version of methods with the same powerful image encoder of PL-UIC. Besides the method Feat2sen, Recons-Align, and WS-UIC, we compare with the PL-UIC (finetuning SP) (3) and PL-UIC (3). The PL-UIC (finetuning SP) (3) means training PL-UIC with finetuned semantic prompts and 3 iterations of the generator refining, where the semantic prompt are extracted by training the whole model of the semantic prompt extraction instead of one feed-forward layer. PL-UIC (3) means that the experiment is carried out by 3 iterations of the caption generator refining. Generally speaking, the experimental results of the proposed PL-UIC achieve the best captioning performance over all five evaluation metrics. For example, the C value of PL-UIC is 6% higher than the C value of Recons-Align [14]. The results basically demonstrate that the extracted prompts, with plentiful contextual vision-language information, can provide efficient guidance for captioning. And the iterative pseudo label filtering scheme can provide high-quality labels for generator refining. Compared the PL-UIC (finetuning SP) (3) with PL-UIC (3), the higher experimental results of PL-UIC (3) verify that the training of only one feed-forward layer of the designed semantic prompt extraction method is more effective than training all parameters of the whole module.

For the second setting, we compare the proposed approach with related methods by utilizing independent datasets. From Table II, it is easy to find that our proposed PL-UIC outperforms all the compared methods by a significant margin in the experiments with COCO images and Shutterstock sentences. More in detail, the C value of PL-UIC is 45.8. Meanwhile, the value of Recons-Align is only 31.3, which is 14.5 lower than PL-UIC. For the experiments with Flickr30k images and Conceptual sentences, the overall captioning performance of PL-UIC exceeds all the compared methods, although it has weak performance on metrics M and C. These experimental results fully demonstrate that our PL-UIC model has the strongest captioning ability in the independent dataset settings since the vision-language prior knowledge learned from VL-PTMs.

To compare the captioning ability of all these methods more intuitively, two images with the corresponding captions generated by these methods are elaborated in Fig. 8. From the figure, we can observe that the captions outputted by the proposed PL-UIC are better than the other compared methods. Take the first image as an example, the Feat2sen* method outputs "a young boy with a drink in front of a person holding a snack in his hand.”, where only the concept "a young boy" is accurate. The method Recons-Align* and WS-UIC* generate incorrect concept “donut” and "holding", respectively. PL-UIC (finetuning SP) (3) generates a totally wrong caption “a person riding a bike down a street next to a building”. The captions outputted by the proposed method PL-UIC (3) and PL-UIC are semantically correct with "a little boy", "eating", and "a piece of cake", which contain almost all the important concepts as in the ground-truth captions. Visualizing these captions intuitively demonstrates the superiority of the proposed PL-UIC with plentiful cross-domain prior knowledge.