Explicit Image Caption Editing

Criteria. Based on the task definition of ECE and essential requirements of each ECE instance, each reference caption (Ref-Cap) and its corresponding ground-truth caption (GT-Cap) should be selected reasonably for each image. We argue that there are several criteria in developing high-quality ECE datasets:

Visual Token Representations. For the given image, we first generate a set of image region features by extracting proposals and their corresponding visual features from a pre-trained object detector. We also encode the spatial location features of each proposal into a 5-d vector (normalized top-left and bottom-right coordinates, and fraction of the region area covered). A visual token feature is the sum of a region proposal feature and its spatial location feature. In addition, a special [IMG] token is placed at the beginning of the visual token sequence to represent the entire image. The token feature of [IMG] is the mean-pooled visual feature with a spatial encoding corresponding to the entire image.

Textual Token Representations. For the given reference caption, we first convert it into a sequence of tokens by tokenization [9]. Then, we put special [CLS] and [SEP] tokens at the start and end of textual token sequence, respectively. Meanwhile, for Inserter, another token [MASK] is used to indicate the position for new words adding. Same as [23], a textual token representation is the sum of token-specific learned embedding [44], position encoding, and segment encoding.

Multimodal Input Token Sequence. Given the image and reference caption, we first encoder them into a sequence of visual tokens $\{v_{1},\dots,v_{K}\}$ and textual tokens {$w_{1},\dots,w_{L}\}$, respectively. $K$ and $L$ is the number of visual and textual tokens, respectively. Then, the input token sequence for the three modules is $\{[\texttt{IMG}],v_{1},\dots,v_{K},[\texttt{CLS}],w_{1},\dots,w_{L},[\texttt{SEP}]\}$. The output representations for the visual and textual tokens are $\{h_{v_{1}},\dots,h_{v_{K}}\}$ and $\{h_{w_{1}},\dots,h_{w_{L}}\}$, respectively.

Tagger${}_{\text{del}}$ & Tagger${}_{\text{add}}$ Modules. As shown in Fig. 4, given the visual-textual token sequence, Tagger${}_{\text{del}}$ and Tagger${}_{\text{add}}$ tag each textual token with a specific edit operation $z$. For each textual token, both Tagger${}_{\text{del}}$ and Tagger${}_{\text{add}}$ conduct a binary classification, i.e., $z\in\{\texttt{KEEP},\texttt{DELETE}\}$ for Tagger${}_{\text{del}}$ and $z\in\{\texttt{KEEP},\texttt{ADD}\}$ for Tagger${}_{\text{add}}$. We pass the final representation of each textual token $\{h_{w_{1}},h_{w_{2}},\dots,h_{w_{L}}\}$ into a two-layer MLP to make the binary prediction, i.e., $z_{w_{i}}=\mathop{\arg\max}f(h_{w_{i}})$. Thus, the entire output of Tagger${}_{\text{del}}$ and Tagger${}_{\text{add}}$ is a sequence of edit operations corresponding to the sequence of input tokens, represented as $\{z_{w_{1}},z_{w_{2}},\dots,z_{w_{L}}\}$. The output textual token sequence can be translated from the input textual token sequence and predicted edit operations.

Inserter Module. As shown in Fig. 4, the input tokens fed into the Inserter is a sequence of tokens including the word tokens and the [MASK] tokens, which is constructed from the Tagger${}_{\text{add}}$ module. Given the image and the input tokens, the Inserter finishes the insertion by predicting the specific word from the vocabulary for each [MASK] token based on the observed tokens and visual information. Specifically, we pass the final representation of each [MASK] token $h_{w_{mask}}$ into a linear layer, mapping it to a distribution over the vocabulary. Lastly, all [MASK] tokens can be replaced with the predicted word, and the output textual token sequence can be formed with the rest word tokens for following the procedures.

Multi-Rounds Editing. As the specific editing process shown in Fig. 4, TIger resembles the way that humans might perform caption editing, i.e., considering what to keep, where to add, and what to add. By tracing these edit operations, the whole editing process is explainable and efficient. Meanwhile, since Tagger${}_{\text{add}}$ only adds one new word after each input word once a time, there might not be enough details if we only apply Tagger${}_{\text{add}}$ once. Thanks to this modular design, we can seamlessly use Tagger${}_{\text{add}}$ and Inserter iteratively for multi-rounds to guarantee enough details. Instead, if we make the Tagger${}_{\text{add}}$ can add more than one word once a time, it also needs to predict the number of new words to add at the same time. Meanwhile, the Inserter needs to predict words for multiple [MASK] tokens that may be placed consecutively. This significantly increases the difficulty of training, and empirically this single-round solution gets worse results.

Evaluation Datasets and Metrics. We evaluated our TIger on both COCO-EE and Flickr30K-EE datasets (cf. Sec. 3.2). For the caption quality evaluation, we followed existing caption generation works, and used four prevalent evaluation metrics: BLEU-N (B-N) (1-to 4-grams) [31], ROUGE-L (R) [20], CIDEr-D (C) [41] and SPICE (S) [2]. Particularly, we evaluated generated captions against its single ground-truth caption. Meanwhile, to evaluate the explicit editing efficiency of editing, we propose two supplementary metrics: Editing Steps (ES), and Gains Per Step (GPS). ES is the total number of meaningful editing steps, and GPS is the average performance gains per meaningful editing step, i.e., we hope ECE models realize the most performance gains with the least number of meaningful editing steps. In this paper, since all baselines apply the same set of edit operations (i.e., KEEP, DELETE, and ADD), we regard the sum of DELETE and ADD operations as ES. Meanwhile, since CIDEr-D is regarded as the most important metric for caption evaluation as to its high agreements with humans, we use the improvements of CIDEr-D score to calculate GPS, denoted as GPS(C).

Baselines. We compared our TIger against state-of-the-art image caption editing models. Specifically, we compared three strong implicit caption editing models: UpDn-E [3], MN [37], and ETN [38]. They are all built on top of the widely-used UpDn architecture [3], and propose some extra modules to encode the reference caption. Meanwhile, for more complete comparisons, we further extended three text explicit editing models (EditNTS [10], LaserTagger [27], and Felix [26]) into ECE, denoted as V-EditNTS, V-LaserTagger, and V-Felix, respectively. For all these three models, their basic editing operations are KEEP, DELETE and ADD. Specifically, V-EditNTS predicts the edit operation sequence iteratively by an LSTM. V-LaserTagger and V-Felix are one-round Transformer-based editing models, which directly predict multiple ADD operations simultaneously. More details about these baselines are left in the appendix.

Implementation Details. The implementation details are left in appendix.

Settings. We evaluated TIger on COCO-EE and Flickr30K-EE by comparing with state-of-the-art methods. Since our target is to propose the ECE task and the first ECE model, we first compared TIger with simple ECE baselines which were extended by text explicit editing models (V-EditNTS, V-Felix, and V-LaserTagger). For completeness, we also reported the results of all existing ICE models (UpDn-E, ETN, and MN). Since all implicit models are built on top of the widely-used UpDn architecture, we only reported the results of the UpDn captioning model rather than all other SOTA captioning models (e.g., VLP[49]) as they actually don’t belong to the caption editing task. Since V-Felix and V-LaserTagger are also Transformer-based architectures, we used the same ViLBERT pretrained weights as TIger. For the other baselines, we converted all the words in each dataset to lower cases and built their respective vocabulary. All baselines were trained with XE loss. Since implicit models do not explicitly predict edit operations, we suppose they delete all the words in the reference caption first and add new words from scratch to output caption, i.e., ES is calculated as the sum of words in reference and output caption. Meanwhile, we mainly focused on the efficiency evaluation of ECE models, so we have used gray font for efficiency evaluation of ICE methods. Results on COCO-EE and Flickr30K-EE are reported in Table 2 and Table 3, respectively.

Results on COCO-EE. From Table 2, we can observe: 1) For the quality evaluation, our model achieves the largest performance gains on all metrics (e.g., 194.8 vs. 190.5 in ETN on CIDEr-D). 2) For efficiency evaluation, SOTA implicit models always outperform their explicit counterparts, but they require more editing steps. Instead, our model achieves the best GPS(C) score by predicting more ADD operations, instead of simply deleting or keeping the words in the reference captions. It also shows our ability to detect and fix detailed errors.

Results on Flickr30K-EE. From Table 3, we can observe: 1) For the quality evaluation, similar with COCO-EE, our model achieves the largest performance gains on all metrics (e.g., 148.3 vs. 143.3 in ETN on CIDEr-D). 2) For efficiency evaluation, our model achieves the best GPS(C) score (e.g., 8.58 vs. 6.90 in V-EditNTS). Compared to the weaknesses of implicit models (need more editing steps) and explicit models (marginal performance gains), our model achieves a decent balance between performance gains and editing steps, i.e., we improved the quality of reference captions with quite a few meaningful editing steps.

Influence of Weighted XE Loss. As mentioned in Sec. 4.3, we used weighted XE loss for training. To explore the influence of different loss weights, we first run ablations by setting different loss weight ratios $\lambda\in\{1.0,1.2,1.5,2.0\}$ on both Tagger${}_{\text{del}}$ and Tagger${}_{\text{add}}$. Results are reported in Table 5. Then, we explored the influence of weighted XE loss to a single Tagger module, i.e., we run ablations by setting one of the Tagger with $\lambda>1.0$, and the other with $\lambda=1.0$. The results are reported in Table 5. Note that all Inserters were trained with $\lambda=1.0$.

Results. From Table. 5, we have several observations: 1) For both the COCO-EE and Flickr30K-EE, TIger with weighted XE loss training always gets better performance than the baseline ($\lambda=1.0$). 2) The model trained with $\lambda=1.5$ gets the best performance, i.e., it boosts the CIDEr-D score from 190.0 to 194.8 for COCO-EE and from 137.0 to 148.3 for Flickr30K-EE. This demonstrates the effectiveness of paying more attention to predicting the KEEP operation. We then used $\lambda=1.5$ to train TIger in all experiments. From Table 5, we can observe that: 1) TIger with only one of the Tagger modules trained with $\lambda=1.5$ alone can still achieve better performance than baseline. 2) The weighted XE loss has more impact on Tagger${}_{\text{del}}$ than Tagger${}_{\text{add}}$. The possible reason is that TIger only applies Tagger${}_{\text{del}}$ once, which determines the basic caption for further adding.

Different Editing Rounds. Since TIger iteratively use Tagger${}_{\text{add}}$ and Inserter multiple rounds to add words, we run ablations to analyse the effect of different edit rounds. The maximum number of editing rounds was set to 5.

Results. From Table 6, we can observe that: 1) For COCO-EE, the performance of TIger keeps improving in the first 3 editing rounds. Then, the quality evaluation metrics reach the best scores and keep unchanged or even slightly drop with more editing rounds. For example, BLEU-1 keeps increasing with more editing rounds, CIDEr-D reaches the best score 194.8 in the 4-th round and drops to 194.6 in the 5-th round. Since most metrics reach their best scores in the 4-th round, considering the trade-off between model performance and editing efficiency, we used 4 editing rounds for the COCO-EE. 2) For Flickr30K-EE, the performance of TIger keeps improving in the first 3 editing rounds. Most quality evaluation metrics reach the best score in the 3-rd round, and then keep unchanged (14.9 for BLEU-4 and 148.3 for CIDEr-D) or drop slightly (SPICE) with more editing rounds. Thus, we used 3 editing rounds for the Flickr30K-EE.

Influence of the pre-trained Weights. Since we took advantage of the pre-trained weights to train TIger, we further ran ablations to examine the influence of the pre-trained ViLBERT weights. The results are reported in Table 7.

Results. From Table 7, we can observe that for both datasets, as the first ECE model, both TIger models with and w/o pre-trained weights all outperform other ECE baselines with same pre-trained weights in both quality and efficiency evaluation. Meanwhile, TIger trained with pretraind weight achieves better performance than the model trained from scratch. For example, in CIDEr-D, the pre-trained weights improve score from 178.1 to 194.8 for COCO-EE.

Results. As shown in Tabel. 8, we can observe that: 1) For COCO-EE, our proposed TIger can significantly improve the quality of all the captions generated by ECE baselines (e.g., CIDEr-D score from 149.0 to 172.7 for V-EditNTS). 2) For Flickr30K-EE, the average improvements are still remarkable (e.g., 135.8 vs. 129.1 in V-EditNTS on CIDEr-D score). This also demonstrates the robustness of TIger when given different reference captions (e.g., these ECE baselines generated captions may erroneously delete or preserve some words).