Zero-Shot Warning Generation for Misinformative Multimodal Content

Liu et al. [12] devised a system that leverages both domain generalization and domain adaptation techniques to mitigate discrepancies between hidden domains and reduce the modality gap, respectively. Shalabi et al. [24] addressed OOC detection by fine-tuning MiniGPT-4’s alignment layer but without message generation and confidence scoring. Zhang et al. [31] introduced a novel approach to reasoning over (image, caption) pairs. Instead of directly learning patterns from the data distributions, as Liu et al. did [11], they extract abstract meaning representation graphs from the captions and use them to generate queries for a VLM. This sophisticated approach enables a nuanced consistency check between the visual features of the image and the extracted features of the caption, but with limited room for explainability given by analysis of the generated queries and respective answers of the VLM. The work by Zhang et al. [31] made it possible to understand the main limitation of closed-domain approaches to this task: evaluation of the veracity of an (image, caption) pair is sometimes challenging because the image may not depict all the statements that can be extracted from the caption.

Popat et al. [19] introduced the concept of detecting textual misinformation using external evidence; however, in this study evidence is not integrated simultaneously. Interpretability of the predictions is provided in the form of attention weights over the words of the analyzed document. Abdelnabi et al. [1] extended the concept of leveraging external knowledge for fact-checking to a multimodal (image, text) domain while also computing the aggregated consistencies considering all evidence at the same time for each of the two modalities. A serious limitation of this approach is the provision of explanations solely in the form of attention scores signaling the most relevant evidence for the purpose of prediction along with limited debunking capabilities.

Yao et al. [29] overcome this limitation by introducing an end-to-end pipeline consisting of evidence retrieval, claim verification, and explanation generation, using a dataset built from fact-checking websites with annotated evidence. A drawback of their approach is the utilization of a large language model (LLM) to summarize the evidence content, potentially overlooking important clues observable in the image. Two parallel works, ESCNet [30] by Zhang et al. and SNIFFER [20] by Qi et al., also explore this area. ESCNet lacks explanation generation, while SNIFFER employs a commercial VLM (ChatGPT-4) for generating explanations.

Our work extends the principles outlined above [1, 29] by introducing a module that reasons upon the source pages of the evidence and the generation of a contextualizing explanation as a zero-shot learning task of a VLM.

In order to develop our contextualizing tool for the purpose of warning generation, we used the NewsCLIPpings dataset [13], which is a synthetic dataset made by Luo et al. built using the VisualNews [11] corpus, which comprises news articles from four prominent newspaper websites: The Guardian, BBC, USA Today, and The Washington Post.Given an (image, caption) pair, images of Visual News[13] are retrieved and substituted to the original image to create falsified samples. Specifically, we used the merge-balanced subset, which consists of 71,072 training, 7,024 validation, and 7,264 test examples.

The retrieved evidence were provided by Abdelnabi et al. [1]. Given a pair $(I^{q},C^{q})$, visual evidence $I^{e}$ is obtained through means of direct search using $C^{q}$ as query; Textual evidence $C^{e}$ is obtained through means of inverse search, these are the result of searching for textual content (ad hoc scraped caption of images in web pages or title of those web pages) using $I^{q}$ as query. Complementary informations, such as labels regarding the images $L^{q}$ and $L^{e}$, are obtained through means of Google Cloud Vision API¹¹1https://cloud.google.com/vision/docs/labels. The authors of [1] provided the URLs to the source pages of both kind of evidence, which were downloaded for the purpose of this research.

Each data point is described formally as

• •

  query image $I^{q}$;

• •

  query caption $C^{q}$;

• •

  visual evidence retrieved using $C^{q}$ to query the web: $I^{e}=[I^{e}_{1},...,I^{e}_{n}]$;

• •

  labels obtained using Google Cloud Vision API:
  $L^{q}$ are labels of $I^{q}$, and $L^{e}$ are labels of $I^{e}$;

• •

  textual evidence retrieved using $I^{q}$ to query the web (inverse search):
  $C^{e}=[C^{e}_{1},...,C^{e}_{m}]$;

• •

  source pages $P^{e}=\operatorname{source}(I^{e})\cup\operatorname{source}(C^{e})$.

Images are represented either using ViT [28] trained on ImageNet [4] or DINOv2 [16] embeddings for visual reasoning. These frozen visual transformers offer improved preservation of spatial information compared with ResNets [8, 22], which is crucial in our context. We aim to maintain structural information to achieve high similarity between images and their cropped/resized counterparts. Images are represented as vectors $I^{q},I^{e}\in\mathbb{R}^{768}$. The consistency score is computed as the cosine similarity:

$\operatorname{cossim}(A,B)=\frac{\mathbf{A}\cdot\mathbf{B}}{\|\mathbf{A}\|\|\mathbf{B}\|}$

$S_{\text{images}}=\operatorname{cossim}(I^{q},I^{e}_{\text{proj}})$

$\quad I^{e}_{proj}=\operatorname{MultiHead}(I^{q},I^{e},I^{e})$,

where $\operatorname{MultiHead}$ is the multi-head attention mechanism, as defined by Vaswani et al. [26].

We use frozen pre-trained sentence transformers, such as Sentence-BERT [23] and Mini-LM [27], for textual reasoning. The decision to use them was influenced by Nikolaev and Padó’s research [15], which showed that sentence transformers prioritize capturing semantic essence over grammatical functions or background details. This leads to increased cosine similarity between sentences with shared salient elements, like subjects or predicates. Labels and captions are represented as vectors $L^{q},L^{e},C^{q},C^{e}\in\mathbb{R}^{\text{ST-dim}}$, where ST-dim is $768$ for Sentence-BERT and $384$ otherwise. The consistency scores are computed as

$S_{\text{labels}}=\operatorname{cossim}(L^{q},L^{e}_{\text{proj}})$

$\quad L^{e}_{\text{proj}}=\operatorname{MultiHead}(L^{q},L^{e},L^{e})$

$S_{\text{cpt}}=\operatorname{cossim}(C^{q},C^{e}_{\text{proj}})$

$\quad C^{e}_{\text{proj}}=\operatorname{MultiHead}(C^{q},C^{e},C^{e})$.

In this reasoning module, source pages are treated as text and represented as $P^{e}_{l}\in\mathbb{R}^{\text{ST-dim}}$. In particular, $P^{e}_{l}$ is computed as the mean of the embeddings of all sentences in the paragraphs on the $l-th$ source page, and $P^{e}_{l}$ is considered to be the embedding of the page itself. This module serves to identify the most important page for subsequent re-contextualization tasks and to compute an inter-agreement score between the retrieved documents by computing self-attention on page representations:

$S_{\text{pages}}=\operatorname{mean}(\operatorname{cossim}(P^{e},P^{e}_{\text{proj}}))$

$\quad P^{e}_{\text{proj}}=\operatorname{MultiHead}(P^{e},P^{e},P^{e})$.

We use late fusion [5, 18] as the architectural pattern to address the lack of compelling explanations for the assessments made by existing detection methods. Our decision to use this pattern stems from both the flexibility it offers in swapping embeddings and the impracticality of training the entire model end-to-end due to its large parameter count. To address the challenges of inter-modality reasoning, we integrate a multimodal consistency block that can be optimized alongside the rest of the model. To represent both $I^{q}$ and $C^{q}$ in the same latent space, we employ a VLM, chosen among two alternatives: CLIP [21] with embedding dimension $\text{MM-dim}=512$ and MiniGPT-4 [32] with $\text{MM-dim}=4096$. Each sample pair is represented as $Pair^{q}=\operatorname{VLM}(I^{q},C^{q})\in\mathbb{R}^{\text{MM-dim}}$. The image-caption consistency block is defined as

$S_{\text{logit}}=\operatorname{Linear}(\operatorname{ReLU}(\operatorname{Dropout}(\operatorname{Linear}(Pair^{q}))))$,

where the innermost linear layer projects $Pair^{q}$ into $\mathbb{R}^{256}$, the outermost linear layer projects it to $\mathbb{R}$, and the dropout probability is set to $0.2$.

The output of the five consistency blocks is a single value, resulting in a vector $V\in\mathbb{R}^{5}$, which is fed into a classification head to obtain a prediction $P\in[0,1]$ (threshold $th$ set to 0.5 or found by analyzing the equal error rate, a pair is falsified for $P\geq th$, pristine otherwise). A batch normalization layer [9] is added between $V$ and the linear classifier for faster convergence and higher accuracy. As the model is optimized for a binary classification task, binary cross-entropy loss is used:

$l(x,y)=\operatorname{mean}(\{l_{1},...,l_{N}\}^{T}),$

$\quad l_{n}=y_{n}\cdot\log x_{n}+(1-y_{n})\cdot\log(1-x_{n})$,

where $x_{n}$ and $y_{n}$ are the predicted and ground truth labels, respectively, in a mini-batch of size $N=64$.

The last step of our pipeline consists of generating either contextual explanations for pristine pairs $(I^{q},C^{q})$, providing additional insights about the depicted entities, or warnings indicating why $(I^{q},C^{q})$ represent a case of image repurposing. Given the versatility of LLMs in solving zero-shot learning tasks and the recent release of GPT-4 [2], which extends these capabilities to a multimodal environment, we adopted a strategy for generating warnings without fine-tuning or reinforcement learning. Due to the unavailability of ground truth data for comparing explanations and optimizing the generation process, fine-tuning approaches are precluded by default.

Similarly, reinforcement learning through human feedback [17] presents challenges, including the need for a dedicated team to label examples and the potential persistence of model mistakes despite corrections. Reinforcement learning through AI feedback [10], which involves querying a larger model like GPT-4 to assess MiniGPT-4’s outputs, also faces limitations. These include the constrained context window and the likelihood that all retrieved evidence may not contribute effectively to warning generation. Moreover, using a VLM to rectify errors made by another VLM inherently introduces comprehension discrepancies. Our approach to prompting was inspired by the work of Guo et al. [6], who evaluated VLMs trained using multimodal pretraining similar to the Flamingo VLM [3], and is akin to the training strategy employed for MiniGPT-4.

MiniGPT-4 is the tool used for contextualization and is guided by the following prompt. Variables that depend on the values obtained by the consistency network are displayed between square brackets.

“You are a tool for out-of-context detection, your task is to give reasons why the submitted image and the caption below are in the same context or not. Submitted Image: [$I^{q}$]. Caption: [$C^{q}$]. The likelihood of the submitted image and the above caption being in the same context is [$P_{\text{class}}$], thus the pair is [Falsified if $P_{\text{class}}\geq th$ else Pristine]”.

Additionally, the title and the first few sentences (up to 400 characters) of the 4 web pages with the highest attention score from each of the attention-based blocks are included in prompt, together with the retrieval modality of the page, e.g.:

“An evidence retrieved using the caption to query the web, obtained because it contains an image with high similarity with the submitted image has title [Title] and content of the paragraphs [Content]”.

If the same source page is selected by multiple blocks, its title and content is added once to the prompt. For example, this redundancy could happen because a retrieved image and a retrieved caption belong to the same web page and both are ranked first by the multi-head attention mechanism (with reference to Fig.  2, $\operatorname{source}(I^{e}_{\text{first}})=\operatorname{source}(C^{e}_{\text{first}})$).

All our experiments were carried out using either one or three NVIDIA A100 40-GB GPUs. Experiments on multiple GPUs were performed employing the distributed data parallel strategy. Early stopping was triggered if the validation loss stopped decreasing for 5 consecutive epochs. The designated mini-batch size was 64. All our experiments employed a cyclic learning rate scheduler with initial learning rate equal to $9\cdot 10^{-5}$ and maximum learning rate equal to $5\cdot 10^{-4}$. Following the principle of scaling the learning rate with respect to the effective batch size, the learning rate was rescaled by multiplying both values by $1/\sqrt{3}$ when training on a single GPU.

NewsCLIPpings [13] established a baseline by fine-tuning CLIP (ViT/B-32)[21], achieving 66.1% accuracy. Shalabi et al.[24] reached state-of-the-art performance for closed-domain approaches by fine-tuning MiniGPT-4 [32], with 80.0% accuracy. Yao et al.[29] (End-to-end in Table1) employed text and image retrieval modules to select evidence, though these only consider the textual claim from the input, as their dataset relies on textual claims from fact-checking websites. Despite this limitation, their approach achieved 83.3% accuracy on NewsCLIPpings. Abdelnabi et al. [1] achieved 84.7% accuracy using a 20.92M-parameter consistency-checking model trainable in 30 hours. Notably, CLIP is preliminarily fine-tuned (excluded from parameter and time counts).

Our lightweight model, employing frozen MiniGPT-4 as VLM, achieves 84.8% of accuracy with 5.2 million parameters, it is trainable in 3 hours and 38 minutes on a single GPU, while it only requires 13 minutes on three GPUs. It is highlighted in blue in table 2. It does not require any preliminary fine-tuning, there is no additional computation overhead due to comparison between entities and input caption, it leverages the analysis of source pages instead of exploiting domain representation and represents labels as text, comparing the labels of the query image with the labels of the visual evidence. Our full-scale model achieves 87.04% accuracy using the rescaled learning rate technique, and 86.70% using the standard learning rate. This model is highlighted in green in table 2. It counts 158 million parameters, of which 151 million belong to CLIP, which is optimized jointly with the rest of the architecture. The training time on a single GPU is 7 hours and 32 minutes, while on three GPUs it narrows to 27 minutes.

The parallel works, ESCNet [30] and SNIFFER [20], demonstrate slightly higher performance than our approach (87.9% and 88.4%, respectively). However, ESCNet lacks explanation generation, while SNIFFER requires additional fine-tuning on the Q-Former in 16 hours. Furthermore, SNIFFER employs a different backbone (InstructBLIP), complicating a fair comparison.

The standard (std) and alternative (alt) sentence transformers are, respectively, Sentence-BERT²²2https://huggingface.co/sentence-transformers/paraphrase-mpnet-base-v2[23] and a fine-tuned version of MiniLM³³3https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2[27]. The key differences between them, in the context of our research, are that the former outputs embeddings that are twice as large as those output by the latter (768 against 384) and that the latter is faster at inference time and also more task specific because its training recipe is based on distillation and it is fine-tuned on sentences and small paragraphs. The std and alt vision transformers are ViT⁴⁴4https://huggingface.co/google/vit-base-patch16-224-in21k[28] trained on ImageNet-21k and DINOv2⁵⁵5https://huggingface.co/docs/transformers/en/model_doc/dinov2[16]. These transformers share the same embedding dimension (768), and, analogous to the two sentence transformers, the latter has a different training recipe, which includes self-supervised learning on a larger corpus of uncurated data, enabling the extraction of visual features that work across image distributions and tasks. Our experiments demonstrated that the std sentence transformer and alt vision transformer was the best combination, regardless of the choice of multimodal model, when considering all four design patterns involving the removal of attention blocks. In fact, model version 17 had accuracy scores comparable to those of version 21 and a slightly lower ROC AUC, while versions 18, 19, and 20, compared with 22, 23, and 24, had lower accuracies (th:05) and lower ROC AUC. Nevertheless, we consider version 17 to be the best model version using the MiniGPT-4 model as it had the highest accuracy and fewer parameters while using all our attention blocks.

With the aim of understanding the true utility of labels, we treat them as sentences and dedicate a separate attention block to evaluate their consistency. With the CLIP backbone, model pair (1, 2) had comparable performance with and without the label-label attention block, whereas for pairs (5, 6) and (9, 10), dropping this block of redundant information resulted in slightly better performance. For the MiniGPT-4 versions, dropping this block meant removing a non-negligible number of parameters. The resulting decrease in performance observable in model pairs (13, 14), (17, 18), and (21, 22) supports our hypothesis that, in this regime, additional information about labels is helpful in achieving better performances. We rehydrated the dataset by retrieving the source pages of each evidence. The embeddings of source pages $P^{e}$ were then used in the self-attention layer with the aim of extracting the most relevant web page for the purpose of prediction, potentially enhancing explainability. Although dropping the page-page attention block tended to reduce performance, model version 23 performed optimally at th=0.5. Nonetheless, the other metrics suggest that versions 21 and 22, which use this block, performed better.

Since the ground truth was unavailable for performance evaluation, unlike with the approach taken by Yao et al. [29], we randomly selected 100 samples from the test set. These samples were evaluated by 20 individuals, each assessing 5 samples. Two evaluation metrics were used in accordance with the work of van der Lee et al. [25]: ‘Informativeness’ and ‘Overall Quality’, both ranging from 0 to 5 with steps of 1. The ‘Informativeness’ score is defined as the relevance and correctness of the output relative to the input specification. Evaluators considered both the generated warnings and the associated links when assigning the Informativeness score. The ‘Overall Quality’ metric represents a judgment regarding the system’s performance across the five samples observed by each evaluator. The average ‘Informativeness’ score was 3.5 out of 5, and the average Overall Quality was 4 out of 5. These high scores can be attributed to the relatively straightforward contextualization of truthful examples when the correct evidence is retrieved, while various factors can render a warning unreliable in the case of falsified examples, leading to lower scores, especially if valuable evidence is not retrieved.

In Table 3, we present samples used in the human evaluation. The “Generated Warning” and “Retrieved Links” columns display the output of the system. In the first two samples, coherent explanations and links are displayed. In the third sample, although the prediction was correct, the VLM failed to distinguish events even when different dates were provided by the caption and the evidence. Additionally, the system focused on why certain evidence had been retrieved instead of generating an explanation based on the content of the source pages. The third sample was also correctly predicted; however, $C^{q}$ is missing from the evidence. Consequently, the explanation provided is inconsistent with the source page of the caption, as evident from the presence of noisy evidence. Specifically, the warning explains that the reported victims in $C^{q}$ were victims of war, whereas the source page indicates that they were victims of child exploitation. Similarly, in the last sample, $I^{q}$ is missing from the retrieved evidence. Consequently, the pair is misclassified as pristine even though the child in $I^{q}$ was not affected by Down syndrome. However, the evidence contains a picture of the actual subject of $C^{q}$, making it simple for a human to distinguish this sample as OOC regardless of the generated explanation.