GIST: Generating Image-Specific Text for Fine-grained Object Classification

Models pretrained on large-scale multimodal datasets have been shown to provide data representations that are transferable to many tasks and domains (Joulin et al. 2016; Li et al. 2017; Desai and Johnson 2021; Sariyildiz, Perez, and Larlus 2020). CLIP (Radford et al. 2021) is pretrained using contrastive learning on a dataset of 400 million image-text pairs and has shown strong performance for both zero-shot learning and other downstream tasks. Other related approaches achieve similar performance with larger and noisier datasets (Jia et al. 2021), or fine-tune CLIP to adapt it to out-of-domain datasets (Kumar et al. 2022; Miller et al. 2021; Wortsman et al. 2022; Zhai et al. 2022). Still other methods have tried to improve the efficacy of CLIP training by incorporating self-supervised training techniques (Mu et al. 2022; Li et al. 2021, 2023), re-writing captions for pretraining images (Fan et al. 2023), and generating captions for images using off-the-shelf captioning models (Nguyen et al. 2023). We show in our ablation experiments that GIST captions contain more specific fine-grained information than off-the-shelf generated captions and result in better classification performance.

Large-scale captioning models (Wang et al. 2022; Yu et al. 2022) leverage image-text pairs to learn a shared representation space that is useful for generating text from images. These visual grounding methods are useful for generating high-level descriptions of objects. However, we show in our experiments that the captioning models do not perform well on fine-grained classes. Large-scale image generation models such as DALL-E (Ramesh et al. 2021), Imagen (Saharia et al. 2022), and stable diffusion (Rombach et al. 2022) are trained on large-scale image-text data in order to learn to generate photo-realistic images from text. In the few-shot learning literature, DALL-E has been shown to help by augmenting small image datasets with synthetic images(Zhang et al. 2023). This works well for Imagenet (Fan et al. 2023) classes, but we show in our experiments the generated images are not specific or accurate enough for fine-grained domains, such as dermatology diseases.

Recently, large language models trained with huge quantities of textual data have achieved remarkable human-level performance on many NLP tasks (Devlin et al. 2018; Lee et al. 2020; Brown et al. 2020a; Touvron et al. 2023; Chowdhery et al. 2022). GPT (Brown et al. 2020a; OpenAI 2023) is a self-supervised pretrained large language model with 175 billion parameters. GPT has knowledge across many domains and can be used for a variety of language tasks, including text generation. GPT can generate accurate, detailed descriptions for many different domains, especially when it is first prompted with structured examples of the desired text format. We use the strong prior knowledge of GPT to generate text descriptions and show that these text descriptions carry more useful information than generated image captions from models such as GIT. We further show that matching these captions to images and using image-text pairs for fine-tuning CLIP improves classification over image-only methods, suggesting that GPT-generated descriptions hold useful knowledge about fine-grained classes.

There are several existing methods that leverage foundation models for improving few-shot classification. CLIP (Radford et al. 2021) presents classification results from using zero-shot and few-shot linear-probe methods, which have become common baselines to compare to in the vision-language few-shot learning literature. Since CLIP is a pretrained model with an aligned image-text representation space, zero-shot classification can be performed by encoding class names in the form “a photo of a…” and then finding the closest class embedding to each encoded image. The CLIP linear-probe classification method involves learning a linear classifier on the frozen CLIP image embeddings. We show in our experiments that fine-tuning CLIP on our image and generated text pairs improves classification accuracy over zero-shot and CLIP linear-probing.

LaBo (Yang et al. 2023) uses GPT-3 to generate candidate concepts for each class and aligns these text concepts to the images using CLIP. Our results demonstrate that our image-text description matching and use of contrastive learning to align the image and text embeddings improves over their approach. The Prompt, Generate, then Cache method (Zhang et al. 2023) use foundation models to generate text and synthetic images to supplement few-shot learning. They then use CLIP and contrastive learning to ensemble predictions. Another recent method (Udandarao, Gupta, and Albanie 2022) also generates images to improve upon zero-shot CLIP performance. We show in our results that synthetic images do not generalize well to fine-grained domains, hindering performance. FLYP (Goyal et al. 2023) uses class names as text descriptions and contrastive loss to align image-text embeddings. Another recent method (Lin et al. 2023) also uses class names to construct text prompts and trains a linear classifier on the shared image-text representation space. We show that generating full text descriptions and matching descriptions to images outperforms using only the class name. ProD (Tianyi, Sun, and Yang 2023) trains on multiple datasets to learn both domain-specific prompts from image features and a domain-general prompt that improves cross-domain generalization. Their experiments demonstrate that having both domain-specific and domain-independent prompts improves few-shot classification accuracy over a “CNN+Transformer” baseline. There is no public code available for ProD. A concurrent paper (Maniparambil et al. 2023) to our work uses GPT-4 and a pre-trained CLIP model to improve zero-shot and few-shot classification.

Consider a set of images $i\in I$ that each have a fine-grained label mapping the image to a class $y\in Y$. Our goal is to generate fine-grained text, $t_{i}$, for each image. Suppose we have a pretrained image-text alignment model (e.g. CLIP (Radford et al. 2021)), where the quality of image-text alignment can vary depending on domain. Let $T$ represent the space of text descriptions for images. This pretrained image-text alignment model consists of an image encoder $f_{I}:I\mapsto\mathbb{R}^{d}$ and text encoder $f_{T}:T\mapsto\mathbb{R}^{d}$. Suppose that we have a pretrained large language model, $L$, from which we can generate sets of class-specific descriptions based on language priors $D_{y}=L(prompt(y))$.

We use large language models to supplement our image dataset with text. For each class $y\in Y$, we generate m class-specific captions, where m depends on domain and language priors $D_{y}=L(prompt(y))$. We then use our pretrained image-text alignment model to match each image in our training set $i\in I$ with our generated class-specific text to get image-specific text, $t_{i}$. We show an overview of the GIST method in Figure 2.

We find that contrastive fine-tuning improves fine-grained image classification over using a linear-probe on frozen pretrained image embeddings. Following our text description generation method, we have training set image $i$ and an associated set of generated text descriptions $t_{i}$, where $|t_{i}|=n$. We use both the images and generated text descriptions to fine tune the image and text encoders to align the images and generated text as shown in Figure 3. Concretely, we optimize the fine-tuning objective,

for a batch of images and their generated text descriptions $D^{\prime}={(i_{0},t_{0}),...}$, where $t_{0}\in t_{i_{0}}$ and $\theta$ represents all of the image and text encoder parameters. Once we have aligned embeddings, we update the parameters in the image and text encoders $f^{\prime}_{I}$ and $f^{\prime}_{T}$. We then train a linear probe classifier on the images by first encoding images using $f^{\prime}_{I}$ into the aligned text-image embedding space.

In this section, we analyze and provide insight into our method design choices. We compare GIST to off-the-shelf captioning methods and we study the impact of CLIP model, number of matched captions per image, and caption length on classification performance. We also compare against a visual grounding method and find that our CLIP fine-tuning results in much better classification accuracy.

The original Fitzpatrick17k dataset (Groh et al. 2021) has 11,788 labeled images collected from two dermatology websites, DermaAmin and Atlas Dermatologico. As mentioned in the CiRCLe paper (Pakzad, Abhishek, and Hamarneh 2023), the Fitzpatrick17k dataset has many erroneous images that are unrelated to dermatology. Additionally, there is large class imbalance amongst the 114 class labels. We chose 40 of the smaller class-size labels to manually clean and form a new dataset. Fitzpatrick40 has 2,609 images, which we split into 2,115 training, 222 validation, and 272 test. There are between 25 and 73 training images per class.

The labels in our Fitzpatrick40 dataset include: becker nevus, epidermal nevus, pilar cyst, erythema nodosum, stasis edema, sun damaged skin, xeroderma pigmentosum, behcets disease, perioral dermatitis, lentigo maligna, disseminated actinic porokeratosis, halo nevus, solid cystic basal cell carcinoma, port wine stain, livedo reticularis, lichen simplex, ichthyosis vulgaris, dyshidrotic eczema, congenital nevus, naevus comedonicus, aplasia cutis, porokeratosis of mibelli, calcinosis cutis, seborrheic keratosis, erythema elevatum diutinum, mucous cyst, pilomatricoma, erythema annulare centrifigum, acrodermatitis enteropathica, pustular psoriasis, pityriasis lichenoides chronica, nevocytic nevus, lichen amyloidosis, keratosis pilaris, granuloma pyogenic, epidermolysis bullosa, drug induced pigmentary changes, basal cell carcinoma morpheiform, acanthosis nigricans, nevus sebaceous of jadassohn.

The CUB200-2011 (Wah et al. 2011) is a well-cited, commonly-used dataset for image classification. The dataset offers great diversity of images and bird species. While working with the CUB200 dataset and examining validation images misclassified by our model, we noticed that some images in the dataset are mislabeled or are partially incorrect (e.g. have two different bird species present). We show a few of these images in Figure 8.

We provide the prompts we use for each dataset to generate the long GPT captions.

Given a set of generic body parts: face, neck, arms, torso, legs, torso, scalp, hands, feet. For the Fitzpatrick40 labels, we generate long class captions with the following prompt:

prompt = """You are a dermatology disease
describer. Describe what an image of
<<disease>> might look like on a person’s
<<body part>>."""

For the CUB200-2011 labels, we generate long class captions for the male and female bird genders with the following prompt:

prompt = """You are a bird species
describer. Describe what an image of a
<<gender>> <<species>> might look like.
"""

For the Flowers102 labels, we generate long class captions for flower species with the following prompt:

prompt = """You are a flower describer.
Describe what an image of a flower of
<<species>> might look like."""

For the FGVC labels, we generate long class captions for aircraft models with the following prompt:

prompt = """You are an airplane model
describer. Please describe
distinguishing characteristics of what
the plane looks like in 2-3 sentences.
What would a plane of type <<model>>
look like?"""

Our method uses GPT to generate long class-specific descriptions. After matching each training image to a description, our method uses GPT again to summarize the matched descriptions into concise captions. We show examples of the original long and the summarized short captions in Table 10. As explained in the main paper and supported by our method analysis studies, having longer captions is useful for the image-caption matching phase. Including details, such as body part, in the longer captions increases the likelihood that matched long captions will accurately and specifically describe the images. After matching the captions and images, we find that using GPT to summarize the captions results in descriptions that contain the key visual features needed for classification.

We show additional generated data from each method.