Distribution Aligned Feature Clustering for Zero-Shot Sketch-Based Image Retrieval

The fundamental problem of the SBIR task is to project the hand-drawn sketches and the real-world images into a common metric space, where samples of the same category are close to each other. The earliest works extract hand-crafted features from the sketches and match them with the edge maps of the natural images (Saavedra, Barrios, and Orand 2015; Hu and Collomosse 2013; Eitz et al. 2011; Hu, Wang, and Collomosse 2011; Eitz et al. 2010). In recent years, with the prevalence of deep neural networks (DNNs), different architectures of neural networks are introduced into this field (Yu et al. 2016b; Song et al. 2017; Sangkloy et al. 2016; Qi et al. 2016; Zhang et al. 2016; Yu et al. 2016a) and achieve superb results. However, the closed-set setting of SBIR does not meet the demand of large-scale applications. To this end, zero-shot setting of SBIR (Yelamarthi et al. 2018; Shen et al. 2018) is proposed. Following zero-shot learning approaches, ZS-SBIR studies (Shen et al. 2018; Dutta and Akata 2019) use semantic information to bridge the seen and unseen classes. Other methods employ generative models, such as generative adversarial networks (GANs) (Zhang et al. 2018) and variational auto-encoders (VAEs) (Yelamarthi et al. 2018) to learn a mapping from sketches to images. However, the inherent sparsity of sketch and intra-class variance of images make it difficult for semantic information to transfer, or for the network to learn a generalizable mapping on the unseen classes. Following works (Huang et al. 2021; Wang et al. 2021b) start to use metric losses to improve retrieval performance, but the pair-wise losses often require large batch size or complex sample mining techniques to perform well. Different from the above approaches, our Cluster-then-Retrieve method reduces intra-class variance by utilizing gallery information.

Data clustering has been adopted for information retrieval for a long time. (Jardine and van Rijsbergen 1971) propose a hierarchic clustering method that significantly improves the efficiency of information retrieval. The study also proposes the cluster hypothesis: the associations between documents convey information about the relevance of documents to requests, and pointed out clustering’s potential of improving retrieval effectiveness. Based on this statement, many studies (Vdorhees 2017; Hearst and Pedersen 1996; Tombros, Villa, and van Rijsbergen 2002) try to investigate whether clustering can bring performance improvement to retrieval algorithms. However, although improvements could be achieved through manually selecting clusters (Sze 2004), clustering-based retrieval fails to gain better results in practice (Shaw, Burgin, and Howell 1997). In contrast, clustering has been widely applied to feature quantization (Jégou, Douze, and Schmid 2011; Ge et al. 2013; Kalantidis and Avrithis 2014) , enhancing retrieval efficiency on large-scale datasets. In the scenario of ZS-SBIR, estimating distances in a single domain could be more accurate than that between different domains, so the potential of cluster-based retrieval can be revealed.

Consider a dataset consisting of the training set and the testing set. We denote the training set as $\mathcal{D}^{tr}=\{\mathcal{I}^{tr},\mathcal{S}^{tr}\}$, where $\mathcal{I}^{tr}$ and $\mathcal{S}^{tr}$ are subsets of images and sketches. Similarly, the testing set is defined as $\mathcal{D}^{te}=\{\mathcal{I}^{te},\mathcal{S}^{te}\}$.

Let $\mathcal{I}^{tr}=\{(\mathcal{I}_{i},y_{i})\mid y_{i}\in\mathcal{C}^{tr}\}^{N_{1}}_{i=1}$ and $\mathcal{S}^{tr}=\{(\mathcal{S}_{i},y_{i})\mid y_{i}\in\mathcal{C}^{tr}\}^{N_{2}}_{i=1}$, where $y_{i}$ is the class label and $\mathcal{C}^{tr}$ is the set of training classes. The images and labels are used to train the model to learn discriminative features. After training, the image encoder generates feature vectors of the gallery images and query sketches, preparing for the retrieval. The testing images and sketches are $\mathcal{I}^{te}=\{(\mathcal{I}_{i},y_{i})\mid y_{i}\in\mathcal{C}^{te}\}^{N_{3}}_{i=1}$, $\mathcal{S}^{te}=\{(\mathcal{S}_{i},y_{i})\mid y_{i}\in\mathcal{C}^{te}\}^{N_{4}}_{i=1}$, and the corresponding feature vectors are denoted as $\mathcal{F}^{\mathcal{I}}=\{\mathbf{x}_{i}\}^{N_{3}}_{i=1}$, $\mathcal{F}^{\mathcal{S}}=\{\mathbf{q}_{i}\}^{N_{4}}_{i=1}$. During testing, given a sketch feature $\mathbf{q}_{i}$, its distances with gallery features $\mathcal{F}^{\mathcal{I}}$ are calculated and sorted, giving an optimal ranking of gallery images. In the zero-shot setting, the testing classes and the training classes do not overlap, i.e., $\mathcal{C}^{tr}\cap\mathcal{C}^{te}=\emptyset$.

The ZS-SBIR is inherently a difficult problem because of its form: a cross-domain zero-shot metric learning task. We calculate and sort the distance between sketch and gallery images for each query. The highly abstract strokes provide merely sparse geometry information, so the model can only infer from the image edges and sketch strokes. However, as Fig.1 shows, objects in gallery images appear in various forms and are surrounded by complex backgrounds. Without the help of large pretrained models or side information, this problem seems intractable.

Thanks to the ImageNet pretrained model, we can obtain an informative feature space on the gallery images, where objects of different views can be grouped. Therefore, we perform clustering on the gallery and use the cluster centroids as proxies for retrieval.

The Cluster-then-Retrieve method can greatly reduce the intra-class variance of gallery images, but the domain gap between sketches and images is not resolved explicitly. Inspired by (Wang et al. 2021a), we use Kullback–Leibler divergence to align the image and sketch features with a common Gaussian distribution. For each batch of training sample $\mathbf{X}=\{\mathbf{x}_{i}\in\mathbb{R}^{D}\}^{N}$, we sample a random feature batch $\mathbf{P}=\{\mathbf{p}_{i}\in\mathbb{R}^{D}\}^{N}$ from a given Gaussian distribution $\mathcal{N}(0,1)$. The distribution loss is formulated as:

where $E$ is the image encoder shared between images and sketches. As this regularization is applied to both image and sketch features, the feature distribution of the two domains can be pulled closer under the guidance of Gaussian distribution, reducing the domain gap in an indirect manner.

The remaining part of our model is adopted from (Liu et al. 2019), where the model is initialized from a CSE-ResNet-50 network pretrained on ImageNet (Deng et al. 2009) and is supervised by the pretrained model through distillation during the training process.

The main training objective is a cross-entropy loss that classifies both sketches and natural images. For each training batch $\mathbf{X}=\{\mathbf{x}_{i}\in\mathbb{R}^{D}\}^{N}$, the classification loss can be written as:

where $\alpha$ and $\beta$ are the weight and bias of the classifier.

To preserve features learned from the ImageNet and improve the model’s discriminability on unseen classes, (Liu et al. 2019) proposes a distillation loss formulated as:

where $\gamma$ and $\delta$ are the weight and bias terms in the ImageNet label classifier of the teacher model. $\mathcal{C}^{T}$ denotes the classes of the teacher model prediction. $p_{i,k}$, i.e., the soft label is given by $p_{i,k}=\textit{Softmax}(\lambda_{t}\cdot\mathbf{t}_{i}+\lambda_{a}\cdot\mathbf{a}_{y_{i}})$. Where $\mathbf{t}_{i}$ is the classification logit for class $i$ given by the teacher network, and $\mathbf{a}_{y_{i}}$ is the semantic similarity between class $i$ and label class $j$ given by WordNet (Miller 1992). $\lambda_{t}$ and $\lambda_{a}$ control the importance of classification logit and semantic similarity.

The full objective function of our network is:

where $\lambda_{1}$, $\lambda_{2}$ and $\lambda_{3}$ are hyperparameters to balance the contribution of each loss.

To verify the effectiveness of our method, we follow the common practice of previous works and experimentally evaluate our model on two popular SBIR datasets: Sketchy (Sangkloy et al. 2016) and TU-Berlin (Eitz, Hays, and Alexa 2012).

We implement our model on the codebase provided by (Liu et al. 2019). The SE-ResNet-50 network pre-trained on ImageNet is chosen to initialize the teacher and student networks. We train our model with the Adam optimizer with $\beta_{1}=0.9$, $\beta_{2}=0.999$. The learning rate is initialized as $1e-4$ and exponentially decayed to $1e-7$. We set $\lambda_{1}$ to $1$, $\lambda_{2}$ to $1$ and $\lambda_{3}$ to $0.1$ unless stated. $\lambda_{t}$ and $\lambda_{a}$ are set to $1$ and $0.3$ for all experiments following the setting of (Liu et al. 2019). The semantic similarity is given by the WordNet python interface from nltk corpus reader. The random feature batch in our distribution alignment loss follows the distribution of $\mathcal{N}(0,1)$. We set the batch size to 80 and train the model for 20 epochs.

For the hyperparameters of clustering, we set $M$ to $2$ and $K$ to $32$ for all experiments unless otherwise stated. Fuse coefficient $\lambda$ is set to 0.2. The effects of different hyperparameters will be discussed in the appendix.

We compare our ClusterRetri model with various methods of ZS-SBIR, including SEM-PCYC (Dutta and Akata 2019), SAKE (Liu et al. 2019), CSDB (Dutta and Biswas 2019), DSN (Wang et al. 2021b) and TVT  (Tian et al. 2022). The results are shown in Table 1. We first retrain the baseline model, i.e., the SAKE (Liu et al. 2019). It should be noted that with batch size of 80, SAKE can achieve 0.582 mAP@all on Sketchy dataset, which surpasses the result in the original paper, so we train our models with batch size of 80.

With the proposed feature distribution loss, our baseline model can achieve 0.613 mAP@all on Sketchy dataset and 0.498 mAP@all on TU-Berlin dataset, which is comparable with state-of-the-art methods.

Apply our Cluster-then-Retrieve method to the feature generated by the distribution regulated network, and we can have a huge boost on the retrieval performance. Specifically, in 512-d real value retrieval, our method achieves 0.762 and 0.597 mAP@all on Sketchy and TU-Berlin, respectively.

Following previous works (Dutta and Akata 2019; Liu et al. 2019), we use iterative quantization (Gong et al. 2012) (ITQ) to generate binary features. Here the input of the ITQ algorithm is the reconstructed feature using clustered centroids. Specifically, the features’ subvectors are replaced with the closest centroids in corresponding subspaces. Then we calculate the hamming distance to conduct retrieval. Surprisingly, the performance is further improved, achieving 0.781 mAP@all on Sketchy dataset and 0.672 mAP@all on TU-Berlin dataset. We assume this is probably because the iterative quantization can reduce redundant features that may affect retrieval.

Using 64-d features, our method’s performance is comparable with the previous SOTA method with 512-d features, demonstrating our ClusterRetri’s effectiveness. Under the 64-d binary setting, the real value features are compact enough, so our method experiences a drop in performance, but it also outperforms other methods by a large margin.

Our gallery feature clustering pipeline can achieve impressive results. It should be noted that the K-Means clustering and subspace division is very similar to the classical vector quantization (VQ) (Gray 1984) and product quantization (PQ) (Jégou, Douze, and Schmid 2011) algorithms in information retrieval, which focus on improving the retrieval efficiency by lookup tables.

While driven by different motivations, our method can also enjoy the speed improvement brought by PQ. We conduct experiments on a machine using Intel Xeon Silver 4215R. Table.2 shows the time usage of distance calculation and corresponding retrieval performance. The distance size is $15229\times 17101$. It can be seen that our method can achieve superior performance and efficiency simultaneously. Using 512-d features and ClusterRetri, the time of distance calculation can be much faster than that of 64-d real number features.