Learning Adversarial Semantic Embeddings for Zero-Shot Recognition in Open Worlds

In ZS-OSR, there are three different types of classes, including seen classes $\mathcal{Y}^{seen}$, unseen classes $\mathcal{Y}^{unseen}$, and unknown classes $\mathcal{Y}^{unknown}$, where $\mathcal{Y}^{seen}\cap\mathcal{Y}^{unseen}\cap\mathcal{Y}^{unknown}=\emptyset$. Compared to the $\mathcal{Y}^{unknown}$ classes that the training data does not provide any prior information, the classes in $\mathcal{Y}^{seen}$ and $\mathcal{Y}^{unseen}$ are considered as known classes since the training data contains the image samples of the $\mathcal{Y}^{seen}$ classes and some side semantic information of both $\mathcal{Y}^{seen}$ and $\mathcal{Y}^{unseen}$ classes. ZS-OSR aims to accurately recognize the images of unseen classes while rejecting the images of unknown classes based on these training image samples and semantic information.

Formally, given a training set consisting of $\mathcal{D}^{seen}=\{(\mathbf{x},y_{\mathbf{x}},\mathbf{a}_{y})\mid\mathbf{x}\in\mathcal{X}^{seen},y_{\mathbf{x}}\in\mathcal{Y}^{seen},\mathbf{a}_{y}\in\mathcal{A}^{seen}\}$ and $\mathcal{D}^{unseen}=\{(\tilde{y}_{i},\mathbf{a}_{\tilde{y}_{i}})\mid\tilde{y}_{i}\in\mathcal{Y}^{unseen},\mathbf{a}_{\tilde{y}_{i}}\in\mathcal{A}^{unseen}\}$, where $\mathbf{x}$ is an image from the seen class sample set $\mathcal{X}^{seen}$, $y_{\mathbf{x}}$ or $\tilde{y}_{i}$ is a class label, $\mathbf{a}_{y},\mathbf{a}_{\tilde{y}_{i}}\in\mathbb{R}^{M}$ are $M$-dimensional class semantic embeddings (i.e., class-level side information), $\mathcal{A}^{seen}$ and $\mathcal{A}^{unseen}$ contain the semantic embeddings of seen and unseen classes respectively. For test images consisting of images of both unseen and unknown classes, i.e., $\mathcal{X}^{unseen}\cup\mathcal{X}^{unknown}$, ZS-OSR aims to learn a model $\phi^{open}$ that maps the images to the class label space $\mathcal{O}=\mathcal{Y}^{unseen}\cup\{^{\prime}unknown^{\prime}\}$, where $\mathcal{O}$ includes an ’unknown’ class label in addition to the labels of the unseen classes.

One of the key challenges in ZS-OSR is the difficulty in distinguishing between the unseen and unknown classes due to the lack of training data for both of them. In typical OSR problems, the most commonly used approach involves conducting OSR first, followed by classification. Nevertheless, in ZS-OSR, none of the currently existing OSR methods have been found to be effective due to the lack of images from the unseen class for training purposes. To address this challenge, a straightforward solution would be to use simple combinations of existing generative ZSL and OSR methods. Specifically, generative ZSL methods can be first applied to generate latent visual features of unseen classes. The ZS-OSR task is then converted to a general OSR task in the visual feature space that contains the features of both seen and unseen classes. OSR methods can then be directly used to learn an open-set classifier using the training set composed of these features to recognize known (unseen) classes while being capable of rejecting unknown classes. Figure 3 shows the results of such solutions that uses the widely-used TF-VAEGAN [6] and MSP [33], OpenMax [28], Placeholder [29], Energy [35], ODIN [34], LogitNorm [48], and MaxLogit [49], as the generative ZSL method and the OSR methods, respectively, where the open scores are the likelihood scores of being unknown class samples yielded by the open-set classifier, with larger open scores indicating higher likelihood (see Table 4 for detailed quantitative results of these methods).

The results show that the method performs poorly in distinguishing between unseen and unknown classes since the open scores of many unseen class samples are large, highly overlapping with that of the unknown class samples. The ineffective performance may be impacted by the fact that the generative ZSL model generates features for the known unseen classes based on the closed-set assumption, without being informed of the possible presence of unknown classes. As a result, these generated unseen features can heavily overlap with that of the unknown classes in the feature space, rendering the subsequent OSR models ineffective. In the next section, we introduce the ASE approach that learns an open-set classifier for the zero-shot setting to address this issue.

Our proposed approach, namely Adversarial Semantic Embeddings (ASE), is a generative framework specifically designed for the ZS-OSR problem. Since ZS-OSR does not provide training samples of unseen and unknown classes, the framework aims to directly generate these samples to train a classifier for this task. Unlike the simple solution in Sec. 3.2 that directly generates the features of unseen classes for the subsequent OSR task, ASE takes a step back and focuses on learning faithful semantic embeddings of unknown classes via an adversarial learning approach before training an unknowns-informed OSR model.

An overview of ASE is provided in Figure 4, which consists of three successive components, including Using off-the-shelf generative ZSL models to generate unseen classes, learning adversarial semantic embeddings of unknown classes, and unifying these learned features and semantic embeddings to train a $K+1$ open-set classifier where $K=|\mathcal{Y}^{unseen}|$. The first component is to directly take an existing generative ZSL model to generate the visual features of the unseen classes based on their semantic embeddings. The second component is the key novelty of ASE, which is designed to take the trained generator and the closed-set classifier in the ZSL model as input to learn a set of adversarial semantic embeddings of unknown classes, such that the learned semantic embeddings are distributed around the boundary between the known and unknown classes in the semantic space. In the third component, these semantic embeddings are subsequently used to generate a set of adversarial feature vectors to represent the samples of the unknown classes, along with the previously generated unseen class samples, to train the $K+1$ classifier for OSR. Below we introduce each component in detail.

Training a zero-shot open-set classifier in the visual feature space requires the features of unseen classes. Generative ZSL models have demonstrated superior performance in utilizing the relationship between semantic embeddings and image features of the seen classes to generate the unseen-class features. Therefore, existing off-the-shelf generative ZSL models are directly taken to generate these features. Briefly, generative ZSL methods learn a generator network $G$ to generate sample $\tilde{\mathbf{x}}=G(\mathbf{a},\epsilon)$ conditioned on a Gaussian noise $\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ and a class semantic embedding $\mathbf{a}$. Meanwhile, a discriminator network $D(\mathbf{x},\mathbf{a})$ is learned by taking an input feature $\mathbf{x}$ and outputs a real value representing the probability that $\mathbf{x}$ is from the real data rather than from the generator network. $G$ and $D$ can be learned by optimizing the following adversarial objective:

where $\mathbf{x}\in\mathcal{X}^{unseen}$ is an image from the seen classes and $\mathbf{a}\in\mathcal{A}^{\text{seen}}$ is its class semantic embedding. The trained generator generates synthetic features $\tilde{\mathcal{X}}^{unseen}=\{\tilde{\mathbf{x}}_{l}^{\tilde{y}_{i}}|\tilde{\mathbf{x}}_{l}^{\tilde{y}_{i}}=G(\mathbf{a}_{\tilde{y}_{i}},\epsilon_{l})\}$ for semantic embedding $\mathbf{a}_{\tilde{y}_{i}}$ of each class, where $\mathbf{a}_{\tilde{y}_{i}}\in\mathcal{A}^{unseen}$. Then we obtain $\tilde{\mathcal{D}}^{unseen}=\{(\tilde{\mathbf{x}},\tilde{y}_{\tilde{\mathbf{x}}})\mid\tilde{\mathbf{x}}\in\tilde{\mathcal{X}}^{unseen},\tilde{y}_{\tilde{\mathbf{x}}}\in\mathcal{Y}^{unseen}\}$, and train a general ZSL (closed-set) classifier $\phi^{closed}$ to classify the test image samples from unseen classes.

Existing state-of-the-art (SOTA) generative ZSL models [24, 5, 6] can be directly used to implement this component. The generator $G$ and the $\phi^{closed}$ classifier (or the generated unseen-class features) are taken as input to adversarially learn the semantic embeddings of unknown classes.

ASE is dedicated to learning an unknowns-informed open-set classifier. However, we are given neither semantic embeddings nor image samples of the unknown classes. ASE aims to learn adversarial representations of the unknown classes to train such a classifier. Given the generated features and the training data, the representations of the unknown classes can be adversarially learned in either the semantic embedding space or the visual feature space. However, as discussed in Sec. 3.2, the generated unseen-class features are not faithful enough as they are generated under the closed-set setting. Consequently, directly using these synthetic unseen-class features to generate the unknown-class features may accumulate and/or amplify the closed-set biases in the generator, leading to non-discriminative features for the unknown classes. Such unseen-class and unknown-class features are ineffective in training the open-set classifier (see Table 5).

Thus, ASE instead learns the unknown-class representations in the semantic space, while enforcing separable unseen-and-unknown representations in the visual feature space. This approach is more plausible since 1) it directly learns the unknown-class semantic embeddings based on the pre-defined embeddings of both seen and unseen classes, while the other approach above is an indirect way that heavily relies on the unstable quality of the generated unseen-class visual features; and 2) it seamlessly leverages the learned relation between the semantic space and the visual feature space in the ZSL models to learn the unknown-class semantic embeddings.

To this end, ASE introduces a class-wise adversarial semantic embedding learning approach to generate a set of semantic embeddings of the unknowns, $\mathcal{A}^{unknown}$. As highlighted in Figure 4, for each unseen class, ASE generates multiple adversarial semantic embeddings for the unknown classes that are tightly distributed around but separable from the unseen-class embeddings. To achieve this goal, it jointly minimizes a distance loss $\mathcal{L}_{\text{dis}}$ in the embedding space that brings the unknown-class embeddings closer to the unseen-class embeddings, and an adversarial loss $\mathcal{L}_{\text{adv}}$ in the feature space that pulls the corresponding prototypical unknown-class features away from the generated unseen-class features:

where $\beta$ is a hyper-parameter, $\mathcal{L}_{\text{dis}}$ is defined as the Euclidean distance between the generated unknown-class embedding $\hat{\mathbf{a}}\in\mathcal{A}^{unknown}$ and the given unseen-class embedding $\tilde{\mathbf{a}}\in\mathcal{A}^{unseen}$:

and $\mathcal{L}_{\text{adv}}$ is defined as a Helmholtz free energy-based loss:

where $\phi^{closed}$ is the ZSL (closed-set) classifier obtained from the off-the-shelf ZSL model, and $\hat{\mathbf{p}}=G(\hat{\mathbf{a}},\epsilon)$ is a generated prototypical unknown-class feature vector corresponding to the adversarial semantic embeddings $\hat{\mathbf{a}}$. Note that since $\phi^{closed}$ was trained using the unseen-class features and its weight parameters are fixed, the energy score that the unseen-class features receive are consistently low. Thus, Eq. (4) is designed to encourage high energy scores for unknown-class feature prototypes only. By minimizing $\mathcal{L}_{\text{ase}}$, the unknown-class and unseen-class class embeddings are close to each other, but they are discriminative from each other; this adversarial relation also applies to the corresponding unknown-class feature prototypes w.r.t. the unseen-class features.

We then train an unknowns-informed ZS-OSR classifier with $K+1$ classes, in which the extra (+1) class is for the ‘unknown’ class and it is trained based on the learned unknown-class semantic embeddings.

Specifically, we first utilize the adversarial semantic embeddings $\hat{\mathcal{A}}^{unknown}$ and the trained generator $G$ to obtain a set of unknown-class features $\hat{\mathcal{X}}^{unknown}=\{\hat{\mathbf{x}}_{l}^{\hat{y}_{i}}|\hat{\mathbf{x}}_{l}^{\hat{y}_{i}}=G(\mathbf{a}_{\hat{y}_{i}},\epsilon_{l}),\mathbf{a}_{\hat{y}_{i}}\in\hat{\mathcal{A}}^{unknown},\hat{y}_{i}\in\mathcal{Y}^{unknown}\}$, where $\mathcal{Y}^{unknown}$ is a set of unknown classes collectively labeled as ‘unknown’, resulting in $\tilde{\mathcal{D}}^{unknown}=\{(\hat{\mathbf{x}},^{\prime}unknown^{\prime})\mid\tilde{\mathbf{x}}\in\hat{\mathcal{X}}^{unknown}\}$. The unknown-class features $\hat{\mathbf{x}}$ are expected to be centered around the unknown-class feature prototypes $\hat{\mathbf{p}}$. The unseen-class and unknown-class features are then combined to form the open-set training data, i.e., $\mathcal{D}^{open}=[\tilde{\mathcal{D}}^{unseen},\tilde{\mathcal{D}}^{unknown}]$, which is used to train the open-set classifier $\phi^{open}$ by minimizing a standard cross-entropy loss:

During inference, given a test image $\mathbf{x}$, $\phi^{open}$ yields a softmax score in its $K+1$-th class prediction, which can be directly used as an open score. If the score exceeds a pre-defined threshold, $\mathbf{x}$ is predicted as ‘unknown’, and otherwise $\mathbf{x}$ is predicted as the class with the highest logit among the $K$ unseen classes. Alternatively, post-hoc OSR methods like MSP [33] and ODIN [34] can also be applied for obtaining the open score, but the $\phi^{open}$-based open score is generally more effective (shown in Table 5), and is used by default.

The ZS-OSR results of ASE and the combined baselines on the four proposed datasets are shown in Table 4. ASE outperforms all eight baseline methods by a significant margin in detecting unknown-class samples and performs comparably well in terms of classification on the unseen-class samples on all four datasets. Details are discussed below.

Following the OSR literature [52, 29, 53, 54], we conduct experiments on the CUB dataset to examine ZS-OSR performance under varying degrees of openness, defined as $1-\sqrt{\frac{|\mathcal{Y}^{\text{unseen}}|}{|\mathcal{Y}^{\text{unseen}}|+|\mathcal{Y}^{\text{unknown}}|}}$. A larger openness indicates the presence of relatively more unknown classes in the test data. The experiment is focused on the CUB dataset. In particular, there are 50 unseen classes in CUB under the ZSL setting [10]. We create four ZS-OSR datasets based on CUB by retaining 10 classes as the unseen classes and the respectively remaining 10, 20, 30, and 40 classes as the unknown classes. This results in four datasets with respective openness of 29.3%, 42.3%, 50%, and 55.3%. The results of ASE and baselines on these four datasets are shown in Figure 7. It is clear that ASE maintains consistently superiority in unknown-class detection and comparably good unseen-class classification across different openness rates, demonstrating strong robustness and stability to the data openness.

Table 5 shows the ablation study results of two main modules in ASE, including $\mathcal{L}_{ase}$ and $\phi^{open}$. We discuss the results in detail below.

Our approach focuses on distinguishing unseen and unknown samples in ZS-OSR, but ASE can be extended to function under generalized ZS-OSR settings with a minor modification to Eq. (3). ASE generates tightly distributed adversarial semantic embeddings around each of the seen-class and unseen-class semantic embeddings, and can generate unknown-class features and train the $\phi^{open}$ classifier with seen-class, unseen-class, and unknown-class features. SOTA generalized ZS model GCM-CF [38] with MSP is included to extend our baselines under this setting. The results are presented in Table 6, where ASE achieves the most effective detection of unknown samples across all four datasets, outperforming the best baseline per dataset by 0.1-4.9% in AUROC. In terms of FPR95, ASE also maintains a similarly leading position. Although APN surpasses ASE on two datasets in FPR95, ASE substantially outperforms it in both AUROC and FPR on the other datasets.

While ZS-OSR focuses on the scenario where known classes and unknown classes belong to the same distribution, we believe that there exists scenarios of ZS-OOD (Out-of-Distribution), where we only know the semantic information of the known classes but need to detect images of unknown classes from a different distribution (e.g., samples from a largely different dataset). To evaluate ASE’s performance in the ZS-OOD setting, we randomly select 25 classes from AWA2, FLO, and SUN as unknown classes and combine them with 25 unseen classes of CUB to obtain three ZS-OOD test datasets, CUB-AWA2, CUB-FLO, and CUB-SUN. Table 7 shows that considering both AUROC and FPR95 metrics, ASE’s performance surpasses all baselines by a significant margin, demonstrating the empirical effectiveness of ASE under the ZS-OOD setting.