LATex: Leveraging Attribute-based Text Knowledge for Aerial-Ground Person Re-Identification

Motivation. Previous methods have shown strong performance for person ReID tasks with the same viewpoints but struggle in scenarios with significant viewpoint variations. Based on our daily experiences and observations of the dataset (Fig.1), we notice that person attributes remain robust in such complex scenarios, providing valuable additional information for target person retrieval. In light of this, we try to fully exploit this advantage to alleviate the challenges posed by viewpoint changes in ReID tasks.

Ideal Demo. We consider an ideal scenario where the ground truth of all person attributes is known in both the training and testing sets. Under this premise, we construct a simple demo using prompt-tuning CLIP to validate the feasibility of our motivation. This setup represents the theoretical upper bound of our framework’s performance. Fig. 2 (a) provides an overview of our demo, with additional details available in the appendix. And Fig. 2 (b) shows the results of our demo on the AG-ReID.v1 dataset, where our method achieves near-perfect scores on both protocols, fully demonstrating the validity of our motivation. Despite the superior performance, we recognize that the above experimental setting is overly idealized and impractical in real-world applications. Specifically, our proposed demo relies on the strong assumption that all person attributes are known during inference. However, annotating attributes for the vast amounts of data collected by numerous surveillance cameras is infeasible in practice. To address this, we propose a feature learning framework, which enables the model to automatically generate person attributes during inference. This eliminates the aforementioned unrealistic assumption, significantly enhancing the feasibility of our approach.

To enhance attribute perception, we propose the Prompted Attribute Classifier Group (PACG), which integrates attribute label information and exploits interdependencies among attributes. More specifically, we define PAC_t as the classifier for the $t$-th attribute. Then, the attribute features $F_{a}^{t}$ is defined as the concatenation of the global feature $F_{v}^{L}$ and corresponding attribute prompt $\tilde{P}^{L,t}_{v}$, where $F_{v}^{L,t}\in\mathbb{R}^{C}$ is the $t$-th attribute prompts of $F_{v}^{L}$:

To learn more discriminative attribute-aware features, we utilize the interdependencies of different attributes. Formally, we denote other attribute prompts as $\hat{P}^{L}_{v}$, i.e., $\hat{P}^{L}_{v}=\{P_{v}^{L,1},\cdots,P_{v}^{L,t-1},P_{v}^{L,t+1},\cdots,P_{v}^{L,T}\}$. Then, we can obtain interacted features as follows:

where $\Theta(\cdot)$ is the multi-head cross attention (Vaswani et al., 2017). $Q\in\mathbb{R}^{C}$, $K\in\mathbb{R}^{C}$ and $V\in\mathbb{R}^{C}$ are generated by the corresponding projection matrix $W_{q}$, $W_{k}$ and $W_{v}$, respectively. $\delta$ is the Softmax function. The residual structure enables the model to process input features with varying emphases more flexibly  (He et al., 2016). As observed in daily life, certain person attributes exhibit strong correlations (e.g., gender and clothing), while others show weaker correlations (e.g., height and weight) or are nearly independent (e.g., ethnicity and gender). Thus, we employ a residual-based Feed-Forward Network (FFN)  (Vaswani et al., 2017) to handle features obtained from two perspectives: direct prediction and attribute dependency-based prediction. This design allows the FFN to adapt to diverse scenarios by effectively capturing attribute correlations, if they exist, while avoiding excessive noise and additional computational overhead caused by irrelevant attribute pairs:

where $\Phi(\cdot)$ represents the FFN and $\mathcal{O}_{t}$ is the final features used to predict attribute confidences. In addition, to align the visual and textual representations in different feature spaces, we transform the visual embedding $F_{a}^{t}$ and obtain attribute-based text tokens as follows:

where $\Psi$ is a Multi-Layer Perceptron (MLP). The above attribute-based text tokens can be taken as the input of CLIP’s text encoder to improve the feature discrimination.

Datasets. We evaluate our methods on three AG-ReID benchmarks. AG-ReID.v1 (Nguyen et al., 2023a) is a challenging dataset, consisting of 21,983 images captured by ground and aerial cameras from 388 identities with 15 attributes. CARGO (Zhang et al., 2024b) is a large-scale synthetic dataset, consisting of 108,563 images captured by five aerial cameras and eight ground cameras from 5,000 identities. Meanwhile, AG-ReID.v1 contains two protocols for evaluation, i.e., A→G and G→A. CARGO contains four evaluation protocols, two of which are view-heterogeneous, while the other two are view-homogeneous. Finally, AG-ReID.v2 (Nguyen et al., 2024) is the extended version of AG-ReID.v1, incorporating three views: aerial (A), ground (G), and wearable device (W). Accordingly, the evaluations are expanded to include cross-view retrieval settings between A and W. Specifically, AG-ReID.v2 consists of 100,502 images from 1,615 unique identities.

Metrics. Following previous works, we employ the mean Average Precision (mAP) (Zheng et al., 2015) and Cumulative Matching Characteristic (CMC) (Moon & Phillips, 2001) at Rank-1 as evaluation metrics.

LATex. On AGReID.v1, our LATex achieves a Rank-1 accuracy of 84.41% and an mAP of 75.85% under the evaluation protocol A→G. Compared with VDT, it delivers 1.5% and 1.41% improvements in Rank-1 and mAP, respectively. As for the evaluation protocol G→A, the performance of our LATex is 88.88% Rank-1 and 79.19% mAP, showing a similar trade in evaluation protocol A→G. The stable improvements on two evaluation protocols clearly demonstrate the importance of further exploring attribute information in AG-ReID. On two large-scale datasets, CARGO and AG-ReID.v2, our LATex achieves highly competitive performance. It is worth noting that, since there are no attribute annotations on CARGO, we remove PACG. Our model achieves superior performance with significantly fewer trainable parameters, fully validating its effectiveness.

LATex†. We believe the limited trainable parameters constrain the performance of LATex. To enable a fairer comparison with previous methods based on full fine-tuning backbones, we introduce a LATex variant, LATex†, which adopts the same strategy. Compared to LATex, LATex† achieves significant performance improvements across all benchmarks. For example, on the CARGO dataset under the ALL protocol, LATex surpasses VDT with a Rank-1 accuracy of 66.99%, while LATex† further boosts this metric to 76.96%, achieving an almost 10% increase. This fully demonstrates the scalability of our methods.

View-homogeneous ReID. As shown in Tab. 2, CARGO provides protocols for person ReID under the same viewpoint, which we used to evaluate our LATex’s performance on view-homogeneous tasks. As previously demonstrated, the insufficient trainable parameters limit the optimization of the feature space, leading to a severe under-fitting of LATex on the A$\leftrightarrow$A protocol. Surprisingly, however, LATex outperforms VDT on the G$\leftrightarrow$G protocol. LATex†, with the support of additional trainable parameters, achieves the best overall performance among existing methods. Notably, under the G$\leftrightarrow$G protocol, it is the first method to achieve a Rank-1 accuracy exceeding 90%. These results demonstrate that our method retains a strong generalization ability in view-homogeneous tasks.

Effect of Different Modules. In Tab. 5, Model A is our baseline. It achieves a Rank-1 of 81.69% and an mAP of 72.36% under the protocol A→G, while attaining a Rank-1 of 83.89% and an mAP of 74.78% under the protocol G→A. With PACG, Model B increases the performance by 3.53% Rank-1 and 3.84% mAP scores under the protocol G→A. Obviously, its performance is already better than previous methods, such as VDT. With CPT, Model C achieves the best result across all evaluation metrics. These results clearly demonstrate the effectiveness of our key modules.

Effect of Leveraging Person Attributes. We further validate the effect of leveraging person attributes. As shown in Tab. 6, Model A and Model B are models without CPT. Model C and Model D are completed models. The key differences are the features used for retrieval. Here, $F_{a}$ is the concatenation of all $F_{a}^{t}$. As can be observed, the performance of Model B is superior to that of Model A. The performances of Model B and Model D are very comparable. The performance of Model C is the best. These results clearly highlight the effectiveness of leveraging attribute-based text knowledge for AG-ReID.

Role of Backbones. We emphasize that the core idea of our LATex is to extract person attributes, embed them as pseudo-texts, and feed them into a text encoder to achieve robust person ReID. With the strong vision-language alignment, we adopt CLIP as the backbone. Considering that our approach uses a different backbone from prior methods, we evaluate the effect of various backbones and report the performance on AG-ReID.v1 in Tab. 7. As can be seen, CLIP-based VDT and ViT-based LATex show a significant performance drop. The main reason is that they cannot fully leverage CLIP’s vision-language knowledge. These results clearly demonstrate that our LATex’s performance stems from effectively utilizing CLIP’s vision-language knowledge, rather than its inherent encoding capabilities.

Impact of Trainable Prompts. Fig. 4 shows the effect of the number of learnable prompts $P^{i}_{v}$. With the increase of trainable parameters, the performance of LATex remains stable, demonstrating its robustness to this factor.

Accuracy of Attribute Predictions. Attribute predictions play an important role in our framework. Fig. 5 shows the accuracy of some typical attributes predicted by PACG. It can be observed that our PACG achieves outstanding performances in these person attributes. These attributes provide discriminative information for person identification.

Rank list Comparison. Fig. 6 compares the rank lists generated by different models defined in Tab. 5. With the sequential addition of PACG and CPT, the rank lists become progressively more accurate and discriminative. This indicates that our model gradually acquires the ability to perceive person attributes, enabling it to better distinguish individuals with similar visual characteristics.

Attribute Retrieval. Fig. 7 illustrates the retrieval results using the attribute features of the query. Despite challenges such as image blurriness and small pixel regions of key areas, LATex accurately retrieves persons sharing a specific attribute (e.g., Upperwear) with the query’s ground truth. This result demonstrates the exceptional capability of LATex in perceiving person attributes.

Attribute Feature Distributions. Fig. 8 shows the feature distribution of all attribute categories in the test set. Each attribute category consists of several subcategories, which are finely perceived by the corresponding PAC. For example, the “Hair style” category includes various styles such as “Bald”, “Short” and so on. The results clearly demonstrate that our method exhibits strong perception and discrimination capabilities for unseen samples across different attributes.