TraND: Transferable Neighborhood Discovery for Unsupervised Cross-domain Gait Recognition

Fig. 2 shows the structure of the Transferable Neighborhood Discovery method with two main stages. In the first stage, we adopt GaitSet [2] as the backbone network to learn gait features from silhouette sequences. We train the GaitSet network on the labeled source dataset to learn the prior knowledge of gaits. In the second stage, we first map unlabeled target samples into the feature space with the trained backbone. After that, target samples are distributed in a manifold with the prior knowledge learned from the source data. Then we adopt the class consistency indicator measured by the entropy of samples to select the confident neighborhoods of samples in a progressive manner inspired by [8]. We explore a high-entropy-first strategy to select confident neighborhoods, which are finally used to update the network.

To learn discriminative representation from gait sequences, we adopt the state-of-the-art CNN based model, i.e., GaitSet [2], as the backbone network in the TraND framework, as shown in Fig. 2. The GaitSet network directly takes the silhouette sequence as the input, which considers the sequence as a set but ignores the order of frames based on the assumption that the appearance of a silhouette has contained its position information. By this means, the spatial features of all frames can first be kept to comprehensively model the gait representation. We then aggregate the frame-level features into a sequence-level feature with the Set Pooling (SP) operation as in GaitSet [2]. At last, to discover the multi-scale features, the Horizontal Pyramid Mapping (HPM) is applied to generate a discriminative representation of gait sequences, which splits the feature map extracted by SP into $\sum_{s=1}^{5}2^{s-1}$ strips on height dimension. Given a sequence of gait silhouettes, $X=\{x_{k}\}^{K}$, where $x_{k}$ is one silhouette image and $K$ is the length of the sequence, the overall process of the GaitSet network, $F(\cdot)$, can be formulated as follows:

where $f(\cdot)$ is the CNN to extract the frame-level feature, $g(\cdot)$ is the SP function to generate the sequence-level feature, and $h(\cdot)$ is the HPM operation as in GaitSet [2].

Since we only have labels of the source data but no knowledge about the target domain, it is necessary to effectively exploit the prior knowledge for gait representation in the source dataset. Inspired by methods of unsupervised person Re-Id [34], we train the backbone on the source dataset $\{\mathcal{X}_{S},\mathcal{Y}_{S}\}$ as shown in the upper part of Fig. 2. Since the IDs in the source data and the target data are not overlapped, it is better to make the model learn the similarity or difference between samples rather than classify them. So, we adopt the Triplet Loss [17] instead of Cross-Entropy Loss for prior knowledge learning. The formulation of the Triplet Loss is

where $N$ is the number of training sequences, $X_{i}^{a}$ is the anchor, $X_{i}^{p}$ and $X_{i}^{n}$ are positive and negative sequences with respect to the anchor, respectively, and $m$ is the margin parameter. Here, all samples are randomly selected from the source dataset $\mathcal{X}_{S}$.

Base on the prior knowledge, the trained backbone network can support a discriminative latent space that makes the samples of different persons in the source domain differentiable. Therefore, we can map the samples of the target dataset into this feature space by feeding them into the backbone to obtain their primary representations in the feature space. However, due to the domain gap between the source domain and the target domain, the samples in the target domain can be shifted that the gait sequences of the same person may be far from each other while the sequences of different persons may be close. If we directly apply the clustering method on the target features based on their pairwise distances, the ambiguous samples can bring noises for refining the model. Moreover, since the number of classes/IDs in the target dataset is unknown, it is difficult to set proper parameters for the clustering methods, such as $K$ in the k-means algorithm.

Inspired by the AND method for unsupervised deep learning [8], we present the TraND framework to transfer the knowledge from the source data to the target data by neighborhood discovery. Here we introduce how to discover neighborhoods with the learned prior knowledge, as shown in the lower part of Fig. 2. First of all, we define the anchor neighborhood (AN) using $k$ nearest neighbors ($k$NN) as in [8]. Given a sample of gait sequence $X_{t}\in\mathcal{X}_{T}$, a neighborhood $\mathcal{N}_{k}(X_{t})$ anchored to $X_{t}$ is formulated as:

where $S$ is the feature space, $s(\cdot,\cdot)$ is a pairwise similarity metric such as the cosine similarity.

Neighborhood Selection with Prior Knowledge. Due to the domain gap between the source dataset and the target dataset, it is hard to select confident samples with class-consistent neighbors. Inspired by AND [8], we adopt the class consistency indicator $H(X_{t})$, where the larger the value of $H(X_{t})$ is, the more reliable of the corresponding sample is. Given a gait sequence $X_{i}\in\mathcal{X}_{T}$, $H(X_{i})$ is formulated as:

where $p_{i,j}$ is the non-parametric softmax measurement and $\tau$ is a temperature for controlling distribution concentration.

The $p_{i,j}$ represents the degree of the similarity between $X_{i}$ and $X_{j}$, and the larger $p_{i,j}$ is, the more similar $X_{i}$ is to $X_{j}$. Since AND [8] is designed for unsupervised learning without any prior knowledge on the training data, the samples in the feature space distribution like the instance discrimination [21], only a few samples get high similarity, which means that these samples are far away from other samples, i.e., these samples will be far away from the large group and reside in a low-density area with sparse visual similar neighbours surrounding. After the operation of softmax, the $p_{i,j}$ will be large and the corresponding $H(X_{i})$ will be small, which means $X_{i}$ resides in a low-density area. Therefore, in the original AND, the low-entropy-first strategy is exploited to select the most reliable samples for optimization. However, in our TraND, more visually similar samples of the target domain have latent relationships due to the prior knowledge learned from the source data, These samples that reside in dense area has small $p_{i,j}$ and the corresponding $H(X_{i})$ will be large. In order to better use the knowledge learned from the source domain, here we adopt the high-entropy-first strategy for neighborhood selection which is proved the effectiveness by the further experiment results.

Model Updating with Neighborhood Supervision. Inspired by curriculum learning [1], we refine the backbone CNN in a from-easy-to-hard manner, by which we train the backbone in $R$ rounds. For each round $r$, we apply above the neighborhood selection strategy to obtain the top $r/R\times 100\%$ anchor samples with their neighbors to update the backbone. At last, the backbone is optimized with the unsupervised anchor neighborhood loss which is formulated as:

Datasets. The experiments is performed on two large-scale gait datasets, i.e., CASIA-B [30] and OU-LP [9]. The CASIA-B dataset contains 124 persons captured from 11 viewpoints ($0^{\circ}$, $18^{\circ}$, …, $180^{\circ}$). Each person walks six times under normal conditions (NM #1-6), two times in their coats (CL #1-2), and two times with bags (BG #1-2) to obtain ten gait sequences in total. Overall, the CASIA-B dataset contains 13,640 sequences. Following [10, 2], 74 persons are used for training and the rest 50 persons are leaved for test. For each subject of the testing set, the first four normal sequences are registered as the gallery while the rest sequences are probes. The OU-LP dataset has 4,016 subjects of which 3,836 subjects are adopted for the cross-camera person identification task. For each subject, two gait sequences are captured by cameras and split into four subsequences based on viewpoints to the cameras (i.e., $55^{\circ}$, $65^{\circ}$, $75^{\circ}$, and $85^{\circ}$). In our experiment, the 60% of the 3,836 subjects are used for training and the rest are for testing. For each subject in the testing set, the first sequence is registered as the probe and the other is the gallery.

Evaluation Metric. The evaluation is performed in a cross-dataset manner. So, the model is trained with the training set of CASIA-B with labels and that of OU-LP without labels while is evaluated on the testing set of OU-LP, and vice versa. During testing, we compute the Euclidean distance between each probe and all gallery, which is similar to person Re-Id [33]. The rank-1 accuracy for each probe is calculated, while the overall result is measured by the average rank-1 accuracy over all probes.

Implementation Details. In our experiments, we adopt GaitSet as the backbone to be trained on the source dataset using settings in [2] and [7]. The margin in Equ. 2 is set as $0.2$. Especially, a batch with size of $p\times k$ is sampled from the training set where $p$ denotes the number of persons and $k$ denotes the number of training samples for each person. We set the batch size as $8\times 16$ and $16\times 8$ for CASIA-B and OULP, respectively. For the unsupervised learning on the target dataset, the number of nearest neighbors, K, in Equ. 3 is set as 1. The $\tau$ of $p_{i,j}$ in Equ. 5 is set as 0.1. During model updating with neighborhood discovery, we train the backbone in $R=4$ rounds and 200 epochs per round. The initial learning rate is set to $10^{-5}$ and scaled-down by 0.1 in every 40 epochs after the 80-th epoch. The experiments were performed on four Tesla P40 GPUs.

We compare TraND with seven methods: 1) Supervised GaitSet [2]: the GaitSet trained on target dataset in the supervised manner as the upper bound; 2) Direct Testing: directly applying the model trained on the source dataset to the target testing set; 3) GaitSet with clustering methods: GaitSet + DBSCAN [5] and GaitSet + k-means [4]; 4) GaitSet + AND: directly using AND to train the target domain in an unsupervised method with Gaitset as the backbone [8]; 5) Variants of TraND: TraND (Random entropy) and TraND (Low entropy) which select neighborhoods with random entropy and low entropy, respectively.

The experimental results are listed in Table I. From the results, we can first find that directly applying the model trained on the source data to the target data obtains very poor results compared with the model trained on the target data. It shows that the domain gap among different datasets can greatly affect the adaptation capability of the model. Moreover, the clustering-based methods, i.e., GaitSet + DBSCAN and GaitSet + k-means, cannot work well for this task. This is because the pseudo labels generated by clustering algorithms may be misleading, which brings much noise during fine-tuning the model. Furthermore, by comparing different neighborhood selection strategy, our method achieves the best performance. This reflects that our method can effectively discover confident neighborhoods with the class consistency indicator and high-entropy-first strategy. However, when transferring the model trained on OU-LP to the conditions of BG and CL on CASIA-B, all methods obtain poor results. This is because OU-LP has no samples with a bag or coat, which makes models learn nothing about this knowledge. Therefore, the cross-condition settings should be further studied to improve the performance.