GRM: Gradient Rectification Module for Visual Place Retrieval

Early approaches to extract global feature descriptor are based on aggregation of hand crafted local features. Methods are like Bag of Words (BoW) [13] and Vector of Locally Aggregated Descriptors (VLAD) [14]. Sparse keypoint location or dense sampling on image are applied to aggregate local features. When deep learning technology is introduced into these frameworks, new methods like NetVLAD [5], NetBoW [15], NetFV [6] emerge. Other methods apply simple pooling layer, eg. max pooling or GeM pooling [16], on top of the CNN architecture [12, 4]. These methods are trained with Triplet Loss [5], or ranking-based loss [17, 18]. Most recently, [4] proposes to use a continuous measure to calculate the similarity between images. Instead of depending on the binary relation, they use the calculated similarity score to train the neural network and achieve huge success in MSLS challenge [19].

Early methods for refinement are separated from global feature matching scheme. Local features like SIFT[20], SURF[21] are extracted and matched based on nearest neighbor criterion. Then model-based methods, such as RANSAC[22] and PROSAC[23] are used to generate transformation hypotheses based on correspondences. The final retrieval result is re-ranked by the number of inliers under the hypothesis. DELF[3] proposes to use the feature map before pooling for later refinement process. PatchNetVLAD[1] proposes to extract NetVLAD[5] descriptors on image patches for later refinement procedure.[2] is the first to jointly learn local and global features for retrieval task.

According to [24], when the features become white, the Fisher information matrix reduces to an identity matrix. In that case, the simple stochastic gradient descent optimization coincides with the natural gradients descent optimization which is more suitable for difficult optimzation tasks. Whitening methods can be classified into two categories. The first category is to add extra penalty terms in loss function. Zhang et al. [8] proposed to add a regularizor to spread vectors uniformly in the feature space. In [9], the author provides a thorough analysis of the uniformity and alignment metric of the feature distribution space. Additional loss penalties are added to adjust the uniformity and alignment of feature space. Another method is to process the features directly, either in the forward pass or the backward pass. Batch Normalization (BN) [10] is the most widely used method in the forward pass of deep learning model. Later, [11] constructs the whitening matrix based on zero-phase component analysis to mitigate the stability issue in BN and PCA [25]. [12] appends a PCA projection layer behind the network to spread descriptors.  [26] finds that the whitening matrix exerted on features can be instead applied on model weights. They prove that their methods are mathematically equivalent to ZCA [11] but the computation complexity is largely reduced. WADAM [27] extends the idea in [26] with momentum, adaptive dampening and proposes new optimizers for network training.

In VPR, [7] points out that whitening can balance out the co-occurrence of features, which is generally beneficial to retrieval tasks. Whitened features can boost the performance of the network on the current datasets and improve the generalization ability of the network across datasets [4]. However, Whitening adds a strong prior on the distribution of the descriptors and is not involved in the training process.

Our work focuses on the global descriptor extraction procedure, since the quality of the global descriptor is the bottleneck of later local descriptor matching process. Unlike previous works in VPR  [4, 17, 28, 5], we do not propose new aggregation methods or new loss functions, but instead focus on the feature distribution that lacks attention. We propose a new method to optimize the feature distribution, which is parallel to most VPR methods before-head.

We claim that pair-wise loss function in VPR is one of the reasons for the degraded distribution of gradients. A general pair-wise loss function can be written as

where $s_{i,j}$ is the similarity between query $i$ and sample $j$. It is usually defined as cosine similarity or L2 distance. These two are equivalent when the descriptors are normalized. For simplicity, we will use the L2 distance.

$\vec{p}_{i}$ and $\vec{p}_{j}$ are the descriptors encoded by the network. The gradient of the descriptor is

Since $2\frac{\partial f}{\partial s_{i,j}}$ is a scalar, Eq. 3 can be written as

Here, we use contrastive loss as an example. The contrastive loss (CL) is shown in Eq. 5.

$\phi_{i,j}=1$ if sample i and j are positive pair and 0 otherwise. $\tau$ is the threshold. The derivative is

In Eq. 6, since $\phi_{i,j}-(1-\phi_{i,j})sign(\tau-s_{i,j})$ is a scalar, the gradient is a linear combination of the query and database descriptors. Since the projection of descriptors in the principal space is much larger than the projection in the complement space, a linear combination of these principal components is generally larger than other components. Therefore, gradients will also have large components in the principal space.

The whole process can be summarized in Fig. 3. During training, we feed images into the network to get descriptors. These descriptors are used to update the estimated covariance matrix $P$. Loss is computed according to the learning task, for instance, GCL [4] or Triplet Loss. Then we backward the loss and capture gradients at the descriptor level. The projection matrix $P^{*}$ is applied here and the modified gradients flow into the network.

We use GCL Loss [4] to train our model. We select ResNet50 [33] and ResNext101 [34] as backbone to extract features. GeM pooling is applied to get the global descriptor. The memory queue size (K) is 10,240. We use Adam optimizer[30] and set the learning rate to 1e-4. We train our model for 200 epochs (10k pairs in MSLS per epoch). Following GCL [4], we use the weights pretrained on ImageNet and only train the last two blocks of the backbone. For GRM, the rectification rate is set to 1. We train our network on MSLS training set and test them on other datasets. Top-N recall (R@N) is adopted as evaluation metric.

For a fair comparison, we only compare the first stage of all methods without the re-ranking process. We compare the performance of GRM with state of the arts in Table I. On MSLS dataset, even though the baseline methods, GCL-ResNet50 and GCL-ResNext101, are superior than all the other methods, our methods can still boost the performance. ResNext101 trained by our method can achieve the best R@5 in the official MSLS challenge, at 77.2%. For ResNet50, the R@5 increases nearly 10%, and is comparable to the larger ResNext101 model. Our methods can also bring huge improvement on Nordland dataset. This is due to the fact that season variation consists a large portion of the MSLS dataset. Therefore, our methods trained on MSLS generalize well on Nordland dataset, increasing over 5% on ResNext101 and over 30% on ResNet50. In urban environment, Pittsburg 30K and Tokyo24/7, our method can bring moderate improvement on the baseline method, between 1~2%, compared to the baseline method. Our best R@5 on Pittsburg30K (91.9%) and Tokyo24/7 (75.2%) is lower than PatchNetVLAD (94.5% on Pittsburg30K and 88.6% on Tokyo24/7). This is because Patch-NetVLAD is fine-tuned on Pittsburg30K training dataset, with images including drastic variations in illumination. However, in order to keep aligned with other methods, we only train our method on MSLS dataset.

We compare the performance of GRM and other feature whitening methods. We select GCL [4] as our baseline and use multiple strategies to expand the feature space. For Spread Loss [8]. We set the weight for the regularization term to 1. For AP Loss [17]. We use the GCL Label in [4] to sample 2 positive images and 7 negative images. Negative images are half hard negative (similarity score greater than 0 but less than 0.5) and half negative (similarity score equals 0). We compare our method to [27]. Standard Adam [30] optimizer is replaced by WADAM  [27] in the GCL [4] framework.

The training procedure follows the previous section and we show the R@N index on MSLS val and test sets in Table II. GRM exceeds all methods in Table II. The performance drops when adding the extra regularization in [8]. We conjecture that the inclusion of regularization might break the original distribution and leads to a sub-optimal distribution. As we can see, AP Loss [17], as the representation of multi-pair form, increase R@5 by 4% on MSLS test set but is still lower than our method. Meanwhile, AP Loss suffers from large memory consumption and long training time. As for the WADAM [27], it can bring slight improvement on MSLS val., but when evaluated on test set, it under-performs the baseline method.

The result corroborates that it is neither the weak repulsive force nor the limited batch size that causes the unbalanced distribution of descriptors but the phenomenon that gradients and descriptors fall into the same low dimensional principal space. Since gradients won’t push the descriptors out of this principal space, descriptors will dwell in this principal space and never be uniformly distributed.