Unsupervised Complementary-aware
Multi-process Fusion for Visual Place Recognition

Fusing multiple sources of information is a common technique in both SLAM and VPR [14, 1]. Traditionally this has been achieved through the use of multiple physical sensors, like LiDAR, depth cameras, RADAR and Wi-Fi [15]. Numerous algorithms have been devised to intelligently fuse these different sensors and produce a more robust localization estimate [11, 10].

Recent work has shown that some of the benefits of multi-sensor fusion can still be attained with a single sensor, by fusing different interpretations of the image data that are obtained by applying different image processing techniques on the same image. SRAL [8] showed that a Convex Optimization problem can be used to optimize the contribution weights of a set of fused image processing modalities, resulting in improved VPR performance. The optimization was computed for a separate training set for each test dataset. Multi-process fusion [9] fuses multiple modalities within a Hidden Markov Model, combining the benefits of both sequences and multiple techniques.

More recently, evidence has been accumulating that certain combinations of modalities are more effective than others on specific datasets [6, 12]. However, both these works rely on either a training set, or access to the full ground-truth information for a given dataset. In a different work, rather than using traditional image data, [16] fuses different temporal windows of event camera data for improved VPR.

In this work we propose a method for selecting the optimal subset of techniques to fuse in a multi-process fusion system. Subset selection (combinatorial optimization) has had previous investigation in the sensor fusion literature [17]. In that work, the sensor selection problem was relaxed from a binary selection to a soft weighted fusion, which allowed for the use of convex optimization theorems to approximate the solution to the problem without requiring a brute-force search of all combinations of sensors.

In VPR specifically, [18] used a Greedy algorithm to find the optimal set of feature maps to use to improve localization performance, and further work used mutual information to determine the ideal subset of feature maps to use [19]. A recent work [20] intelligently selected the optimal reference set of images to use in VPR, by using Bayesian Selective Fusion. Given a set of reference images taken at different times, their algorithm found both an optimal subset of reference times and also weighted the different sets to optimize the localization performance.

[21] used change removal to detect and remove common conditions across an environment (since common conditions are not useful at identifying unique locations [22]). Another recent work proposes an unsupervised algorithm for automatically selecting the optimal parameters for VPR algorithms [23], although only one technique was at a time. Our work can be considered a search through a large set of image representations to find the optimal subset of representations to fuse, while removing misleading representations.

In VPR, each query image observed by the autonomous platform is compared against a database of $D$ prior images using a specified feature extraction and matching technique. This produces a $D$ dimensional similarity vector, containing a list of similarity scores between the query image and each database image. A larger score denotes a stronger VPR match.

In a multi-process fusion (MPF) algorithm [9], a variety of different VPR techniques are used simultaneously and a fusion of similarity scores can improve the overall VPR performance of a system. Formally, given a set of techniques $\mathbf{N}=\{1,\dots,N\}$, each technique $n\in\mathbf{N}$ separately computes a similarity vector $D_{n}$ containing the similarity scores between a query image and a set of database images. Since the set of techniques is arbitrary, the distribution of scores within each technique will not be consistent with the distribution in other techniques. Therefore, we use a normalization process prior to fusing the set of $\mathbf{N}$ techniques, so that each similarity vector has a minimum value of zero and a maximum value of one:

$$\hat{D}_{n}(i)=\frac{D_{n}(i)-\min(D_{n})}{\max(D_{n})-\min(D_{n})}\quad\forall i,\quad n\in\mathbf{N}$$

(1)

The collection of similarity vectors $\hat{D}$ are then summed to produce a combined similarity vector:

$$D_{\text{MPF}}=\sum_{n=1}^{N}\hat{D}_{n}$$

(2)

The matching image $X$ is then the image with the maximum score in $D_{\text{MPF}}:X=\operatorname*{argmax}D_{\text{MPF}}$.

While such MPF algorithms have previously been shown to outperform single technique baselines, typically only a sub-selection of techniques will be suited to a particular dataset. The drastic difference in performance between different datasets and even within different sections of the same datasets of particular methods was shown in [7, 24]. Therefore, the inclusion of poorly performing techniques (which is the case for standard MPF) can potentially offset any advantages of using multiple methods simultaneously. Instead, in the following we describe how the best set of techniques can be found dynamically, online and without requiring training.

In this subsection we begin by describing how a pair of techniques can be selected, from the set of all possible combinations of pairs out of a collection of individual techniques. Our selection criteria is based on an estimate of the perceptual aliasing of a similarity vector. In the subsequent section we describe how our approach can be scaled to larger subsets of techniques.

The intuition behind our proposed dynamic multi-process fusion is as follows. The aim of any VPR system is to find the ground-truth match in the reference database given a query image. The difficulty lies in the problem that in a sufficiently large reference database there will be images that have similar appearance that can cause mismatches – an effect called perceptual aliasing. Our hypothesis is that there are complementary techniques in which the locations of perceptual aliasing differs, but which have correlated similarity scores around the ground-truth database image. Therefore, by estimating the perceptual aliasing of different sets of fused techniques, we can find the set of techniques that minimises this aliasing.

As established in prior literature [13, 9], this estimate can be computed by finding the ratio between the similarity score at the best matching database image (the best guess for the ground-truth database image) and the score at the next highest similarity score in the database (the perceptually aliased image) – a larger ratio denotes reduced perceptual aliasing. Unlike prior literature, which calculates this ratio score for individual techniques, we calculate this ratio for similarity vectors that are the addition of individual technique vectors.

We thus estimate the perceptual aliasing score for a pair $(k,l)$ of techniques by finding the ratio between the maximum value in the summed vector and the next largest value outside a window around the maximum value. As per:

$$\hat{D}_{k+l}=\hat{D}_{k}+\hat{D}_{l}$$

(3)

$$S_{k,l}=\frac{\max(\hat{D}_{k+l})}{\max(\tilde{{D}}_{k+l})}\quad k\in\mathbf{N},\ l\in\mathbf{N},\ k\neq l$$

(4)

where $\tilde{D}$ denotes the subset of $\hat{D}$ that excludes all values $\pm R_{\text{window}}$ around $\operatorname*{argmax}(\hat{D})$. The chosen set of techniques $(k^{*},l^{*})$ is the pair with the highest score $S_{k^{*},l^{*}}$.

This scoring algorithm can be scaled up to analyze any larger combination of techniques. To do so, we find the maximum score across any combination and number of techniques allowable by the total number of techniques, all the way up to a sum of all techniques. Formally, the power set $P(\mathbf{N})$ includes all possible subsets of the set $\mathbf{N}$. We exclude the empty subset and all subsets containing only a single technique. The number of subsets is thus given by $M=2^{N}-N-1$.

We then re-write Eq. (3) in the general case:

$$\hat{D}_{\mathbf{M}}=\sum_{m\in\mathbf{M}}\hat{D}_{m}\quad\mathbf{M}\in P(\mathbf{N})$$

(5)

For each $\hat{D}_{\mathbf{M}}$, we then calculate the score $S_{\mathbf{M}}$ following the principle outlined in Eq. (4) and find the set of techniques $\mathbf{M}^{*}$ with the maximum score:

$$\mathbf{M}^{*}=\operatorname*{argmax}(S_{\mathbf{M}})\quad\forall\ \mathbf{M}\in P(\mathbf{N}).$$

(6)

While this extensive approach that compares all possible variations is guaranteed to find the set of techniques that maximizes the ratio score, we note that such a search will become intractable with large values of $N$ ($N>>10$). For such cases, the problem can be approximated by considering the technique selection problem as a convex relaxation selection process, which can be solved using interior point methods [17]. We leave such a solution to future work.

As our complementary scoring technique does not require access to ground truth data, the calibration can be performed at deployment time and be continuously updated to reflect changes to the observed environment during deployment. We re-run the calibration procedure every $F$ frames, where a larger value of $F$ reduces the computational requirements. For query frames that are not used for calibration, rather than extracting features for all $N$ techniques, instead we only extract and match features for the best subset techniques decided by the calibration.

A place recognition decision is determined based on the maximum score in the summed similarity vector, for the sum of the set of complementary techniques as decided by the online calibration. The sum of techniques is weighted by a set of technique weighting scores $w_{m}$:

$$\overline{D}=\sum_{m\in\mathbf{M}^{*}}w_{m}\hat{D}_{m},$$

(7)

where $\mathbf{M}^{*}$ is the set of techniques chosen by the calibration procedure. The values of $w_{m}$ are determined using the same ratio calculated as shown in Eq. (4), except substituting $D_{k+l}$ for the single techniques $\hat{D}_{m}$. This weights techniques based on their individual confidence in their place match estimate, on a frame by frame basis.

We finally normalize this summed similarity vector to have a mean of $1$ and a standard deviation of $0$, following [25]. The best matching database index is found by:

$$\overline{X}=\operatorname*{argmax}\overline{D}$$

(8)

In Table I, we show the recall@1 of dynamic MPF versus the fair baselines, for all datasets. While the localization performance varies across the datasets, on average, dynamic MPF outperforms all baselines with a Recall@1 of 79% versus a Recall@1 of 74% for Full MPF. In particular, Dynamic MPF outperforms Full MPF by large margins on the datasets Pittsburgh, ESSEX3IN1 and Tokyo247. This is because some individual techniques perform poorly on these datasets, thus reducing the overall effectiveness of the recall performance of the Full MPF baseline. In fact, on the Pittsburgh dataset five techniques out of ten have a Recall@1 less than 5%. Dynamic MPF is selecting the best subset of techniques for each query image, which prevents poorly performing individual techniques from negatively impacting the VPR system.

Another advantage of dynamic MPF is the consistency in the recall across the datasets – compared against all the benchmarks without the hindsight advantage, dynamic MPF is either the best performing system or a close second best performing system. Across the datasets, dynamic MPF has the best performance on the Tokyo247 dataset, with a Recall@1 of 78% versus the benchmarks with 34%, 67% and 41% for Random pair, Full MPF and Hier MPF respectively. We hypothesize that the combination of severe appearance change (day to night) and severe viewpoint change requires a VPR solution that can suit the given query image conditions. In the Tokyo247 dataset the viewpoint and appearance change is not consistent across the dataset, therefore a single, static, technique is unable to reliably perform place recognition.

We observe that Dynamic MPF is less effective on more consistent datasets, where the appearance of the environment does not change much over the query set. This is especially the case for the Nordland and Corridor datasets, with either a static season or the same indoor building respectively. As shown in Table I, on these datasets dynamic MPF does not outperform Full MPF (however is very close in performance with a drop of just 1 and 2% Recall@1 respectively).

In Table II, we compare our approach with three oracle benchmarks, all utilizing prior knowledge of the datasets to make decisions on the best techniques to use. These benchmarks are the best performing individual technique, the fusion of NetVLAD and DenseVLAD (NV+DV), and the fusion of the four techniques NetVLAD, DenseVLAD, RegionVLAD and Ap-GeM (NV+DV+RV+AP). In a real deployment situation, such prior knowledge would not be available.

Even though these baselines have this advantage, dynamic MPF still outperforms all of them, with a mean recall of 79% versus the second highest benchmark NV+DV+RV+AP with a recall of 78%. The main cause of reduced performance for the pair and quadruplet of techniques is the poor performance on the Nordland dataset; there dynamic MPF continues to provide high performance by utilizing other techniques that function well on this dataset. The best individual technique (again as determined on hindsight via an oracle) is a strong competitor (that would not be available on deployment), and yet the ensemble of multiple complementary techniques significantly outperforms this baseline on a number of datasets, such as Gardens Point, Tokyo247 and SPEDTest.

To evaluate the effect of varying the calibration frequency, we ran multiple experiments with different frame separation values $F$. Calibrating the algorithm every $F$ frames is only sensible on sequential datasets, that is, where each query image follows the previous image. The VPR-Bench datasets that follow this format are Gardens Point Walking, Nordland, Synthia, and Corridor.

Figure 2 shows the effect of varying the value of $F$. One can observe a gradual reduction in localization performance as the frame separation is increased, with different rates of reduction depending on the dataset. The Synthia dataset is an exception, which maintains recall performance even at large frame separation values. We hypothesize that the synthetic nature of this dataset means that a consistent subset of techniques is generally applicable for all the query images. For all datasets, it can be observed that even at large values of $F$, the recall only reduces by a moderate amount and the algorithm can still identify an approximately applicable subset of techniques for the environment.