Intelligent Reference Curation for Visual Place Recognition via Bayesian Selective Fusion

Arguably the most prominent probabilistic approach to recognizing places is FAB-MAP [19]. One of FAB-MAP’s main innovations was to explicitly account for perceptual aliasing, such that highly similar but indistinct observations receive a low probability as being the same place; a generative model of bag-of-words observations implemented this. FAB-MAP has been extended in multiple ways, including the incorporation of odometry information and image sequences [20, 21].

While FAB-MAP relies on local image features, Lowry et al. [22] present a probabilistic model for whole-image descriptors, which leads to greater robustness in environments with large perceptual changes. Ramos et al. [23] propose a Bayesian framework for place recognition that leverages 5-9 training images per place and can then recognize a place from previously unseen viewpoints. Recently, Oyebode et al. [24] have proposed a Bayesian framework that leverages pre-trained object detectors to recognize indoor places based purely on object categories, which is related to the concept of visual place categorization [25], i.e. predicting the semantic category of a place.

Dubios et al. [26] use Bayesian filtering to model the dependency between sequences of images. Similar probabilistic temporal sequence modeling has also been investigated in [6] and in [27] using a Monte Carlo-based algorithm that achieves state-of-the-art performance on several datasets. In contrast to these methods that use sequences to temporally fuse place information, our VPR approach operates on single query images and fuses multiple reference images.

Our work is also related to approaches that preprocess query images to match conditions in reference images (or vice versa). For example, Pepperell et al. [28] and recent domain translation methods (cf. [13]) cast query images into the same visual conditions as the reference images via sky removal or night-to-day image translations (see also [29], [1, Section VII-A] and references therein). However, these approaches still raise the question of which reference images and translations are best to use for a given query image. For example, night-to-day translation implicitly assumes that query images are known to have been captured at night, and is unnecessary and could degrade performance on daytime query images.

Kosecka et al. [9] described indoor places with a set of representative views and then found the most likely reference image using a maximum voting scheme. Carlevaris-Bianco and Eustice [30] proposed an extension to that approach by learning temporal observability relationships between the representative views. In a similar vein, the approach by Johns and Yang [31] learns feature co-occurrence statistics, such that each place is described using a set of co-occurring features. This allows for reliable matching of images that were captured at different times of the day.

When robots are deployed for prolonged periods, it is common to update the environment model over time. Biber and Duckett represent the environment using multiple timescales simultaneously, whereby older memories are faded at a rate that depends on the particular timescale being used [32]. For each place, the timescale with the highest log-likelihood is chosen. A more sophisticated approach in robot navigation is to store only persistent configurations in a long-term memory, so that dynamic objects do not become part of the map [33]. Another approach is the use of coresets [12] to drastically compress the images that represent a place. These coresets have subsequently been used to detect loop closures in real-time [34].

Impressive results have been achieved using so-called experience maps [10]. The accumulation of multiple experiences for the same place allows for both adaptation to changing appearance due to lighting and weather conditions as well as structural changes. At localization time, the robot can then predict the most appropriate experience using a probabilistic framework. Selecting the most appropriate experience results not only in reduced computational time but also in significantly reduced failure rates. A similar method by Doan et al. [11] accumulates additional experiences for the same place while avoiding an unbounded growth of computation and storage. Vysotska and Stachniss [7] match places in a data association graph that can contain images of multiple reference traverses.

Further references to methods that contain multiple representations of the environment can be found in the survey on visual place recognition by Lowry et al. [1] in Section VII-B.2. While our method aims to leverage the complementary nature of multiple reference traverses, the multiprocess fusion approach by Hausler et al. [8] fuses predictions of multiple complementary image processing methods applied to the same input image. This also relates to [35], where appearance changes between the query and reference traversals are removed to some degree by using as few as 100 training samples. Contrary to [35], our method is training free. Our method is also related to [17], where in-sequence condition changes are investigated; however, only a single reference set is used.

Let $X$ be an unknown place at which a robot captures a query image $I_{X}$ and extracts a corresponding image descriptor vector $z_{X}\in\mathbb{R}^{N_{z}}$ with dimension $N_{z}$ (e.g., using NetVLAD [2]). Let $X$ belong to the set $\mathcal{X}=\{1,\ldots,N\}$ of possible places. The robot has access to reference images of the places in $\mathcal{X}$ captured during previous visits under different visual appearance conditions. These reference images are stored in a total of $M$ reference sets, with individual reference sets indexed by the scalar $u\in\{1,\ldots,M\}\triangleq\mathcal{U}$. Each reference set $u\in\mathcal{U}$ contains (at most) one image from each place in $\mathcal{X}$ (together with associated image descriptor). Let $I_{i}^{u}$ and $z_{i}^{u}\in\mathbb{R}^{N_{z}}$ denote the image and corresponding descriptor from reference set $u\in\mathcal{U}$ corresponding to place $i\in\mathcal{X}$. Given a reference set $u\in\mathcal{U}$, the robot can compute non-negative distances $d:\mathbb{R}^{N_{z}}\times\mathbb{R}^{N_{z}}\mapsto\mathbb{R}_{\geq 0}$ between the descriptor $z_{X}$ of the query image $I_{X}$ and the descriptors $z_{i}^{u}$ from the reference set $u$ for each place $i$ in $\mathcal{X}$. We collect the distances corresponding to reference set $u$ in the vector

We shall use $D_{i}^{u}$ to denote the $i$th component of $D^{u}$, i.e. $D_{i}^{u}\equiv d(z_{X},z_{i}^{u})$, and without loss of generality, we consider the Euclidean distance for $d(z_{X},z_{i}^{u})$ (other distance measures could also be used). The VPR problem is to infer the place $X$ using the vectors $D^{u}$ from some or all of the reference sets $\mathcal{U}$.

The vast majority of recent VPR approaches rely on the minimum-value principle (or best-match-strategy) [6, 22]. The principle asserts that the best place match $\hat{X}$ corresponds to the minimum descriptor distance in each reference set, i.e.,

for any $u\in\mathcal{U}$. The minimum ensemble distance approach of [18] is a recently proposed state-of-the-art method based on the minimum-value principle that fuses information from multiple reference sets. In this minimum ensemble distance approach, place matches $\hat{X}$ are found by minimizing the average (equivalently the total) of the descriptor distance vectors $D^{u}$ over all reference sets, namely,

Clearly, VPR approaches based on the minimum-value principle perform poorest when the place that minimizes the descriptor distances is not the true place — a situation which for example might occur when the query and reference images are captured under different appearance conditions. Whilst their robustness can be improved by fusing reference sets captured under a variety of appearance conditions using addition or averaging (as in Eq. (3)), these operations implicitly assume that all reference sets provide equally useful information.

We hypothesize that reference sets will have different utility for different query images and that this utility will be inherently uncertain. Therefore, we propose a novel Bayesian selective fusion approach that: 1) selects informative reference sets based on the minimum-value principle; and, 2) fuses the selected reference sets using Bayesian fusion to handle the uncertainty of the minimum-value principle holding.

Given the distance vectors $D^{u}$ from all reference sets $u\in\mathcal{U}$, we select a variable subset of reference sets, defined as

with which to compute a place decision. The collection $\mathcal{S}$ is formed by first selecting the “best” reference set under the minimum-value principle, that is, the reference set $u^{*}$ that contains the minimum distance such that

All reference sets $u\in\mathcal{U}$ with minimum descriptor distances $\min_{i\in\mathcal{X}}D_{i}^{u}$ within a fraction $\gamma>0$ of the minimum descriptors distance $\min_{i\in\mathcal{X}}D_{i}^{u^{*}}$ of the reference set $u^{*}$ are also added to $\mathcal{S}$. Our reference set selection approach resembles outlier rejection with the parameter $\gamma$ controlling the rejection of reference sets from $\mathcal{U}$ with minimum descriptor distances that are outliers compared to the “best” reference set $u^{*}$.

Given the descriptor distance vectors $\mathcal{D}^{\mathcal{S}}\triangleq\{D^{u}:u\in\mathcal{S}\}$ from the selected reference sets $\mathcal{S}$, the Bayesian belief $P(X|\mathcal{D}^{\mathcal{S}})$ over the places in $\mathcal{X}$ is given by Bayes’ rule:

for $i\in\mathcal{X}$. Any prior knowledge that the robot has about its location (arising for example from its motion model or previous place matches) is incorporated here through the prior probability distribution $P(X)$. In the absence of any prior knowledge, this is taken as uniform (i.e. $P(X=i)=N^{-1}$ for all $i\in\mathcal{X}$). The likelihood $P(\mathcal{D}^{\mathcal{S}}|X)$ enables our novel fusion of the place information from the selected references $\mathcal{S}$.

To make the novel fusion operation in Eq. (5) explicit (and scalable), note that the descriptor distances $D_{i}^{u}$ are solely functions of the reference image descriptors $z_{i}^{u}$ after $X$ (and hence $I_{X}$ and $z_{X}$) is given. The vectors $D^{u}$ from different reference sets are thus conditionally independent given $X$, and the likelihood $P(\mathcal{D}^{\mathcal{S}}|X)$ simplifies to the product:

for $i\in\mathcal{X}$ where $P(D^{u}|X)$ are the likelihoods for the single (individual) reference sets $u\in\mathcal{S}$. Constructing the single-reference likelihoods $P(D^{u}|X)$ is easier and more scalable than constructing the joint likelihood $P(\mathcal{D}^{\mathcal{S}}|X)$ as we shall discuss in the following subsection. Here, we note that since the likelihood $P(\mathcal{D}^{\mathcal{S}}|X)$ in Eq. (6) has a product form and the denominator in Eq. (5) serves only to normalize, we may rewrite Eq. (5) as simply:

for $i\in\mathcal{X}$. The robot’s place recognition decision with our approach, denoted $\hat{X}$, is then the place with the maximum (unnormalized) Bayesian belief, namely,

No place recognition decision is made if the belief fails to exceed a threshold $h>0$, which balances recognizing the wrong place (small $h$) with missing the true place (large $h$). We next describe a novel efficient construction of the single-reference likelihoods $P(D^{u}|X)$ used in Eq. (8).

To construct the single-reference likelihoods $P(D^{u}|X)$, note that by definition they are the joint likelihoods of the distances, i.e., $P(D^{u}|X)\equiv P(D_{1}^{u},\ldots,D_{N}^{u}|X)$. Note also that in order for the distances $D_{i}^{u}$ to all be equally useful, their (marginal) distribution should depend only on whether the places $X$ and $i$ match or not, and not on the specific places $X$ and $i$. Hence, we model the distances $D_{i}^{u}$ as conditionally independent given $X$ and distributed according to a (marginal) distribution $P(D_{i}^{u}|X\neq i)$ when the place $X$ does not match $i$ and a different (marginal) distribution $P(D_{i}^{u}|X=i)$ when the place $X$ matches $i$. The (joint) likelihood of any place $i$ after observing the vector $D^{u}$ from reference set $u$ is thus the product of the (marginal) likelihoods after observing the individual distances, namely,

for $i\in\mathcal{X}$. Equivalently, we may write this likelihood as

for $i\in\mathcal{X}$. Here, the proportionality constant $C^{u}\triangleq\prod_{j=1}^{N}P(D_{j}^{u}|X\neq j)$ is constant for all $i\in\mathcal{X}$, which allows us to simply ignore it in our approach (cf. Eq. (8)).

Place-match and non-place-match likelihoods, $P(D_{i}^{u}|X=i)$ and $P(D_{i}^{u}|X\neq i)$, have previously been constructed via training for (whole-image) descriptors (cf. [22, 1]). These approaches can be used directly to evaluate Eq. (9), however we propose an alternative training-free approach based on a probabilistic encoding of the minimum-value principle (Eq. (2)). For a given reference set $u$, we model the place-match likelihood $P(D_{i}^{u}|X=i)$ as being proportional to the number of places with descriptor distances larger than $D_{i}^{u}$ in $D^{u}$, i.e.,

where the indicator function $\mathbb{1}\{D_{i}^{u}\leq D_{j}^{u}\}$ takes a value of $1$ when $D_{i}^{u}\leq D_{j}^{u}$ and is zero otherwise. Again, the proportionally constant is independent of $i$, and can be ignored.

We model the non-place-match likelihood $P(D_{i}^{u}|X\neq i)$ by recalling that the distribution of descriptor distances is independent of the place, and hence most elements in the vector $D^{u}$ are realizations of the descriptor distance with $X\neq i$. Due to the computational efficiency of computing the mean $\mu$ and variance $\sigma^{2}$ of the elements in $D^{u}$, we simply model the non-place-match likelihood as the Gaussian $P(D_{i}^{u}|X\neq i)=\mathcal{N}(D_{i}^{u};\mu,\sigma^{2})$ (accepting that some bias will be incurred due to one element $i$ of $D^{u}$ corresponding to $X=i$).

In summary, our proposed approach first selects reference sets according to Eq. (4), then computes the single-reference likelihoods given by Eq. (9) using Eq. (10) and our Gaussian construction of non-place-match likelihood, and finally declares a place recognition decision via the maximization in Eq. (8).

We use two, widely used, established benchmark datasets.

The Nordland dataset [36] consists of four recordings of a train traversing a 728km long route in Norway captured in spring, summer, fall and winter. We extract one image per second from a one hour and twenty minute long section of these videos (20:00 to 1:40:00) and, as typically done in the literature, manually remove tunnels and sections where the train is stationary or traveling at speeds below 15km/h (based on the available GPS information). This resulted in a total of 3000 frames per season. As frame $i$ in one of those videos corresponds exactly to frame $i$ in the other videos, we find ground-truth correspondences by matching the query traverse frame number with the reference traverse number and allow a ground-truth tolerance of $\pm 2$ frames. The Nordland dataset has been widely used in the literature, e.g. in [8, 4, 35].

The Oxford RobotCar dataset [37] contains over 100 traversals of a consistent route through Oxford, captured at different times of the day and in varying weather conditions. Recently, centimeter-accurate ground-truth annotations (based on post-processed GPS, IMU and GNSS base station recordings) were made available for a subset of these traversals [38]. We selected five traversals of the same route¹¹1Dusk: 2014-11-21-16-07-03, Night: 2014-12-16-18-44-24, Overcast: 2015-05-19-14-06-38, Sun: 2015-08-12-15-04-18, Rain: 2015-10-29-12-18-17 and subsampled them such that places $i$ and $i+1$ have a regular spatial separation of one meter between them, which resulted in 3350 frames. We used images recorded using the front left stereo camera. Similarly to the Nordland dataset, the Oxford RobotCar dataset is widely used in the literature, e.g. see [8, 4]. As in [4], we use a ground-truth tolerance of $\pm 10$ meters.

We computed NetVLAD [2] descriptors for the images extracted from the datasets. Following [2] and as has become standard in subsequent studies (cf. [18, 4]), the descriptors were reduced to 4096 components via PCA. We use these NetVLAD descriptors for $z_{X}$ and $z_{i}^{u}$ in all experiments except for one comparative study (Section V-F), for which we also compute 4096-d DenseVLAD [3] descriptors.

We use the state-of-the-art minimum ensemble distance approach of [18] (see also Eq. (3)) as the Baseline Fusion approach for comparison with our proposed Bayesian Selective Fusion approach. The reference sets used by these fusion methods constitute the remainder of the traversals on the relevant dataset after removing the query traversal (i.e., if the query is Nordland Summer, then the reference sets are Winter, Fall and Spring). We also implement standard single-reference NetVLAD (cf. Eq. (2)) and versions of our Bayesian approach with single reference sets (i.e. $M=1$). These single-reference methods are labeled according to their reference set (e.g., given only a Nordland spring reference, NetVLAD is termed NetVLAD Spring and the single-reference Bayesian approach is termed Bayesian Spring or simply Spring).

We report VPR performance using Precision-Recall (PR) curves (with the threshold $h$ swept for our proposed method). Precision is defined as the ratio of correct place matches to the total number of place matches whilst recall is defined as the ratio of correct place matches to the total number of possible true matches. Our datasets have a true match for every query image. In some experiments, we also report the area under the PR curve (AUC) as a summary statistic. As in [14, 15, 16], the AUC serves as a proxy for recall at 100% precision (which is of direct concern for SLAM) for comparing single-image VPR methods since many offer no recall at 100% precision.

Our approach relies on a single parameter, $\gamma$ (see Eq. (4)). In our experience, values of $\gamma\in[0.04,0.1]$ provide reasonable performance on the Nordland and Oxford datasets. To illustrate this insensitivity, we use a fixed $\gamma=0.04$ in all experiments, noting that improvements could be obtained by tuning $\gamma$.

Fig. 2 shows that Bayesian approaches with our new training-free single-reference likelihood outperforms NetVLAD given the same reference set for query images drawn from the Nordland summer traversal. The Bayesian approach offers a $5.5\%$ (relative) improvement in AUC over NetVLAD when both perform their best (Fall reference) and a $70\%$ improvement when both perform their worst (Winter reference). Fig. 2 motivates the use of (selective) fusion since the performance of the methods is strongly dependent on the reference images they are provided. We thus now focus on evaluating the performance of our Bayesian Selective Fusion approach.

We next compare our Bayesian Selective Fusion approach with the state-of-the-art fusion approach of [18] (i.e., Baseline Fusion) and NetVLAD (unfairly) given the best performing reference on each traversal. Figs. 3 and V-B show that our approach outperforms the Baseline approach by a large margin on Nordland Summer and Winter query traversals with AUCs of $0.97$ and $0.84$ for our approach compared to $0.89$ and $0.68$ for the baseline method ($9\%$ and $24\%$ increase, respectively). Our approach performs slightly worse than the Baseline Fusion approach on the easier Oxford Sun traversal (with an AUC of $0.995$ compared to $0.997$) but outperforms it on the more difficult Oxford Night traversal, with an AUC of $0.77$ compared to $0.74$ ($4\%$ increase).

Our approach can be seen to select informative reference sets (also shown in Fig. 1) since it attains equal or better performance than the single-reference Bayesian methods. Such a scenario is shown in Fig. V-B(b) on Nordland Winter, where our method’s AUC is higher than the mean AUC of the single-reference Bayesian methods by $5\%$ and the best by $3\%$. Finally, our approach outperforms NetVLAD on all but the easiest case (i.e., Oxford Sun with an absolute difference in AUC of less than $0.003$); improvements in AUC range from $8.5\%$ on Oxford Night to $25\%$ on the more difficult Nordland Winter. Examples of correct matches from our approach and false matches from NetVLAD are shown in Fig. 10.

We now perform an ablation-type study to examine the performance of our approach with and without Bayesian fusion and/or reference set selection. We consider a Bayesian Full Fusion approach that omits reference selection; and, a Baseline Selective Fusion approach based on the Baseline Fusion method of Eq. (3) that uses only the selected reference sets $\mathcal{S}$. We also consider NetVLAD (advantageously) given the best reference on each traversal.

Figs. 5 and V-C show that only our proposed Bayesian Selective Fusion approach works consistently well without any modification or parameter tuning across six query traversals from the Nordland and Oxford datasets. All other approaches have instances of total failure or inferior performance. For example, the Baseline Selective Fusion method fails almost completely on the difficult Nordland Summer, Winter, and Fall traversals, whilst the Baseline Fusion method and NetVLAD perform poorly. The Bayesian Full Fusion approach compares the most favorably with our proposed Bayesian Selective Fusion approach but does not outperform it, illustrating the additional benefit of reference set selection.

Our reference set selection approach is motivated in part by the need for VPR systems to adapt in situations where the visual appearance of places can change dynamically and unpredictably such as indoors with a flickering light. For this study, we consider a scenario in which the query image at each place in the Oxford dataset is drawn randomly (with equal probability) from either the Sun or Night traversals.

The performance of our proposed approach on the randomized Sun/Night query traversal is shown in Fig. 7 together with the performance of the Dusk and Overcast single-reference Bayesian methods that perform best on the standalone Oxford Sun and Night traversals. We also consider NetVLAD with it again given the advantage of the best reference set (overcast). From Fig. 7, we see that Bayesian Selective Fusion outperforms the single-reference approaches and (the advantaged) NetVLAD with higher precision at recalls greater than $0.9$, and AUC improvements over (the advantaged) NetVLAD, Overcast, and Dusk of $1.0\%$, $6.6\%$, and $29\%$, respectively.²²2We expect worse performance from single-reference methods when query images can originate from more than two conditions (e.g. Raining Night/Day, Snowing Night/Day, Raining Dusk/Dawn, etc.) but we lack sufficient data to perform these studies.

Sequence matching techniques are commonly used in conjunction with single-image place matching for VPR under challenging environments. Fig. 8 shows that the performance of our approach on the randomized query from our previous study is improved via integration with an existing sequence matching approach (SeqSLAM [5]) with a sequence length of $5$. It also performs better than NetVLAD integrated with the same sequence matching approach despite NetVLAD having prior knowledge of the best reference set.

In our previous studies, we used NetVLAD [2] descriptors. Fig. 9 shows the results of repeating our study from Section V-B for Nordland summer and winter traversal queries with DenseVLAD [3] descriptors instead of NetVLAD descriptors. We see that our approach delivers a similar performance gain compared to the best single-reference Bayesian methods without any modification, parameter tuning, or training.

On an i7-8750H CPU, our approach took an average of $0.0224$s ($0.0002$s for reference selection and $0.0222$s for fusion) per query image with $M=3$ and $N=3000$ places. Complexity is linear in the number of references $M$, however, the expensive Bayesian operations (Eq. (8) and Eq. (9)) scale linearly with the number of selected reference sets in $\mathcal{S}$.