Saliency-Associated Object Tracking

Figure 2 illustrates the overall framework of the proposed SAOT, consisting of two core modules: Saliency Mining module and Saliency-Association Modeling module.

Taken as input an exemplar image and a search image in a tracking sequence, our SAOT first employs a Siamese feature extractor to learn deep representations $\bm{F}_{x}\in\mathbb{R}^{h_{x}\times w_{x}\times c}$ and $\bm{F}_{s}\in\mathbb{R}^{h_{s}\times w_{s}\times c}$ in the same feature space for the target exemplar (cropped from the exemplar image according to the bounding box) and the search image, respectively. Herein we adopt widely used ResNet [17] pre-trained on Imagenet [12] as the feature extractor due to its excellent performance of image feature learning.

The Saliency Mining module is designed to capture the local saliencies of the target exemplar which are discriminative for tracking. It calculates similarity maps to measure the pixel-to-pixel correspondences between $\bm{F}_{x}$ and $\bm{F}_{s}$, and selects local sharp maximum points as saliencies. These captured saliencies correspond to the most discriminative regions of the exemplar, which can be easily localized with high confidence and accuracy.

The captured saliencies are then associated together by the Saliency-Association Modeling module of SAOT to learn effective global correlation representations between the exemplar and the search image. The obtained correlation representations are expected to reflect the target state in the search image precisely by aggregating the distributions of all saliencies in the search image with the learned interactions between them. Finally, the target state is estimated by a classification head for confidence estimation and a regression head for predicting the bounding box of the target.

Typically, not all local regions of the target exemplar are easy to be tracked. Thus we design the Saliency Mining module to capture the saliencies corresponding to discriminative local regions of the target exemplar that can be easily localized in the search image.

The proposed Saliency Mining module performs saliency mining in two steps: 1) constructs similarity maps for each pixel in the feature maps of the target exemplar $\bm{F}_{x}$ to achieve the distribution of matching score in the search image; 2) measures the saliency value of each pixel in $\bm{F}_{x}$ based on the obtained similarity maps to select saliencies.

The intensity of a peak distribution is used to measure the relative strength of the maximum value compared to other values in the whole similarity map. A straightforward way to measure the intensity of a peak distribution is Peak-to-Sidelobe Ratio (PSR) [4] which is defined as:

Herein $\Phi$ denotes the sidelobe w.r.t. to a peak distribution in the similarity map $\bm{S}_{(u,v)}$, which is defined as the region of $\bm{S}_{(u,v)}$ excluding the neighboring region around the maximum point (referred to as main lobe $\Psi$). Here main lobe and sidelobe are defined to roughly indicate the relevant and irrelevant regions to the peak distribution around the maximum point, respectively. $\mu_{\Phi}$ and $\sigma_{\Phi}$ are the mean value and standard deviation of $\bm{S}_{(u,v)}$ of the sidelobe, respectively. In the initial definition [4], the size of main lobe $\Psi$ for arbitrary similarity maps is pre-defined as a fixed value. We argue that such a definition is unreasonable since the distribution characteristics of similarity maps are not taken into account. Figure 4 shows two examples of similarity maps with different peak distributions around the maximum points, which apparently correspond to different sizes of the main lobes. Instead of fixing the size of main lobe, we define the boundary of main lobe $\Psi$ as the closest contour around the peak, whose height value is equal to the mean value of the similarity map. Consequently, the intensity $\gamma$ of a peak distribution in the similarity map $\bm{S}_{(u,v)}$ is defined as:

where $\text{avg}(\bm{S}_{(u,v)})$ is the mean value of the similarity map.

Another measurement we use for saliency evaluation is the concentration of the peak distribution, which is inversely proportional to the coverage area of the peak distribution around the maximum point. Thus, we measure the concentration $c$ of a peak distribution in the similarity map $\bm{S}_{(u,v)}$ by the reciprocal of the area of main lobe $A_{\Psi}(\bm{S}_{(u,v)})$:

Combining the defined intensity and the concentration, we evaluate the quality of a saliency $s(\bm{S}_{(u,v)})$ for the similarity map $\bm{S}_{(u,v)}$ by:

where $\alpha$ is a hyper-parameter to balance between the effects of intensity and concentration. The rationale behind this design is that the defined intensity and the concentration jointly reveal the sharpness of the peak distribution around the maximum point. A larger saliency value $s$ implies that the corresponding pixel in the feature maps of the exemplar $\bm{F}_{x}$ is more discriminative for tracking and easier to be localized in the feature maps of the search image $\bm{F}_{s}$.

Considering that the tracker should be encouraged to focus on tracking the central area of the target exemplar, a regularized term, which is a Gaussian mask, is added into the saliency evaluation metric $s$:

Herein $g_{\mu_{g},\sigma_{g}}(u,v)$ is a Gaussian function aligned to the center of the exemplar feature $\bm{F}_{x}$, and $\lambda$ is a balance weight. During end-to-end training, the gradients can be back-propagated through the saliency evaluation metric, and we detail its back-propagation in the supplementary materials.

Based on the defined saliency evaluation metric in Eq. 7, we compute the saliency for each pixel in the feature maps of the exemplar $\bm{F}_{x}$ and select $K$ most salient pixels as the set of captured saliencies $P_{x}=\{\bm{p}_{x}^{k}\}_{k=1}^{K}$. The matched positions of these saliencies in the feature maps of the search image $\bm{F}_{s}$ composes the counterpart of the saliency set in the search image $P_{s}=\{\bm{p}_{s}^{k}\}_{k=1}^{K}$.

The captured saliencies, which are discriminative local parts of the target for tracking, are further associated together by the Saliency-Association Modeling module of SAOT to learn effective global correlation representations between the exemplar and the search image. The obtained correlation representations are finally used for estimating the target state in the search image for tracking.

An intuitive way to associate the captured saliencies is to make connections between these local saliencies to form a global graph that is able to characterize the whole target. Following this way, the Saliency-Association Modeling module of our SAOT performs saliency association in two steps: 1) constructs an effective graph among the captured saliencies to model the interactions between these saliencies; 2) aggregates the saliencies based on the constructed graph to learn global correlation representations between the exemplar and the search image.

A key step of constructing the saliency graph is to model the interactions between nodes by connecting edges. Since we aim to associate the captured saliencies to achieve the effective global representation of the tracking target, we make pairwise edge-connections between $K$ saliencies specified in $P_{s}$. Besides, we also connect each node in $\bm{F}_{g}$ to its eight neighbors to perform neighboring information interactions for feature fusing between adjacent nodes. The resulting connection set including these two types of edges is denoted as $\mathcal{C}$, which is illustrated in Figure 5.

To precisely model the interactions between the above specified connections, the edge weights are learned by the proposed Saliency-Association Modeling module rather than being fixed as binary values. Particularly, we customize a two-layer perception network to learn the edge weight for each connection specified before. Thus, the weighted adjacency matrix $\bm{A}\in\mathbb{R}^{N\times N},N=h_{s}w_{s}$ for the saliency graph is modeled by:

where $\bm{v}_{i}\in\mathbb{R}^{h_{x}w_{x}+c}$ and $\bm{v}_{j}\in\mathbb{R}^{h_{x}w_{x}+c}$ are features of two nodes in a connection. $\phi_{1}$ and $\phi_{2}$ denote the parameters of two fully connected layers, while $\sigma$ refers to Sigmoid function which transforms the edge weights to lie in $(0,1)$.

Specifically, we construct the two-layer GCN to perform saliency aggregation. Inspired by Li et al.[27], we adopt the high-order polynomial of $\bm{A}$ to model multi-scale interactions between nodes. Formally, the $l$-th layer graph convolution is formulated as:

where $m$ and $M$ are the polynomial order and total number of orders, respectively. Here, $w_{m}$ is a trainable weight for the order $m$. $\hat{\bm{A}}=\tilde{\bm{D}}^{-\frac{1}{2}}\tilde{\bm{A}}\tilde{\bm{D}}^{\frac{1}{2}}$ is the normalized adjacency matrix [23], where $\tilde{\bm{A}}=\bm{A}+\bm{I}$ and $\tilde{\bm{D}}$ is the diagonal degree matrix of $\tilde{\bm{A}}$. $\bm{X}^{(l)}\in\mathbb{R}^{N\times d_{l}}$ and $\bm{X}^{(l+1)}\in\mathbb{R}^{N\times d_{l+1}}$ are the input and output features of all nodes at layer $l$ respectively, where $d_{l}$ and $d_{l+1}$ are the corresponding feature dimensions, and $d_{0}$ is equal to node feature dimension ($h_{x}w_{x}+c$). $\bm{\Theta}_{(m)}^{(l)}\in\mathbb{R}^{d_{l}\times d_{l+1}}$ denotes the learnable parameter matrix at layer $l$ for $m$-th order. $\sigma_{l}$ is the activation function at layer $l$.

Through constructing the saliency graph and further performing saliency aggregation, the Saliency-Association Modeling module of SAOT is able to learn global correlation representations between the target exemplar and the search image, which is further used for predicting the target state in the search image.

Our model can be readily integrated into various typical tracking frameworks. As shown in Figure 2, we integrate our algorithm with a typical online learning tracker, namely online discriminative filter [2]. The output global correlation representations by our algorithm is fed into a classification head for predicting the classification map and a regression head for predicting the bounding box of the target. In particular, the output $\bm{p}_{o}$ of the classification head is used to regularize the response map $\bm{p}_{r}$ produced by online discriminative filter via weighted element-wise summation to generate the final classification map $\bm{p}_{cls}$. The predicted bounding box by the regression head, which is corresponding to the maximum classification score in $\bm{p}_{cls}$, is used as the final tracking result. Both the classification and regression heads are designed following FCOS [40].

To investigate the effectiveness of each proposed component, we perform ablation studies with six variants of SAOT:

Table 1 presents the experimental results of these variants on the OTB2015 [43] and NFS30 [21] benchmarks.

Herein we compare our SAOT with 17 representative state-of-the-art methods on five benchmarks, including OTB2015, NFS30, LaSOT, VOT2018, and GOT10k. The methods involved in the comparison include 16 holistic-strategy trackers (KYS [3], Ocean [49], SiamBAN [5], SiamAttn [44], PrDiMP [9], Retina-MAML [41], DiMP [2], GradNet [28], ATOM [7], SiamRPN++ [25], C-RPN [14], GCT [15], SPM [42], DaSiamRPN [50], SiamRPN [26], and DSiam [16]) and one part-based tracker (PG-Net [32]). We discuss the experimetal results per dataset below.

To obtain more insights into our method, we visualize the correlation representations and the saliency values.