A Novel Long-term Iterative Mining Scheme for Video Salient Object Detection

Although the main scope of this paper is deep learning-based VSOD, we give a brief review regarding the most representative conventional approaches. Liu et al. [31] performed the VSOD task via superpixel-based graph smoothing. By using color and global motion histograms, both spatial and temporal saliency cues are obtained in advance and then diffused via temporal propagation. Xi et al. [32] devised a novel method for computing the background prior. This prior is combined by averaging the prior of the spatial [33] and temporal backgrounds [34]. Li et al. [35] proposed a motion-based bilateral network to estimate the backgrounds in a framewise manner, which are later embedded into a graph for saliency propagation. Chen et al. [36] proposed a novel solution to distinguish moving salient objects from diverse changing background regions. Zhou et al. [37] considered both temporal consistency and correlation among adjacent frames to compute the temporary saliency map. Then, both spatial and temporal saliency maps are fused via the proposed spatiotemporal refinement. Guo et al. [38] designed a primitive approach to identify the salient object by ranking and selecting the salient proposals. Chen et al. [7] adopted the low-rank strategy to alleviate the problem of false-alarm error accumulation induced by the widely used saliency propagation. However, this approach basically follows the batchwise methodology, which fails to take full advantage of the beyond-scope saliency consistency, producing massive false alarms when the majority of the intrabatch saliency clues are incorrect. To further improve, the same authors [39] devised bilevel metric learning to include more spatial-temporal saliency cues in the current saliency prediction. Guo et al. [40] proposed a fast VSOD method by using the principal motion vectors to represent the corresponding motion patterns, and such motion message coupling with the color clues together are fed into the multiclue optimization framework to achieve the spatiotemporal VSOD. Zhao et al. [41] proposed a weakly-supervised VSOD model based on eye-fixation annotation. Compared with the fully-supervised VSOD models, the proposed new annotation method dramatically reduces the consumption of time.

In view of the video instance segmentation task [42, 43], Liu et al. in [44] have presented a one-stage, end-to-end, and proposal-free deep model, whose key highlight is its capability of dividing instances into sub-regions dynamically and performing segmentation on each region for spatial granularity. Thus, it achieves a more appealing segmentation behavior as it can enrich object details and produce masks with more accurate edges. In view of the video object detection (VOD) task [45], Cui et al. in [46] have proposed to depict the temporal feature relations and blend valuable neighboring features, thus the spatiotemporal feature representation can get enhanced. Also targeting at the VOD task, Liu et al. in [47] proposed an approach which aggregates features calibrated at both pixel and instance levels, thereby achieving a better detection performance. More works about VOD tasks can be referred to the recent survey paper [48].

Let us now move to the most representative deep learning-based SOTA models, whose performances have significantly outperformed the conventional models. Wang et al. [49] adopted an end-to-end network to compute spatial saliency, which was later coupled with two consecutive frames to serve as the input of another network to compute the spatiotemporal saliency. Li et al. [23] adopted the optical flow results as the network input to compute motion saliency. Because motion saliency could only occasionally benefit the VSOD task, this work developed an attention mechanism, which is empowered by gate logic to filter the less trustworthy motion saliency cues when performing spatial and temporal saliency fusion. Tokmakov et al. [50] fed the concatenated spatial and temporal deep features into the ConvLSTM for spatiotemporal saliency fusion.

Following the bistream structure, Le et al. [51] computed the motion saliency via 3D convolutions, and the VSOD results were derived simply by concatenating and convolving the spatial and temporal deep features. Chen et al. [52] proposed a new version of 3D convolution, which involves multiple 3D convolutions with the newly proposed temporal shuffle operation to enhance the network’s ability in sensing temporal information. This modification also enables the spatial branch to fully interact with the temporal branch in a multiscale manner. Song et al. [26] devised the bi-LSTM network to sense multiscale spatiotemporal information. This work also adopted pyramid dilated convolutions to extract multiscale spatial saliency features, which are fed into the abovementioned bi-LSTM network to achieve multiscale VSOD. Wang et al. [53] proposed coattention to enhance the consistency between different frames. Fan et al. [22] developed an attention-shift baseline and released a large-scale saliency-shift-aware dataset for the VSOD problem. Zhou et al. [54] have presented a novel end-to-end zero-shot video segmentation network. In this work, the proposed network follows the traditional bi-stream structure, yet, different from previous works, it newly devised a novel module to interact temporally with spatial. Generally, this method follows a typical coarse-to-fine rationale. The temporal information is treated as the coarse indicator to locate the target object. Then, based on the coarsely localized regions, the proposed fusion module further learns the spatial appearance to refine the segmentation process, ensuring the output has a crisp object boundary. Recently, Zhou et al. [55] have devised a novel bottom-up instance discrimination network which takes advantage of temporal context information in videos for more accurate segmentation. In view of the methodology innovation, the authors have converted the video segmentation task to a tracking paradigm, which can better capture the appearance information of the target objects, yielding better segmentation results.

Recently, Gu et al. [56] learned the nonlocal motion dependencies across several frames, and then it followed the pyramid structure to capture the spatiotemporal saliency clues at various scales. The major highlight of this paper is the proposed constrained self-attention operation, which can capture motion cues via the prior that objects always move in a continuous trajectory, achieving a very high FPS rate. Ren et al. [24] proposed a novel triple-stream network, which includes a spatiotemporal network, a spatial network, and a temporal network, where the saliency cues computed from the spatial and temporal networks serve as the attention to enhance the main network. Jiao et al. [57] proposed a Guidance and Teaching Network (GTNet), GTNet introduces a temporal modulator to bridge features from motion into appearance, and a motion-guided mask is used to propagate the explicit cues during the feature aggregation. Bi et al. [58] devised STEG-Net, which uses the extracted edge cues to guide the extraction of spatial-temporal cues and combines deep texture cues with shallow edge cues. This strategy can simultaneously retain edge information and enhance object’s global cues, making the object’s position more accurately.

Clearly, most of the current SOTA models mentioned above have one critical problem, i.e., their methodologies tend to be short-term in essence, and the limitations have been mentioned in the introduction. Several previous works have become aware of this problem, and multiple modifications have been attempted to alleviate this problem.

Lu et al. in [59] have presented a GNN based zero-shot video object segmentation (ZVOS) network. By using the newly devised attentive graph neural network (AGNN), the ZVOS can be treated as an end-to-end message passing based graph information fusion procedure. The major highlight of this approach is its capability to make the high-order relations among frames be captured. In the later journal version [60], the AGNN was extended to diverse segmentation tasks with additional in-depth discussions and explanations. To further improve the SOTA models that primarily focus on learning discriminative foreground representations in a short-term manner, Lu et al. in [61] have proposed to re-design the ZVOS in a holistic fashion. The authors have devised the co-attention layers to learn global correlations and scene context, and thus the short-term spatiotemporal feature fragments can be interacted in a joint feature space, achieving significant performance improvement.

Chen et al. [62] used SIFT-Flow to introduce beyond-scope saliency cues to the current problem domain, making its spatiotemporal saliency computation relatively ‘long-term’. Li et al. [63] utilized tracking consistency to mine keyframes, and then the high-quality spatiotemporal saliency cues could be diffused from each keyframe to other normal frames. The keyframe selection scheme proposed in this work belongs to the long-term scope, but the saliency diffusion process is still a short-term method. Another representative work is [64], in which the keyframes were selected offline by measuring the consistency degree between spatial and temporal saliency cues, and then an ISOD model was fine-tuned on these keyframes to adapt for this sequence. It is worth mentioning that this idea is inspired by [65], where the model fine-tuning scheme was used, while the major difference relies on whether the annotation of the 1st frame is given. Wang et al. [66] proposed a novel attentive graph neural network to explore the relationship between different frames. Compared with the previous short-term method, the saliency sensing scope of this work was expanded significantly. However, because the network training and testing processes still followed the framewise sequential order, this work still belongs to the short-term category. Following the model online matching and updating schemes that were widely used object trackers, Wang et al. [1] proposed the episodic graph memory network, where multiple submodels are trained, stored, and updated to adapt the current VSOD task to the current video sequence long-term. However, the VSOD task carried out by this work was solely along the temporal direction; thus, not all spatiotemporal saliency cues embedded in the video were simultaneously available to the current frame. Wang et al. [67] converted the framewise VSOD task to a graph problem, which converted the video frames to supervoxels, and the graph convolutional network, taking the supervoxels as the graph nodes, was used to explore the long-term spatiotemporal VSOD. However, because this graph structure was still constrained by the temporal neighboring topology, the long-term information explored by this work might be rather weak in essence. Zhang et al. [68] proposed a dynamic context-sensitive filtering network (DCFNet). This net generates dynamic convolution kernels containing rich context information at multiple scales by estimating location correlation weights to improve the model’s adaptability to dynamic video scenes. Chen et al. [69] proposed a new concept, named motion quality, to re-balance the complementary fusion status between spatial information and temporal information.

Given a video sequence, we use the off-the-shelf object detection tool SEOD [30] to acquire object proposals. For a single frame, we rank all its object proposals according to their objectness confidences, and a maximum of 10 object proposals are considered (we also test to consider more than 10 object proposals, but no obvious performance improvement can be observed). Thus, a video sequence with a total of $T$ frames can now be expressed as $N=T\times 10$ object proposals. We use $P_{i}$ to denote the $i$-th object proposal, and all proposals can be represented as $\mathcal{P}=\{P_{1}...,P_{N}\}$.

The aim of this section is to find which object proposals are the salient proposals (see ‘Part 1’ in Fig. 2). Our key idea is to iteratively mine all potential salient object proposals from $\mathcal{P}$ from easy to hard, and in each mining iteration, only a small group of $\mathcal{P}$ could be determined to be either salient or nonsalient. Initially, we place the accuracy rate as the highest priority, while the recall rate can be gradually ensured by the upcoming iterative mining steps. The proposed iterative mining process can be visualized in Fig. 1-B, where more salient object proposals, which are difficult to determine in the early iterations, can be correctly mined later because our mining approach can take full use of the knowledge derived from the previous iterations.

Once all salient object proposals have been determined, we utilize the online fine-tuning scheme to convert these salient object proposals to framewise saliency maps (see ‘Part 2’ in Fig. 2).

For each object proposal in $\mathcal{P}$, we use the pretrained feature backbone (ResNet18) to extract the corresponding semantical deep feature ($f_{i}$), and the deep features of $\mathcal{P}$ can be represented as $\textbf{F}=\{f_{1}...,f_{N}\}$.

Based on the L2 similarity between each pair of F, all object proposals ($\mathcal{P}$) can be clustered into $K$ clusters. Considering the limitation of GPU memory, long video sequences should be divided into multiple short sequences in advance. It might also be rare for typical short video sequences (less than 100 frames) to contain more than three salient objects. The clustering process can be represented as Eq. 1.

where we use $C_{i}$ to denote the $i$-th cluster, $i\in\{1,2...,K\}$. Any off-the-self clustering tool can be applied here, where we simply choose the K-means.

To locate salient object proposals coarsely, the most intuitive method might be computing the ‘cluster-level saliency degree’ first and then selecting the top salient object proposals as the salient clusters. Our rationale is a salient cluster’s object proposals tend to exhibit very strong saliency cues. Following this rationale, for each cluster, we use the averaged ‘motion saliency’ to represent the cluster-level saliency degree. Generally, compared with spatial saliency, motion saliency has one clear advantage: fewer false alarms; for example, at worst, motion saliency becomes absent, while spatial saliency might be totally incorrect for unpredictable reasons. To continue introducing the technical details of the clustering-based salient object proposal coarse localization, we leave the contents regarding the motion saliency computation to the next subsection.

We select $\nu$ clusters from $\mathcal{C}$ as the salient clusters, where we empirically constrain $\nu<\lfloor 0.5\times K\rfloor$. The exact value of $\nu$ can be automatically determined via Eq. 2.

where $PMS_{i}=\{PMS_{i}(1)...,PMS_{i}(g)\}$ is a set in which each element represents the motion saliency value, e.g., $PMS_{i}(1)$ denotes the motion saliency value (1-dimension) of the 1st object proposal in the $i$-th cluster, and $g$ is the number of object proposals in this cluster; function $avg(C_{i})$ returns the averaged motion saliency value of all object proposals in the $i$-th cluster, thus $ams$ is a $K$-dimensional vector; function $\mathcal{DES}(\cdot)$ sorts its input elements in descending order; $\nabla$ is the first-order derivation operation.

Clearly, the rationale of the dynamic $\nu$ relies on representing the initial ‘localization’ regarding which object proposals among the cluster $C_{i}$ are very likely to be salient.

Intuitively, one can obtain motion saliency cues directly via an off-the-shelf image salient object detection (ISOD) model if it has been fine-tuned on the optical flow data, i.e., $\{OF_{j},GT_{j}\}$ (see below).

Assume two consecutive video frames can be represented as $\textbf{I}_{j}$ and $\textbf{I}_{j+1}$, and the corresponding optical flow $OF_{j}$, which will be used for ISOD model fine-tuning, can be computed by Eq. 5.

where ‘FlowNet’ represents the off-the-shelf optical flow tool [70], whose input includes two consecutive video frames, and the outputs are two gradient matrixes representing the spatial displacement over the vertical and horizontal directions; $ce\{\cdot\}$ denotes the widely used color encryption tool, which converts the abovementioned two optical flow matrixes to a three-dimensional matrix ($\mathbb{R}^{W\times H\times 3}$), in which all gradient directions and values are uniformly encoded by different colors after this operation (see the third column of Fig. 3).

Thus, $OF_{j}$ and the corresponding raw ISOD ground truth $GT_{j}$ can be used to fine-tune the off-the-shelf ISOD model. The loss function of ISOD deep model fine-tuning can be represented as Eq. 6, for which we use a small learning rate ($10^{-7}$).

After being fine-tuned, the adopted ISOD model ($\mathcal{M}$) takes $OF$ as input and then outputs a motion saliency map (see the last column in Fig. 3). This process can be represented by the following equation:

where $\Theta$ is the learnable hidden parameters, $\mathcal{M}$ can be any off-the-shelf end-to-end ISOD model following the typical encoder-decoder structure (we choose the CPD19 [71]), and $MS_{i}$ is the motion saliency map (see the last column of Fig. 3).

The motion saliency maps are usually blurred in object boundaries, which are mainly determined by the optical flow quality. To the best of our knowledge, there exists no better method for improving this problem at present. Thus, the motion saliency maps can only be used for the coarse localization of the salient object proposals.

In addition, the motion saliency, a typical unstable saliency cue, tends to vary from frame to frame; for example, it may become helpless if the salient object is temporally static, as demonstrated in the #54 frame in Fig. 3. However, the motion saliency cue might be very helpful in some other scenarios that have clear object movements, e.g., the #57 frame. Therefore, based on $MS$ (Eq. 7) and $\mathcal{P}$ (Sec. III-A), we detail the computation of $PMS$, which is mentioned in Eq. 4, as below.

Taking $PMS_{i}(u)$ as an example, suppose this object proposal is the $h$-th object proposal in $\mathcal{P}$ and it also belongs to the $j$-th video frame; thus, the exact value of $PMS_{i}(u)$ can be obtained by:

where the $crop$ operation crops a patch from $MS_{j}$ according to the coordinates provided by the object proposal $P_{h}$.

By performing the previous steps (e.g., Eq. 2), $\nu$ salient clusters have been selected. The remaining problem here includes two aspects: 1) these $\nu$ salient clusters might contain some nonsalient object proposals, and 2) some salient object proposals might belong to the $\{K-\nu\}$ nonsalient clusters. To precisely separate these two types of object proposals (we call them the ‘undetermined’ proposals), we resort to an auxiliary binary classifier.

Our idea is that for the ‘undetermined’ object proposals, the binary classifier serves as an indicator to tell which object proposals are more likely to be salient. In practice, this binary classifier can be weakly trained by instances belonging to salient clusters or nonsalient clusters. The main reason that we call this training process a ‘weakly supervised’ process is that for each video sequence, both positive and negative instances for the binary classifier training are determined online. Without knowing the ground truths, some of the most trustworthy object proposals in the salient clusters are selected as the positive instances ($PseudoGT=1$). The most trustworthy negative instances ($PseudoGT=0$) are selected from the nonsalient clusters. To focus on the main topic in this subsection, we leave the technical details regarding instance selection to the next subsection.

Based on the positive and negative instances mentioned above (total $Q$ instances and $Q\leq N$), the loss function of the binary classifier, a typical cross-entropy loss, can be represented as Eq. 9.

where $FC(\cdot)$ denotes the widely used multilayer fully connected layers, $f_{q}$ represents the high-dimensional deep feature of the $q$-th object proposal selected under the current salient and nonsalient cluster partition, $PseudoGT_{q}$ is the corresponding binary label of $f_{q}$. Note that the training process of this binary classifier of this classifier can be performed very quickly; thus, it can be applied in the VSOD task. In the next subsection, we detail the computational process of $PseudoGT$.

Considering that the performance of the binary classifier is positively related to the quality of the pseudolabels (i.e., $PseudoGT$), we should ensure that those less trustworthy object proposals are excluded from the pseudotraining set. To achieve this, we utilize two measurements: 1) the distance to the cluster’s centroid and 2) the motion saliency degree of each object proposal.

The rationale of measurement 1) is that the trustworthy degree of an object proposal to be a real salient proposal should be positively related to the confidence degree of belonging to its cluster. Actually, this confidence degree can be measured by the feature similarity degree ($sim$) between the cluster’s average profile ($AP_{i}$) and the object proposal (see Eq. 10), where $AP_{i}$ could be automatically obtained during the K-means clustering process.

where $f_{j}$ represents the deep feature of the $j$-th object proposal in its cluster $C_{i}$, $||\cdot||_{2}$ is the $L_{2}$-norm, and $AP_{i}$ represents the clustering profile of $C_{i}$, which shares an identical size to the $f_{j}$; $sim_{i}(j)$ measures the feature distance of the $j$-th object proposal to the cluster centroid. The primary objective of Eq. 10 is to serve the proposed mining process to filter out those less trustworthy object proposals in a given cluster. Our motivation is clear and intuitive, i.e., given a salient cluster, an object belonging to yet with a large feature distance to the cluster’s average profile tends to be non-salient. Thus, this object proposal shall have a large chance of being filtered during the subsequent mining process. Once $sim$ (Eq. 10) is obtained, we use it to rank all object proposals in ascending order.

The rationale for measurement 2) relies on the fact that object proposals with clear movements are more likely to be salient proposals. Thus, all object proposals in each cluster can also be ranked in descending order according to their motion saliency degrees, i.e., $PMS$ (Eq. 8).

Based on the abovementioned factors, the less trustworthy object proposals in the salient clusters ($\mathcal{C}^{+}$) or the nonsalient clusters ($\mathcal{C}^{-}$) can be filtered by using the equations documented below; $\mathcal{C}=\mathcal{C}^{+}\cup\mathcal{C}^{-}$ and $\cup$ is the union operation.

where we use the superscripts $+$ and $-$ to distinguish salient clusters and nonsalient clusters determined by the previous steps; $SIM_{i}\in\mathbb{R}^{1\times g}=[sim_{i}(1)...,sim_{i}(g)]$, and $sim_{i}(1)$ denotes the similarity degree of the 1st object proposal in the $i$-th cluster computed via Eq. 10; $PMS_{i}$ can be found in Eq. 8; $SI_{i},MI_{i}\in\mathbb{R}^{1\times g}$ are two subscript vectors of object proposals in the $i$-th salient cluster; $\mathcal{ASD}(C_{i},SIM_{i})$ returns the reranked object proposals’ subscripts via $sim_{i}$ in ascending order, and $\mathcal{DES}(C_{i},MS_{i})$ returns the reranked object proposals’ subscripts via $MS_{i}$ in descending order.

Next, the intersection between the top-$\alpha$ elements in $SI_{i}$ and the top-$\beta$ in $MI_{i}$ can be the most trustworthy object proposals in the current cluster, where $\alpha$ and $\beta$ can be obtained via Eq. 13 and Eq. 15.

where the definitions of $SI$ and $MI$ can be found in Eq. 11 and Eq. 12, the operation $Re(SIM_{i},\ SI_{i}^{+})$ reorders the elements in $SIM_{i}$ according to the $SI_{i}^{+}$, similar to Eq. 3, $\nabla(\cdot)$ computes the gradient of its input vector, i.e., the output of $Re(\cdot,\cdot)$, $\xi$ is the lower bound, and we empirically assign it to $0.6\times g$, and $g$ is the total proposal number in the $i$-th cluster.

The filtering process outputting trustworthy positive and negative instances can be detailed as Eq. 17.

where $\mathcal{ASD}/\mathcal{DES}$ is identical to that of the previous equations; $\alpha/\beta$ can be found in Eq. 13, Eq. 14, Eq. 15, and Eq. 16; $TOP(SIM_{i},\alpha)$ returns the top-$\alpha$ elements of $SIM_{i}$, $SIM/PMS$ can be referred from Eq. 11, and $\cap$ is the operation that returns the intersection of two sets. Compared with the original $\mathcal{P}^{+}$ and $\mathcal{P}^{-}$, samples in $Pos$ or $Neg$ are clearly more trustworthy in general, and the total sample numbers of $Pos$ and $Neg$ should be less than the total object proposal number ($N$) of $\mathcal{P}$.

Based on both $Pos$ and $Neg$, the binary classifier can be well trained. By using this classifier as the indicator, we can determine the saliency labels (salient or nonsalient) for more object proposals that are relatively less trustworthy than those belonging to either $Pos$ or $Neg$, and we call them the ‘uncertain’ object proposals.

For each uncertain object proposal in the salient clusters ($\mathcal{C}^{+}$), we add it into the $Pos$ set if the binary classifier predicts it as ‘salient’. Similarly, for each uncertain object proposal in the nonsalient clusters ($\mathcal{C}^{-}$), we add it to the $Neg$ set if the binary classifier predicts it to be ‘nonsalient’. By using the mining step mentioned above, both sets ($Pos$ and $Neg$) can be gradually expanded to comprise more feature patterns. However, the binary classifier is not always trustworthy and predictions might be occasionally incorrect, making both $Pos$ and $Neg$ sets noisy.

To alleviate this problem, we propose two additional constraints to ensure that the uncertain object proposals included in $Pos$ and $Neg$ sets are the most trustworthy. We use $Pos+$ and $Neg+$ to denote these two types of uncertain proposals. Thus, only those object proposals, which belong to the $Pos+$ or $Neg+$ set and meet the constraint addressed below, would be eventually added into the $Pos$ or $Neg$ set. Taking $Pos+$ as an example, all proposals are reordered via their similarity degrees to the mean profile of its cluster (Eq. 10); thus, we denote the reordered $Pos+$ as $Pos+^{\prime}$. We constrain only the top-$\gamma\%$ proposals in $Pos+^{\prime}$ to be added to the $Pos$ set. We empirically choose a relative slack value (60%) to $\gamma$ to balance the diversity and the trustworthiness degree.

In our implementation, we repeat the above mining process multiple times, where both $Pos$ and $Neg$ can be expanded each time. Specifically, to acquire more knowledge, the binary classifier is updated/fine-tuned after both $Pos$ and $Neg$ are expanded. Clearly, the binary classifier becomes more powerful as the above mining process continues.

We visualize the iterative mining process regarding the salient object proposals in Fig. 5. As seen in ‘iteration 1’, the mining process tends to simply select the most trustworthy object proposals (the top-left cell is the most trustworthy object proposal, and the bottom-right cell is the cell with the smallest trustworthy degree). Then, in ‘iteration 2’, some of the less trustworthy object proposals are selected to enhance the diversity of the $Pos$ set. Finally, in ‘iteration 3’, again, some of the most trustworthy object proposals that are misdetected by the previous iterations can be selected. Compared with the previous two iterations, only a very small group of object proposals are selected in ‘iteration 3’; thus we omit ‘iteration 4’ to ensure efficiency.

Previous steps (Sec. III) can only tell us which object proposals are salient. To obtain a framewise saliency map, we simply resort to the existing image salient object detection (ISOD) model, i.e., each object proposal is fed into the ISOD model to obtain a patchwise saliency map.

Compared with the conventional ISOD, the patchwise SOD task here is much simpler because it has a relatively smaller problem domain. Thus, we prefer off-the-shelf ISOD models with lightweight designs. In our implementation, we choose the CPD [71], and we fine-tune it on the MSRA10K set, where we converted the original image-level training instances to patchwise instances. The well-segmented patchwise saliency maps can be visualized in the second row of Fig. 4. Almost all salient objects or tiny parts have been well segmented by the patchwise refiner. However, failure cases still exist, such as the middle case in the far left column, where only the dog’s leg was detected, while the main body of the dog was missed.

Intuitively, the frame-level saliency map can be easily obtained by ‘pasting’ all its patchwise saliency maps. However, due to the abovementioned failure cases, most of which are missing detections and false-alarm cases are very rare, the ‘pasting scheme’ - the widely used average operation-based fusion - might not be suitable here, which can lead to frame-level saliency maps suffering from missing detections. Therefore, we formulate the ‘pasting process’ as follows:

where $FS_{i}$ represents the frame-level saliency map of the $i$-th frame $\textbf{I}_{i}$; assume $\textbf{I}_{i}$ contains $b$ salient object proposals, the function $crop$ crops a patch from $\textbf{I}_{i}$ according to the coordinates provided by the object proposal $P_{j}$, and $1\leq j\leq b$; $\mathcal{PR}$ denotes the proposed patchwise refiner, $\Theta$ is its learnable hidden parameters, $\mathcal{PR}$ outputs the patchwise saliency map, $Z(\cdot)$ is a typical min-max normalization function, and function $proj$ pastes its input to the current video frame; thus, the output of this function is a matrix with the same height and width as the current frame.

The qualitative demonstration of $FS_{i}$ can be seen in the middle part of Fig. 4. Compared with the existing ISOD models [76, 77, 78, 79, 80, 81, 82, 83], which consider only short-term information, our framewise saliency map is clearly built from the long-term perspective, which has a clear advantage in terms of robustness.

Despite the merits mentioned before, framewise saliency maps ($FS$) still have two problems. First, the quality of $FS$ is heavily dependent on the previous salient object proposal localization steps. Second, this quality is also influenced by the ‘patchwise refiner’. As a result, imperfect $FS$ can be observed frequently, such as the far-right column of Fig. 4. Thus, we propose the ‘framewise refiner’, and only the most trustworthy $FS$ serves as the teacher to guide the online learning process.

In practice, the framewise refiner can be any ISOD model using only spatial information, where we simply continue using the CPD as the framewise refiner due to its lightweight design. Note that if the framewise refiner is taught by some of the high-quality $FS$, it can perform very well on the remaining unseen frames, producing very high-quality saliency maps. We show the difference between $FS$ and the final saliency map in Fig. 6.

Outwardly, using only the high-quality $FS$ to fine-tune the framewise refiner omits the temporal information completely. Nevertheless, it should be noted that the temporal information is implicitly embedded in the $FS$ because the previous salient object proposal localization steps fully consider the motion saliency cues (i.e., Eq. 8).

Let us now move to the details regarding how to select the high-quality $FS$ — the proposed fast keyframe selection scheme. Compared with spatial cues, motion cues are more likely to attract our visual attention. Thus, in some cases, motion saliency cues can be a very effective indicator for locating real salient objects. However, the motion saliency cue has one critical drawback, i.e., it is unstable in essence, and we noted this issue multiple times before. Therefore, we consider both spatial and temporal saliency cues to derive robust VSOD.

Considering the common attribute of high-quality $FS$, both their corresponding spatial and temporal saliency maps tend to be high-quality maps, and the reverse usually holds. Hence, for each frame, we can use the similarity between the spatial saliency ($FS$) and temporal saliency ($MS$) to measure the quality degree of $FS$ because those frames with low-quality $FS$ are unlikely to have an ‘extremely’ high spatiotemporal saliency consistency.

To be more specific, our keyframe selection process can be summarized into the following two steps:
First, for each frame, we compute the consistency between its $FS$ and $MS$, where we use the S-measure [84] as the similarity measurement because $MS$ focuses more on the overall structural similarity rather than the tiny saliency details that are quite rare in $MS$.
Second, for every $B$ frames (the exact choice of $B$ can be found in the ablation study), only the one with the largest spatiotemporal saliency consistency degree is selected as the key frame. For example, given a sequence with $N$ frames in total, there are $N/B$ keyframes to be selected, and all these keyfames are used to fine-tune the framewise refiner. The fine-tuning process can be performed very quickly (10 epochs), and the final VSOD results can be obtained by taking each nonkeyframe as the input of the framewise refiner. To maintain strong generalization ability, for each sequence, the framewise refiner is recovered to its initial status.

To facilitate readers’ understanding, we have provided a method flow chat, where all key steps have been included into it. The detailed flow chat can be found in Algorithm 1.

Following the conventional experimental setting, we evaluated the proposed approach on five widely used publicly available datasets, including Davis [73], Segtrack-v2 [74], VISal [2], DAVSOD [22], and VOS [75].

• •

  The Davis dataset contains 50 video sequences with 3,455 frames in total, and most of its sequences only contain moderate motions.

• •

  The Segtrack-v2 dataset contains 13 video sequences (excluding the penguin sequence) with 1,024 frames in total, containing complex backgrounds and variable motion patterns, which is generally more challenging than the Davis dataset. This dataset is dominated by a temporal source with fast object movements.

• •

  The VISal dataset contains 17 video sequences with 963 frames in total, and this dataset is relatively simple. This dataset is dominated by spatial sources, where most of the existing SOTA ISOD models using spatial information alone performed very well on this dataset.

• •

  The DAVSOD dataset contains 226 video sequences with 23,938 frames in total, which is the most challenging dataset, involving various object instances, different motion patterns, and saliency shifting between different objects. Since most SOTA VSOD models have failed to consider the attention shifting mechanism, their quantitative scores over this set are very low.

• •

  The VOS dataset contains 40 video sequences with 24,177 frames in total, yet only 1,540 frames were annotated well, in which the sequences were all obtained in indoor scenes.

We implemented our method on a PC with an Intel(R) Xeon(R) CPU, NVIDIA GTX2080Ti GPU (with 11G RAM) and 64G RAM. We choose SEOD [30] as the object detector to obtain object proposals. The patchwise and framewise refiners (Sec. IV and Fig. 4) follow an identical structure to the CPD [71]. The patchwise refiner is retrained on patchwise data generated from the widely used MSRA10K [85]. The framewise refiner follows the online learning scheme, which is trained on DAVIS-TR [73] in advance, and then fine-tuned online on the keyframes mined from the current input sequence. Note that for each input video sequence, we perform the online fine-tuning process only once. The motion saliency network (Eq. 6) also follows an identical structure to the CPD, which is initially trained on the MSRA10K set and then fine-tuned on the DAVIS-TR set. The binary classifier is a multilayer fully connected network whose network structure follows the lightweight ResNet18, and the updating process takes 20 epochs. To avoid overfitting problems, we adopted random horizontal flips for data augmentation.

To accurately measure the consistency between the predicted VSOD and the manually annotated ground truth, we adopt five commonly used evaluation metrics, including the precision rate, recall rate, maximum F-measure value (maxF) [86], mean absolute error (MAE) [87], and structure measure value (S-measure) [84].

We compared our method with 14 SOTA approaches (see Table I), including DCFNet [68], MQP [69], TENet20 [24], U2Net20 [72], PCSA20 [56], LSTI20 [62], SSAV19 [22], MGA19 [23], COS19 [53], CPD19 [71], PDB18 [26], MBN18 [35], SCO18 [36], and SFLR17 [7]. Among all the SOTA competitors compared here, TENet20 and LSTI20 are mainly designed for introducing more long-term information into the current problem domain. However, compared with these two models, the long-term attribute of our approach is clearly stronger because all object proposals in the input sequences are simultaneously available to the current problem, while the basic methodology of both TENet20 and LSTI20 might be short-term in essence. For example, the graph structure adopted by TENet20 is a clear short-term manner, and the long-term alignment process of LSTI20 can be quite limited because long-term spatial alignment might be very difficult if the spatial appearance of the salient object has changed significantly.

Compared with other short-term models (e.g., PCSA20, SSAV19, and PDB18), our method outperformed them significantly, especially on the SegTrack-v2 set, where the salient objects tend to exhibit fast object movements and vary their spatial appearances very quickly; thus, the conventional short-term methodology can easily produce failure detections if both the current spatial and temporal saliency cues are incorrect. In sharp contrast, the proposed long-term method can to utilize the beyond-scope information to help the current saliency prediction.

Specifically, we noticed that our method failed to achieve the best performance on the VISal set. This can be explained by the attribute of the VISal set — all video sequences are dominated by spatial saliency cues, while the temporal saliency cues become completely helpless. This issue can be evidenced by the fact that the ISOD model (i.e., the CPD19), which only considers the spatial information, could still perform very well on the VISal set. In addition, we have presented the qualitative results in Fig. 7

In this subsection, we verify the effectiveness of each major component in the proposed approach, and the quantitative results are detailed in Table II.

Both baseline models, i.e., the motion saliency deep model (Eq. 6) and the original CPD [71], exhibit the worst results. Then, by using the proposed clustering-based coarse localization (i.e., the ‘+SOPM(MAX)’) to find salient object proposals, the overall performance, which is already comparable to some SOTA models published in 2019, can be improved significantly. Then, by using the proposed data mining scheme, i.e., the binary classifier, the overall performance can be further improved. Quantitative evidence can be seen by comparing ‘+SOPM(MAX)’ with ‘+SOPM(MAX)+CL’, showing the effectiveness of the proposed iterative mining scheme. Since the binary classifier-based data mining scheme is the major component of this paper, we provide an additional quantitative evaluation to further verify its effectiveness in the next subsection.

In our implementation, all object proposal-based saliency maps are pasted back to the original video frame via the maximizing operation (Eq. 19), and we also test the averaging operation, and the result is shown as ‘+SOPM(AVE)+CL’. Clearly, the maximizing operation can more suit the pasting scheme than the averaging operation.

To verify the effectiveness of the proposed online fine-tuning scheme, we test the ‘+SOPM(MAX)+CL+OLF’, where, for each five video frames, one frame is randomly selected as the key frames, and the ‘framewise refiner’ will be fine-tuned on these key frames using their $FS$ (Eq. 4) as the pseudoGTs. Because this random selection process easily introduces some less trustworthy $FS$ into the fine-tuning process, the performance gain achieved by this online updating is very marginal. By using the proposed keyframe selection strategy, which has been denoted as ‘+SOPM(MAX)+CL+OLF(KFS)’, the overall performance can be improved by an average of 2%.

The overall performances of the final saliency maps obtained by using or not using the online fine-tuning scheme are shown in Table III, where ‘Iter0-PA’ and ‘Iter0-OL’ denote the performance of $FS$ (Eq. 19) and the final result (Sec. IV-B) in the ‘Iteration 0’, respectively. Specifically, we also observed that the overall performance seems to reach a plateau after ‘Iteration 3’, where some quantity metrics decrease slightly, e.g., the maxF on the Segv2 set (0.893$\rightarrow$0.885). Therefore, we decided to omit ‘Iteration 4’ and set the maximum iteration number to 3 to ensure computational efficiency.

Actually, the proposed network online fine-tuning process (Sec. IV-B) costs some additional computational time. However, in our case, we adapt only the framewise refiner to the current video sequence, where the training data are relatively small. In addition, to avoid overfitting and consider the fact that the framewise refiner performs very well if it has weakly learned by taking the $FS$ (Eq. 19) of those trustworthy frames as the learning objective, the online fine-tuning process only requires several epochs. As shown in Table IV, we chose 10 epochs as the optimal choice to ensure the overall computational efficiency because the performance gains achieved by using some additional epochs could be very marginal.

The hyperparameter $K$ determines the initial cluttering number, and we have tested multiple choices, i.e., $K$={4,6,8,10}, where the $K$=8 is the empirically selected choice in the previous version. The quantitative results have been shown in Table V. Clearly, $K$=8 is the best choice, where other choices of $K$ may lead to some performance degenerations. Specifically, we have noticed that overall performance decreased significantly when $K$ = 10 over the SegTrack set. The main reason is that the entire birdfall sequence is completely ill-detected. Since the falling bird in this low-resolution sequence is a quite small size object, our clustering process has assigned multiple background proposals with similar appearances to the bird into the salient clusters. W.r.t. other choices of $K$, the overall performances tend to stay stable. Also, another reason for the $K$=8 to be the best choice might be the fact that all other parameters (e.g., the mining iteration times) are selected under the default $K$=8. Thus, it is quite reasonable for other choices of $K$ to exhibit inferior results.

The hyper parameter $B$ is the frame batch size, which determines the overall keyframe number to be used during the model fine-tuning process. Generally, a large choice of $B$ correlates to sparse key frame selections, and a small choice of $B$ will lead to a dense sampling of keyframes. The exact ablation study towards $B$ can be seen in Table VI, where we have tested multiple choices, including $B$={3,5,8,10}, and $B$=5 is the default choice. As shown in the table, the overall performances with different choices of $B$ are very stable, even though $B$=5 slightly outperforms other choices. The main reason for this phenomenon is that there exists a clear trade-off, i.e., a large number of keyframes may lead the fine-tuning process difficult to reach convergency, while a small number of keyframes may result in over-fitting. Therefore, we continue to use $B$=5 as the best choice.

Actually, both object proposal and optical flow methods affect the overall performance of our method, of course. And the main reason is clear, i.e., all other parameters are assigned based on the combination of SEOD + FlowNet, where new choices of object proposal and optical flow may lead our model to stay in the sub-optimal situation. As shown in Table VII, using either YOLOv4 or LiteFlow could lead to some performance degenerations as expected. However, the quantitative results illustrated in Table VII also suggest that our approach is generally stable. Though the overall performance has been decreased, the overall performances of the two modified versions (i.e., YOLOv4+FlowNet and SEOD+LiteFlow) still significantly outperform the motion saliency (MS) baseline. Also, this experiment has been included in the revised version.

We show some failure cases in Fig. 9, In fact, these failure cases are mainly induced for three main reasons: 1) the binary classifier is not always correct, 2) the patchwise refiner can produce some missing detections, and 3) the proposed key frame strategy might include some low-quality $FS$ in the online fine-tuning process. One possible method for improvement might be to fully implement our approach in an end-to-end manner, which deserves our future investigation.

The major limitation of our approach might be its offline data mining methodology, which includes multiple sequential steps needing to use multiple off-the-shelf tools such as optical flow tools, object detectors, and ISOD models. Since our approach is not an end-to-end model, our method is quite time consuming in essence, and we detailed the time cost of each major step in Table VIII. Compared with the most representative SSAV19 with 20 FPS (frame per second), the FPS rate of our approach is only 2.39.