A positive feedback method based on F-measure value for Salient Object Detection

FCNs-based SOD models mainly design a series of decoders around feature extraction, refinement or enhancement, and multi-level feature fusion and set specific loss functions to assist model training.

The successful performance of SOD models depends on the effective integration of multi-level features. Several studies have highlighted the significant differences between the low-level and high-level features extracted from backbone networks. To address this issue, Chen et al.[21] proposed a progressive context aware feature aggregation module, while Dai et al.[10] developed a middle layer feature extraction module to achieve better feature fusion results. Moreover, the integration of multiscale features has been shown to improve SOD detection performance in various studies. For instance, Zhang et al.[15] developed a neural structure based search unit to automatically determine the multiscale features that need to be aggregated. Fang et al.[9] proposed a densely nested-based network framework to utilize multiscale high-level feature maps effectively. Meanwhile, Zhuge et al.[8] used various multiscale feature extraction methods to obtain diverse multiscale features and designed an integrity channel enhancement module to highlight salient objects. Base on work[22], Wu et al. [14] proposed a dynamic pyramid convolution to extract multiscale features instead of fixed size convolutions. Effective supervision strategy have also been employed to help models predict salient objects more accurately. For instance, Wei et al.[22] proposed pixel position aware losss to guide networks to pay more attention to local details, while Yang et al.[23] proposed progressive self-guided loss to guide network learning to more complete salient regions. Additionally, Xu et al.[16] designed a knowledge review network to roughly locate salient regions first and then finely segment salient objects. Wu et al. [7] proposed a decomposition and completion network to predict the saliency, edge, and skeleton maps respectively, and then filled the saliency map using edge and skeleton maps.

In the field of SOD, the incorporation of global context information is extremely crucial for accurately and completely predicting salient regions. Transformer-based network architectures, renowned for their ability to capture long-range dependencies, have been successfully introduced in this domain.

To this end, Liu et al.[17] proposed a unified transformer-based model for SOD, which facilitates the propagation of global context information among image patches. Meanwhile, Zhang et al.[18] developed a generative vision transformer network that generates a pixel-level uncertainty map, effectively representing the significance confidence of salient objects. Additionally, Ren et al.[24] proposed a simple yet effective deeply-transformer network that preserves more unifying global-local representations to gradually restore spatial details.

To enhance the accuracy and completeness of prediction results in the field of SOD, it is important to effectively extract and utilize both global context from deep features and local context from shallow features. However, achieving this in network frameworks can be challenging. Therefore, some hybrid network frameworks have emerged as the times require.

Zhu et al.[25] proposed a deep supervised fusion transformer network, extending the applicability of FCN to the transformer architecture for the first time. They employed a transformer encoder to extract multiscale features and designed a multiscale aggregation module to aggregate these features in a coarse-to-fine manner. Similarly, Yun et al.[19] proposed a self-refined transformer network that leverages the transformer encoder to capture long-distance dependencies and designed a context refinement module. The module is employed to integrate global context with decoder features and refine and locate local details automatically. In contrast, Wang et al.[26] still used FCN as an encoder and utilized a transformer module for multi-level feature fusion to address the limited receptive field of FCN.

Figure 1 shows structure of the proposed method, which includes a multi-branch model structure and a positive feedback prediction structure. The former is used to place existing SOD model sequences, including traditional algorithm models, deep-learning supervised or unsupervised models. The positive feedback prediction structure will iteratively calculate the respective weights according to the outputs of all branches to obtain the final saliency maps, where the weight calculation process is completely adaptive. In order to verify our method more conveniently, we use the precious models in advance to generate saliency maps, which are marked as $Smp_{i}(i=1,2,3,4)$.

The positive feedback prediction algorithm is summarized in Table 1, and its conceptual principle is illustrated in Figure 2. Hereinafter, $smp_{n},N$ refers to the saliency maps generated by the multi-branch models and the number of branch. $\varepsilon$ is a threshold, which is set to 0.95. The $imbinarize$($\cdot$) is a binarization function and the $mat2grey$ is a graying function. The $computerFmeasure$($\cdot$) is the function for calculating $F$-$measure$[27].

The key steps of the algorithm:
Step1. Input: outputs ($smp_{n}$) of the multi-branch model structure are used as inputs of the positive feedback prediction structure, and perform the binarization function to get binary maps ($b\_smp_{n}$).
Step2. Initialization: the above inputs are fused in the way of pixel-level addition, and perform the binarization function to get binary maps ($B\_Smp_{0}$). At this time, the weights of each branch are consistent.
Step3. Iteration: calculate $F$-$measure$ with the input ($b\_smp_{n}$) of each brach and the latest fusion result ($B\_Smp_{i-1}$). Update the weights of each branch, recalculate the fusion result ($Smp_{i}$) from the new weights and perform the binarization function to get binary maps ($B\_Smp_{i}$).
Step4. Judgement: calculate $F$-$measure$ with the latest and last fusion result and compare it with $\varepsilon$. If it is greater than the threshold, the two successive resluts are considered to be simmilar enough, and the latest fusion result is taken as the final output; Otherwise, execute Step3 again.

To validate the efficacy of the proposed method in this paper, we performed a series of experiments on five publicly available datasets. The datasets utilized in this study are briefly described as follows: DUTS[28] comprises a total of 10,000 images for training and 5,000 images for testing, with only the test set being used in this study. DUT-OMRON[29] consists of 5,019 images with complex structures and backgrounds. HKU-IS[30] contains 4,447 maps with multiple salient objects. PASCAL-S[31] contains 850 natural images. ECSSD[32] comprises a collection of 1,000 images obtained from the internet.

To quantitatively evaluate the proposed method, we adopt six evaluation metrics as the performance measures, including mean absolute error ($MAE$)[33], maximum F-measure ($mF$) score, and S-measure ($Sm$) score[34], precision-recall ($PR$) curves.

To facilitate the testing of the proposed positive feedback method, we implemented the following steps. Firstly, we generated prediction maps of a multi-branch model using the model disclosed by the author and Python tools. Secondly, we evaluated the positive feedback method using Matlab tools.

To evaluate the effectiveness and characteristics of our proposed method, we conducted comparative, ablation, and robustness experiments. In the comparative experiment, we compared the performance with other state-of-the arts models. Results showed that the positive feedback mechanism improved the model’s performance on multiple metrics. In the ablation experiment, we compared the performance of the positive feedback mechanism with pixel-level addition fusion on five datasets. In the robustness experiment, we manually selected good and bad renderings from each model and input them into the positive feedback mechanism to observe the visual comparison results.

In total, we conducted two sets of experiments. In the first set, we used four branches, namely MSFNet (2021)[15], PAKRNet (2021)[16], ICONet (2022)[8], and SelReformer (2022)[19]. The performance of the 2021 methods are weaker than that of the 2022 methods. In the second set, we used two branches, namely DPNet (2022)[14] and SelReformer. The performance of these two methods is relatively close. For detailed experimental procedures and analysis, please refer to each subsection.

We compare the proposed method with 12 state-of-the-art methods, including DCNet[7], BiconNet[35], DNTD[9], EDNet[12], EFNet[10], MRINet[11], RCSBNet[13], MSFNet[15], PAKRNet[16], ICONet[8], DPNet[14], SelReformer[19]. To ensure a fair and objective comparison, we will utilize the saliency maps provided by the authors and employ the same standardized evaluation function to calculate each metric.

In Table 2, we present the quantitative comparison results based on MAE, maximum F-measure, and S-measure. Our methods demonstrate the most comprehensive performance across all five datasets, as evidenced by our superior results across all three metrics. Notably, our method, Our-SS, consistently outperforms the second-best result in terms of mF and Sm by 1.5%, 0.4%, 0.4%, 0.5%, and 0.9%, 0.8%, 0.6%, 0.9% on DUTS, ECSSD, HKU-IS, and PASCAL-S datasets, respectively. Additionally, our method, Our-SSSS, performs exceptionally well on ECSSD, HKU-IS, and PASCAL-S datasets, particularly on the DUT-OMRON dataset, where it consistently outperforms SelReformer[19], the strongest model among the four-branch structures. Moreover, in Figure 3, we provide precision-recall curves (PR) for all five datasets. Our curves demonstrate exceptional performance across most thresholds, particularly on the DUT-OMRON and DUTS datasets.

Based on the results of the two sets of comparative experiments on ablation, we can derive several professional conclusions. Firstly, in cases where there is a significant difference in performance between the selected methods and their placement in a multi-branch structure, the proposed method exhibits greater advantages over direct fusion. Conversely, when the performance difference between the selected methods is minor, the proposed method can only exhibit limited superiority over direct fusion. Additionally, it is important to note that the performance synthesis level achieved through positive feedback fusion is consistently superior to that of each individual model within a multi-branch structure.

To assess the proposed method’s robustness against interference, we conducted adversarial experiments on a four-branch structure. Despite the outstanding detection performance of many existing methods, their capabilities are not consistently strong, as depicted in Figure 4. To better showcase the superiority of our approach, we manually selected several sets of input predictions with both desirable and undesirable effects for the proposed method, and presented visual comparisons in Figure 4. This not only explains why the proposed method is superior to methods introduced into a multi-branch structure, but also superior to direct fusion, as it allows for automatic weight updating to achieve satisfactory results, instead of treating all branches equally. Furthermore, it should be noted that even in the absence of good predictions in the four-branch structural model, a favorable outcome may still be yielded, as demonstrated in the seventh row.