PixelGame: Infrared small target segmentation as a Nash equilibrium

Infrared small target segmentation models are divided into two categories: Traditional methods based on mathematical modeling with strong prior assumptions and CNN-based methods emerging in recent years. Specifically, the traditional ISTS methods mainly include spatial domain filtering methods and optimization-based methods.

Traditional spatial domain-based methods, such as Top-Hat filtering [9], Max-Median filtering [35], focus on suppressing the background. Compared with the classic Top-Hat filter, Bai et al. [10] constructed a ring filter template through two related but different structural elements to better suppress the background and noise. Deng et al. [36] weighted the image entropy by multiscale gray difference to improve the SCR of small targets. Gao et al. [37] treated the target as a special sparse component in the noise, so as to distinguish the target from similar background noise. Huang et al. [38] used a relatively large density gap between targets and their neighbors to eliminate the interference caused by clutters in complex backgrounds. Moradi et al. [39] proposed a directional approach to enhance the target area and suppress structural backgrounds. Many methods apply multi-scale technology to suppress the background. Nie et al. [40] proposed a multi-scale local homogeneity measure to improve the saliency of small targets. Gao et al. [41] utilized multi-scale gray and variance difference metrics to enhance the feature representation of small target and mitigate background fluctuations, which improves the detection accuracy. In cluttered background, He et al. [42] enhanced targets by exploiting multi-scale differences in intensity distribution changes and gray values.

Inspired by the human visual system, many local contrast measure (LCM) based methods [11], [43] have been explored. Wei et al. [44] presented multiscale patch-based contrast measure to increase the contrast between the target and background. Huang et al. improved LCM from the aspects of multiscale [45], target shape [46] and the difference between the target and the background [47]. Lu et al. [48] utilized a division scheme of surrounding area to capture the derivative properties of the target. These methods extract the difference between the target and the background from various aspects, but their performance is limited when the background changes dramatically or the target is hidden in the background.

The optimizing methods based on low-rank matrix recovery theory assume that the raw image is generated by a low-rank subspace, and the small targets are formulated as sparse singularity. The infrared patch-image (IPI) model [49] regarded small target detection as an optimization problem of recovering low-rank and sparse matrices. Dai et al. [50] used the partial sum of singular values instead of the nuclear norm of IPI to constrain the low rank background patch image. For various highly complex background scenes, Wang et al. [51] combined the total variation regularization term and principal component pursuit (TV-PCP) to comprehensively describe background feature. Wang et al. [52] analyzed the multi-subspace structure of heterogeneous background data, and proposed a stable multi-subspace learning method (SMSL) based on the internal structure of actual images to improve the robustness of the model. Self-regularized weighted sparse (SRWS) [53] model mined the potential information in the background, and transformed the small target segmentation into the optimization problem of extracting clutter from multiple subspaces.

Compared with matrix, tensors have more advantages in handling high-dimensional data [54]. To distinguish real targets from background residuals in heterogeneous scenes, Zhang et al. [55] proposed an edge and corner awareness-based spatial-temporal tensor (ECA-STT) model. Sun et al. [56] extended the properties of multi-subspace to infrared patch-tensor (IPT) structure to better characterize the highly heterogeneous infrared image background. Kong et al. [57] promoted t-SVD to multimodal t-SVD and enhanced the accuracy of background rank representation in the IPT model. The more accurate the assumptions, the better the performance of these existing IPT-based methods. Therefore, the performance of the model based on optimization relies on the constructed data structure and prior assumptions.

Specific assumptions cannot adapt to the open and diversified background environment. The model based on deep learning has a large capacity and contains a variety of different scenes. In recent years, the release of multiple infrared small target datasets has promoted the research of methods based on CNN. Dai et al. [7] proposes the first CNN-based single-frame infrared small target segmentation model, and designs an asymmetric context modulation (ACM) module to fuse high-level semantics and low-level details. ALCNet [59] transforms the traditional local contrast measurement method into a nonlinear module in the convolution network by combining domain knowledge, which alleviates the problem of the minimum internal characteristics of the pure data-driven methods. ISTDU-Net [60] improves the U-Net [61] segmentation model by increasing the response of small target features and suppressing similar background information, which improves the recognition ability to small targets.

ACM, ALCNet and ISTDU-Net take ResNet-20 [62] as the backbone network to extract infrared small target features, and use the IoU-based weighted loss function to guide model optimization. Nevertheless, there are obvious differences in size, energy and layout between infrared and visible targets.

On the one hand, the ResNet structure obtains a large receptive field to fuse context information by sacrificing spatial resolution. Different from large-scale RGB targets, the resolution degeneration may cause IR small targets fail to be segmented. On the other hand, the IoU-based loss function minimizes $FNs$ and $FPs$ simultaneously, which increases the difficulty of model optimization and leads to antagonistic decisions. Minimizing $FNs$ aims to predict as many target pixels as possible, while minimizing $FPs$ trends to predict less number of high-confident target pixels.

Under the framework of generative adversarial networks (GAN) [63], MDvsFA-cGAN [16] used two different segmentation networks as generators, and aims to balance the results of the two generators through adversarial learning between the generators and the discriminator. MDvsFA-cGAN tries to find the Nash equilibrium of the generator and discriminator. In addition, GAN networks are difficult to train and prone to model collapse, which causes the generators to produce samples in the same mode. Differently, our method aims to find the equilibrium state of the two segmentation networks.

The core idea of GAN is game theory [24]. To take advantage of game theory, we design two novel segmentation networks as two players. Under the guidance of the utility function, the two networks play games directly. When two players reach Nash equilibrium in the game, the final results are generated by the segmentation networks.

Generally, the deep learning based game is simplified, which consists of three parts [27]: 1) the participants of the game are neural units or neural networks; 2) the choices of each participant; 3) the objective function of each participant. A game in strategic form is given as follows:

where $\mathcal{T}\{1,2,\ldots\}$ is a set of players. $\varTheta_{k}$ is the set of actions of player $k\in\mathcal{T}$, which is essentially a weight. $\varPhi_{k}$ is a loss function of the player $k$. The strategy of player $k$ is to optimize $\varPhi_{k}$.

Inspired by the outstanding performance of game theory on principal component analysis [28], we regard the task of infrared small target segmentation as a competitive game. Different networks as game players focus on different incorrectly segmented pixels. Under the guidance of the utility function, the two players continue to play the game, and finally reach the Nash equilibrium to output the segmentation results.

Lacking large-scale data sets severely hampered the application of deep learning on small infrared targets. The recently emerged NUST-ISTS [16] is the first large-scale dataset for infrared small target segmentation. NUST-ISTS contains small targets with various real backgrounds, which results in rich data samples. Hu et al. [58] released the first multi-scene dataset for low slow infrared dim-small aircraft target image sequence (DSAT), the centroid coordinates of objects are marked. NUAA-ISTS [7] is the first real scene infrared small target dataset. NUAA-ISTS provides various annotations including mask and bounding box. The detailed properties of those three datasets are shown in TABLE I.

In terms of dataset size, the NUST-ISTS and DSAT data sets have more than 10,000 labeled image samples. The difference between them is that the NUST-ISTS dataset is per-pixel labeled, while DSAT only provides the center coordinates of targets. The advantage of the NUAA-ISTS dataset is that the samples cover a variety of natural scenes, and all samples with various sizes are taken from the real background.

The experiments of our work are mainly carried out on NUST-ISTS and NUAA-ISTS datasets with pixel-wise target masks. Some representative infrared images of the two datasets are shown in Fig. 3.

To further analyze the two datasets, we statistically count the number and size of the targets. The distribution of the number of targets in each image in those two datasets is shown in Fig. 4 (a). It can be observed that more than 80% images contain only one target. In detail, the images in NUST-ISTS mainly contain single target, while images in NUAA-ISTS usually contain multiple targets. Fig. 4 (b) shows the statistical distribution of the target area (number of pixels contained in the target) on each dataset. From the cumulative line chart on the right, we found that more than 90% infrared targets contain less than 100 pixels, which occupy less than 1% in the image. We can see that the targets of infrared images are small, dim and scattered. It can be seen from Fig. 4 (b) that more than half of the targets contains about 20 pixels.

The signal-to-clutter ratio (SCR) [49], [64] is used to measure the target intensity and background intensity. In general, the higher SCR of the target, the easier the target is to be segmented. In infrared dim small target segmentation, SCR is defined as follows,

In Eq. (2), $\mu_{t}$ represents the target intensity, which is the mean gray value of the target region. $\mu_{c}$ and $\sigma_{c}$ represent the mean and standard deviation of the gray value in the target neighborhood region, respectively. The target neighborhood region is set to be three times the size of the target region in this paper.

As shown in Fig. 4 (c), the SCR of about 70% IR small targets is lower than 5. The clutter signals, such as clouds, trees, rocks, ground, etc. account for most energy of the IR images. The target can be regarded as the local extreme point in a specific region, but its energy is very weak in the global background.

Given the above analysis, Fig. 4 indicates that the challenges of ISTS not only lie in the limited information from target, but also lie in the complex and changeful background.

Inspired by the game theory in solving antagonistic decisions, two segmentation networks as two players in the game optimize their utility functions, respectively. The sub-networks $S_{1}$ and $S_{2}$ segment the infrared image $\mathbf{I}$ pixel by pixel, as shown in Fig. 5. Formally, it can be represented as

where $\textit{{O}}_{1}$ and $\textit{{O}}{{}_{2}}$ denote to the segmentation results of two players, respectively.

In the small target segmentation task, the $FNs$ and $FPs$ are difficult to balance delicately. We separate $FNs$ and $FPs$, and employ two players to divide and conquer. In order to obtain better performance, the two sub-networks use different structures according to tasks. FNs-player and FPs-player use fully dilated convolution network ($\mathrm{FDCN}$) with different network depth and dilation factor.

The detailed encoder-decoder structures of $\mathrm{FDCN_{9}}$ and $\mathrm{FDCN_{13}}$ are shown in TABLE II. $\mathrm{FDCN_{9}}$ and $\mathrm{FDCN_{13}}$ are the backbone networks of FNs-player and FPs-player, respectively. In all of the models, the convolutional layers except the last one are followed by batch normalization (BN) [65] and leaky rectified linear unit (leakyReLU) [32]. Specifically, the goal of $S_{1}$ player is to reduce the false negative pixels of targets, optimizing $TNs$ and $FNs$. We employ the shallow encoder-decoder network to extract local information and segment all the pixels of the suspected target. The $\mathrm{FDCN_{9}}$ uses 9-layer convolution, and the dilation factor is increasing from 1 to 16.

Compared with FNs-player, FPs-player increases the accuracy of the predicted pixels belonging to the target class, by optimizing $TPs$ and $FPs$. The pixels predicted by $S_{2}$ may be as precise as possible. FPs-player needs a larger context and better local receptive field, so $\mathrm{FDCN_{13}}$ is deeper and its dilation factor is larger. The $\mathrm{FDCN_{13}}$ contains 13 convolutional layers, and the maximum dilation factor is 64. Finally, the head layer is used to predict the class of each pixel, generating the binary mask of foreground and background.

The information modulation methods represented by the attention mechanism aim to make the model focus on task-related information. In RGB object detection and segmentation, SENet [66] adaptively enhances task-relevant channels by learning the dependencies between different channels. Non-local network [67] is used to capture long-range dependencies and establish the interaction between two pixels with a certain distance on the image. GCNet [33] improves the Non-local network and SENet, enabling query-independent lightweight modules to effectively extract global context information. Triplet attention [34] encodes inter-channel relation and spatial relation, and establishes the dependencies between them to calculate attention weights.

Unlike RGB images, the SCR of infrared dim small targets is very low, and the useful target information is usually submerged in irrelevant clutter and noise. Considering the small targets are arduous to segment, we introduce global max pooling (GMP) [68] and cross-channel max pooling (cMaxPool) [34] to enhance the local salient information of these targets. The $\mathrm{MIM}$ aims to increase the pertinence and capacity of extracted features. In $\mathrm{FDCN_{9}}$ and $\mathrm{FDCN_{13}}$, we add the $\mathrm{MIM}$ module in the skip connection.

The differences between $\mathrm{MIM}$ and other attention modules are highlighted in Fig. 6. It can be seen from the visualization results that the $\mathrm{MIM}$ performs better in capturing infrared small targets with low SCR than other attention mechanisms.

The $\mathrm{MIM}$ enhances the salient information related to the target, and suppresses a large amount of noise and clutters that are not related to the target. In Eq. (4), the $\mathrm{MIM}$ is performed on the features $\mathbf{X}\in\mathbb{R}^{C\times H\times W}$ of each layer of two encoders.

where $m(\cdot)$ represent the $\mathrm{MIM}$, modulation feature $\mathbf{Z}\in\mathbb{R}^{C\times H\times W}$, $C,H$ and $W$ represents the channels, height and width of the feature map, respectively.

The network structure of $\mathrm{MIM}$ is shown in Fig. 7. First, the pixel-wise correlation in the spatial domain is used to obtain cross-channel attention $\mathbf{V_{1}}\in\mathbb{R}^{C\times 1\times 1}$. Specifically, in Eq. (5), the feature map $\mathbf{X}$ is first transformed through point-wise convolution (PWConv) [69] to fuse the relationship between the features of different channels. Eq. (6) makes the $\mathrm{MIM}$ pay more attention to the correlation between different spatial positions of feature.

where the PWConv contains BN, and leakyReLU sequences. $\rho$ is the Softmax function. Dimension reshaped as $\mathbf{X}\in\mathbb{R}^{C\times H\times W}\rightarrow\mathbb{R}^{C\times HW}$, $\mathbf{Y}\in\mathbb{R}^{1\times H\times W}\rightarrow\mathbb{R}^{HW\times 1\times 1}$.

Secondly, different from RGB target segmentation, the targets in ISTS are often relatively small. GMP selects the extreme values in the feature map, it can effectively improve the saliency of the feature map in the region and suppress noise. Global average pooling (GAP) [70] usually pays more attention to large objects. GAP tends to give higher responses to large irrelevant objects and neglect the small extreme regions. However, the small targets to-be-segmented in our task are usually local extreme points in images, GMP extracts the target-related features more effectively. Thus, GMP can reduce the impact of useless background information and highlight striking target information. In Eq. (7), we obtain channel-wise attention $\mathbf{V_{2}}\in\mathbb{R}^{C\times 1\times 1}$:

where $\sigma$ is the Sigmoid function.

We combine different target information obtained by Eq. (6) and Eq. (7) to obtain dual-channel attention $\mathbf{M}_{\mathbf{1}}\in\mathbb{R}^{2C\times 1\times 1}$:

As shown in Fig. 7, the two-layer PWConv realizes the full interaction of different channel information through squeezing and excitation, and further enhances the cross-channel global attention $\mathbf{M}_{\mathbf{2}}$, in Eq. (9).

where $\mathbf{M}_{\mathbf{2}}\in\mathbb{R}^{C\times 1\times 1}$, and the last layer does not include leakyRelu activation.

Then, we use cMaxPool to obtain the most significant information of different channels. $\mathbf{M}_{\mathbf{3}}$ in Eq. (10) is the max channel pooling feature map, where each point in the feature map is the max of the points at the same position within feature maps.

where $\sigma$ is the Sigmoid function, $\mathbf{M}_{\mathbf{3}}\in\mathbb{R}^{1\times H\times W}$.

Finally, guided by Eq. (9) and Eq. (10), $\mathrm{MIM}$ enhances small target features in both spatial and channel dimensions. The final feature is obtained by

where $\mathbf{Z}\in\mathbb{R}^{C\times H\times W}$.

When evaluating segmentation results, the predicted results are often divided into four parts: The number of correctly segmented pixels of target ($TPs$), the number of incorrectly segmented pixels of background ($FPs$), the number of correctly segmented pixels of background ($TNs$), the number of incorrectly segmented pixels of target ($FNs$). The target pixels are composed of $TPs$ and $FNs$. The background pixels contain $TNs$ and $FPs$. The confusion matrix of ISTS is shown in TABLE III. In the confusion matrix, the columns represent the predicted masks O of the pixelGame, and the row represents the ground truth G of the input images.

In order to achieve high-quality results for the FNs-player and FPs-player games, we design a novel utility function according to each player’s own focus and the overall constraints of the game. This utility function consists of three parts: Player utility, game utility and small target constraints. Scientific and effective utility function help the model reach the equilibrium state in the competitive game.

$\mathrm{FDCN_{9}}$ and $\mathrm{FDCN_{13}}$ are the backbones of pixelGame network. A high-resolution prediction feature map is indispensable for small target segmentation. The dilated convolution captures a larger receptive field without reducing the spatial resolution of the feature. The dilation factors of the decoder are symmetrical to that of the encoder. In the encoder-decoder structure, the feature mapping with the same dilation factor exchanges information across layers through skip connection.

The larger dilation factor results in a larger receptive field. On the one hand, some pixels in the large receptive field are not fully utilized. On the other hand, the long-distance dependence of the pixels captured by the large receptive field is not accurate. Therefore, we use $\mathrm{MIM}$ module to improve small object feature representation.

The specific implementation is shown in Algorithm 1.

The experiment is conducted on a computer with 3.0 GHz CPU, 128 GB RAM, and four NVIDIA GeForce RTX 3090 GPUs. The pixelGame is implemented by PyTorch. We use Adam as the optimizer. The mini-batch size is set as 8. The learning rate is set as $10^{-5}$ for the FNs-player and FPs-player. The weights of the players are initialized using the identity initialization technique. All models are trained from the scratch, and the training epoch of NUST-ISTS is 70 and that of NUAA-ISTS is 400.

In this section, we study questions Q1-Q3 raised above.

Finally, we solve the problem Q4 by comparing our pixelGame with several CNN-based methods and other traditional mathematical modeling methods. For a long time, the lack of an open benchmark has been one of the bottlenecks hindering the development of ISTS, which allows various algorithms to be compared fairly. We have summarized the existing target detection and segmentation methods of infrared small targets, which is helpful to promote ISTS work. The comparison between the pixelGame and the SOTA methods is reported in TABLE VI. In experiments, we evaluate the performance of the algorithm at the pixel level using precision, recall, $\mathrm{F_{1}}$ and $\mathrm{IoU}$.

Compared with traditional algorithms, pixelGame is comfortably ahead in both single score (precision and recall) and comprehensive score ($\mathrm{F_{1}}$ and $\mathrm{IoU}$). In CNN-based models, pixelGame better suppresses $FNs$ and $FPs$, and achieves a more delicate balance between precision and recall. In TABLE VI, on both the NUST-ISTS and NUAA-ISTS datasets, our method achieves the highest $\mathrm{F_{1}}$ and $\mathrm{IoU}$ on both datasets, which are 0.64, 0.86 and 0.51, 0.75, respectively. For the foreground-background imbalance problem, our pixelGame achieves the best balance between $FNs$ and $FPs$. From the comparison of precision and recall, which are 0.66, 0.74 and 0.88, 0.86, we can see that the precision and recall of our model are closer and more balanced.

Moreover, it can be seen from TABLE VI that deep learning models perform better than the traditional algorithms. Specifically, the CNN-based algorithms mostly design the loss function of the model at the pixel level, such as MDvsFA-cGAN [16] (based on miss detection and false alarm), ACM [7] and ALCNet [59] (based on $nIoU$). This demonstrates that the pixel-wise loss function has significant advantages in dense segmentation tasks. The traditional algorithms, such as Top-Hat [9] and LCM [11], mostly suppress the background, and enhance the difference between the target and the background at the image level. Similar to the algorithm of recovering the target from the feature space, such as IPI [49] and IPT [53] series, the target position and approximate shape can be determined, but the precise control at the pixel level cannot be achieved.

In the CNN-based methods, most of the existing methods treat different kinds of error pixels as the same to optimize. For example, ACM [7] and ALCNet [59] use $IoU\ loss$ as the optimization function. The integrated loss methods are not suitable for conquering extreme foreground-background imbalance problem of ISTS. MDvsFA-cGAN [16] makes the two sub-networks focus on false positive pixels and false negative pixels by loss re-weighting, respectively. However, in essence, MDvsFA-cGAN optimizes $FNs$ and $FPs$ at the same time, which is not pure antagonistic learning between $FNs$ and $FPs$. Differently, our method based on game theory optimizes false positive pixels and false negative pixels separately. The pixelGame is conducive to flexible selection of different optimization strategies and is easier to achieve Nash equilibrium.

To demonstrate the effectiveness of $\mathrm{MIM}$, we visualize the feature maps enhanced by $\mathrm{MIM}$ module. It can be observed from Fig. 6 that $\mathrm{MIM}$ performs better in capturing IR small targets with low SCR than other attention methods. The main content in infrared image is the building, while the region of target is very small compared with the main building. SENet, GCNet and Triplet attention tend to focus on the main building and neglect the small target. Differently, $\mathrm{MIM}$ prominently enhances the target information and suppresses the background. Benefiting from the different dilation factors, $\mathrm{MIM}$ effectively focuses on the salient region including small targets, which better suits our purpose. Fig. 6 suggests that the enhanced features by $\mathrm{MIM}$ are powerful.

In addition, we design ablation experiments by removing $\mathrm{MIM}$ (denoted as w/o $\mathrm{MIM}$). We apply simple addition when removing each module. From the last two rows of TABLE VI, compared to pixelGame (w/o $\mathrm{MIM}$), pixelGame improves the segmentation accuracy by about 3% and 1% respectively on the two datasets. As can be seen from TABLE VII, $\mathrm{MIM}$ achieves the best performance on both datasets. The F1 score of MIM is about 2% higher on average than other methods. TABLE VI and TABLE VII present the effectiveness of $\mathrm{MIM}$ for local significant context information enhancement in ISTS tasks. The efficient of $\mathrm{MIM}$ is fully proved.

We further study the spatiotemporal complexity of CNN-based methods. The results are shown in TABLE VIII. ACM and ALCNet use ResNet-20 [62] as the backbone architecture. Both MDvsFA-cGAN and pixelGame employ dilated convolutional networks. MDvsFA-cGAN needs more parameters due to more channels of features and fully connected layers in the discriminator. The computational complexity is relatively high. It can be seen from TABLE VIII that pixelGame trades off speed for a high improvement in segmentation accuracy.

Some segmentation results are shown in Fig. 11 and Fig. 12. It can be illustrated from the Fig. 11 and Fig. 12 that the pixelGame segments dim small targets with low SCR accurately and robustly. Due to the low spatial resolution, the targets of NUST-ISTS dataset are small and dim. Some bright background noise is incorrectly predicted as target pixels. The background of NUAA-ISTS dataset mostly comes from real scenes, the contrast between the target and the background is relatively larger, and the segmentation results are more refined.

By and large, the method based on deep learning has obvious advantages in feature representation compared with the traditional methods. Combining the advantages of deep learning and attention model, our method based on game theory achieves a better balance in precision and recall, and achieves the best performance in $\mathrm{F_{1}}$ and $\mathrm{IoU}$. For small targets in infrared images with low SCR, our proposed $\mathrm{MIM}$ effectively improves the quality of extracted features.