MTSIC: Multi-stage Transformer-based GAN for Spectral Infrared Image Colorization

Infrared images often suffer from detail loss and low quality, making conventional grayscale colorization methods ineffective. Specialized models are needed to improve feature extraction and reconstruction. Current approaches fall into two categories: CNN-based and GAN-based methods [10]. CNNs, such as lightweight UNet variants like TIR [21], are efficient for simple scenes but lack detail recovery. Limmer et al. [22] introduced high-frequency feature transfer to preserve some details, but performance remains limited. GAN-based methods, leveraging strong generative capabilities, are widely adopted. Kuang et al. [6] enhanced Pix2Pix [16] with improved generators and loss functions to boost realism, but small target details were lost. Chen et al. [7] and Sigillo et al. [23] refined TICC-GAN [6] to enhance edges, yet unrealistic colors and texture loss persist. Lou et al.’s MornGAN [8] uses a ResNet generator and memory guidance to reduce feature aliasing. However, its segmentation loss relies heavily on the performance of the segmentation network, and the generated colors are unrealistic. He et al. [24] combined UNet with ViT for better semantic decoding, though small target confusion remained. Chen et al. [10] improved generator (UNet++) in FRAGAN for better detail and semantics, but small target colorization and structure reconstruction are still limited. Liao et al.’s MUGAN [4], merging UNet++ and UNet3+ variants as generator, also struggles with small target loss and geometric distortion in complex scenes.

Overall, despite progress in deep learning for infrared image colorization, two main challenges remain: (1) semantic confusion and unrealistic colors in cross-domain colorization, which hinder accurate small target recognition and content consistency; (2) reliance on single-band images and focus on vehicular scenes, limiting effectiveness in complex scenarios like remote sensing. As infrared images are widely used in remote sensing tasks such as object extraction, change detection, and environmental monitoring [2], integrating multi-band information is essential to improve colorization quality and address current limitations.

Multi-band infrared spectral images offer richer radiative information, higher material identification accuracy, and better adaptability than single-band images [12]. By capturing responses across multiple narrow bands, they enable precise identification, especially for distinguishing similar hetero-isomers, and are widely used in applications like driver assistance and road segmentation [25].

Colorizing multi-band infrared images enhances visualization and detail perception. TeX Vision, introduced in HADAR [15], generates pseudo-color images in HSV space using a thermal signal equation based on temperature (T), emissivity (e), and texture (X). However, the multi-to-one mapping from thermal radiation (S) to T/e/X makes the problem ill-posed [26], reducing recognition accuracy in complex scenes and for small targets. TeX Vision also depends on pre-built material libraries, which may not generalize to new materials, and suffers from high computational complexity and sensitivity to environmental factors like temperature and humidity.

To address these issues, this paper proposes a deep learning-based, data-driven method that learns semantic and color mappings between multi-band infrared and visible images to generate natural-looking visible images. The approach generalizes well to complex scenes such as remote sensing and can also be applied to single-band infrared image colorization, improving the applicability of infrared image colorization.

Originally designed for machine translation, Transformers have been widely adopted in vision tasks like super-resolution [12], classification [27], and object detection [28]. However, traditional Transformers focus on spatial modeling and are less effective for spectral images due to spatial sparsity and strong spectral correlations, making them ill-suited for handling spectral redundancy. To address this, we introduce Spectral Multi-head Self-Attention (SMSA) for infrared image colorization. SMSA computes self-attention along the spectral dimension, capturing both local and global spectral dependencies for improved representation. While GAN-based generators typically use ResNet or UNet variants, Transformer-based approaches for multi-band spectral image colorization remain largely unexplored.

In addition, frequency-based deep learning methods have gained interest. Most infrared colorization methods rely on spatial features and overlook frequency information. CNNs are biased toward low-frequency components [13, 29], prompting recent efforts to incorporate high-frequency representations—such as Fourier [30] and Haar wavelet [31] transforms—into vision tasks [29]. Frequency-domain analysis complements spatial features, particularly in regions of infrared images with significant grayscale variations. Since wavelet transforms provide both spatial and frequency information, support multi-resolution analysis, and are computationally efficient [29]. In this work, we integrate wavelet transforms into the Transformer framework to enhance multi-band infrared spectral image colorization.

The overall architecture of MTSIC is shown in Figure 2 (a). A pre-trained ConvNeXt [32] is employed to efficiently extract spatial features, inspired by pixel-wise semantic segmentation to meet the fine-grained requirements of colorization. Given an input image $I_{x}\in\mathbb{R}^{L\times H\times W}$, where $L$ is the number of spectral channels, a $3\times 3$ convolution is first applied to map the channels, followed by ConvNeXt to extract four hierarchical features: $x_{1}\in\mathbb{R}^{C\times H\times W}$, $x_{2}\in\mathbb{R}^{2C\times(H/2)\times(W/2)}$, $x_{3}\in\mathbb{R}^{4C\times(H/4)\times(W/4)}$, $x_{4}\in\mathbb{R}^{8C\times(H/8)\times(W/8)}$, where $C=96$. Features $x_{2}$, $x_{3}$, and $x_{4}$ are individually aligned via interpolation and then concatenated with the feature $x_{1}$. The concatenated features are subsequently fused through convolution, enhancing the representation of both local details and global structures.

The MTSIC comprises $N_{s}=3$ cascaded STformer modules that reconstruct the corresponding daytime visible image from pre-trained spectral features. Local and global skip connections alleviate vanishing gradients and enhance feature reuse, improving training stability. The fusion of shallow and deep features further boosts convergence and representation.

Figure 2 (b) illustrates the U-shaped STformer architecture, which consists of an encoder, decoder, and a bottleneck containing the MSWB module. Both the embedding and mapping layers use single-layer 3$\times$3 convolutions. In the encoder, features pass through the embedding layer, two SARB modules, two $4\times 4$ downsampling convolutions (stride=4), and four additional SARB modules. The decoder mirrors this structure, using 2$\times$2 transposed convolutions (stride = 2) for upsampling. Features from the encoder and decoder are fused via channel-wise concatenation followed by a 1$\times$1 depthwise separable convolution (DSConv), enhancing multi-resolution representation and reducing computational complexity. This fusion strengthens the connection between local and global information, improving overall network performance.

Figure 2 (c) illustrates the structure of the SARB, which consists of a residual block, a feed-forward network (FFN, as shown in Figure 2 (d)), spectral multi-head self-attention, and two layer normalization operations.

Spatial-Spectral Attention Residual Block (SARB): The module fuses spatial-spectral information through multiple sub-units. Residual blocks first extract spatial features from the input $\mathbf{y}_{in}$, improving spatial information modeling. A spectral multi-head self-attention (SMSA) mechanism then captures inter-band correlations to enhance spectral features. Finally, a feed-forward network (FFN) learns the mapping between spatial and spectral features using nonlinear activation and fully connected layers. Residual connections ensure efficient information flow, boosting expressiveness and mitigating overfitting.

Spectral Multi-head Self-attention (SMSA): Assuming the input $Y_{\text{{in}}}\in\mathbb{R}^{H\times W\times\textit{dim}}$ is used as the input to the SMSA, it is first reshaped into a matrix $Y\in\mathbb{R}^{HW\times\textit{dim}}$. Then, $Y$ is linearly projected to obtain the Query matrix $Q\in\mathbb{R}^{HW\times\textit{dim}}$, Key matrix $K\in\mathbb{R}^{HW\times\textit{dim}}$, and Value matrix $V\in\mathbb{R}^{HW\times\textit{dim}}$, which are computed as follows:

The learnable projection weight matrices $W^{Q}$, $W^{K}$, and $W^{V}$ are used for the Query, Key, and Value, respectively, forming the basis for spectral self-attention computation. For simplicity, biases are omitted. The model follows a U-shaped architecture with multiple SARB modules, where the feature dimension changes dynamically across layers. The number of heads $N$ is determined by the feature dimension dim and base dimension $C$, with $N=\textit{dim}\mathbin{//}C$. The feature dimension doubles in the encoder and halves in the decoder, ensuring their symmetry. $Q$, $K$, and $V$ are split into $N$ heads along the spectral channel: $Q=[Q_{1},\dots,Q_{N}]$, $K=[K_{1},\dots,K_{N}]$, and $V=[V_{1},\dots,V_{N}]$, each with dimension $d_{h}=C$.

Multi-scale Wavelet Block (MSWB): As depicted in Figure 4, the MSWB consists of three branches to enhance feature selection and representation. The global branch employs CBAM [33] to enhance key feature identification through global spatial and channel recalibration, offering lower computational overhead compared to SMSA. The local branch employs consecutive multi-scale spatial pyramid pooling (5$\times$5, 9$\times$9, 13$\times$13) to extract features with varying receptive fields, while LayerNorm is incorporated to enhance the network’s nonlinear representation capability. Multi-scale information is fused through dimensional concatenation and convolution, effectively enriching detail representation. Since infrared images are discrete non-stationary signals containing multiple frequency components, the Haar wavelet transform is employed for multi-resolution decomposition to effectively capture local features and adapt to signal characteristics. In this work, the transform is used to map features into the frequency domain, decomposing them into a low-frequency component and three high-frequency components corresponding to horizontal, vertical, and diagonal directions, thereby enhancing edge and texture extraction. The wavelet branch converts spatial features into high-frequency features rich in local details and low-frequency features containing global information, providing more comprehensive frequency domain information. All branches utilize CBR units composed of convolution, Batch Normalization, and ReLU, where Batch Normalization improves training stability and ReLU enhances nonlinear representation capability.

where $Y$ is the STformer encoder output, and $f_{g}(\cdot)$, $f_{m}(\cdot)$, and $f_{w}(\cdot)$ represent the global, local, and Haar wavelet transform branches, respectively. The features are denoted as $y_{g}lobal$ (global), $y_{local}$ (multi-scale local), $y_{L}$ (low-frequency), and $y_{H}$ (high-frequency). The Haar wavelet transform downsamples by a factor of 2 ($\downarrow_{2}$).

Subsequently, the spatial features and frequency-domain features are jointly fed into a spatial-frequency feature fusion module (SFFM), which aligns and integrates local high-frequency and global low-frequency information to obtain more discriminative hybrid features. Finally, the fused features are upsampled by a factor of 2 ($\uparrow 2$), and a global skip connection integrates shallow features to further improve performance.

where, $f_{\text{SFFM}}(\cdot)$ denotes the spatial-frequency feature fusion module. $y_{gL}^{\prime}$ and $y_{lH}^{\prime}$ represent the global features fused with frequency-domain information and the multi-scale local features, respectively. The final output $y^{\prime}$ is obtained by fusing the shallow input features and the hybrid features via a global residual connection.

The MSWB adopts a simple structure, avoiding the complexity of STformer. By fusing spatial and frequency features, it expands the representation space while retaining frequency information, improving accuracy and robustness in category recognition. Next, we introduce its core component—the SFFM module for semantic fusion.

The SFFM employs large strip convolution kernels ($1\times k$ and $k\times 1$) for multi-scale feature extraction, which expands the receptive field and reduces parameters. To overcome the fixed-direction limitation, direction-aware strip convolutions [34] ($45^{\circ}\,\nearrow$, $0^{\circ}\,\rightarrow$, $135^{\circ}\,\searrow$) are introduced, enabling flexible multi-directional feature extraction. Feature fusion is performed via dimensional concatenation and $1\times 1$ convolutions. Additionally, the SFFM adopts a mirrored structure to fuse features from two branches. After extracting features with strip convolutions, the convolution results along the same direction are cross-fused between two branches. On this basis, local features extracted by 3×3 square convolutions are added to complement the linear features from the strip convolutions, further enhancing the feature representation capability.

where, $y_{\mathrm{spa}\downarrow 2}$ denotes the spatial-domain input features after downsampling. To aggregate features from different branches, the spatial-scale features are downsampled at the same rate as in the wavelet transform, denoted as $\downarrow 2$. $y_{\mathrm{fre}}$ represents the input features in the frequency domain. $\operatorname{\textit{Conv}}_{k}$ denotes a $k\times k$ convolution, $\operatorname{\textit{SConv}}_{k}$ refers to asymmetric convolutions with kernel sizes of $1\times k$ and $k\times 1$, $SConv_{\nearrow\rightarrow\searrow}^{1\times 9}$ refers to directional strip convolutions applied along the directions ($\nearrow$, $\rightarrow$, $\searrow$), and $\operatorname{\textit{LN}}(\cdot)$ represents layer normalization. $y_{\text{SConv}_{7}}$, $y_{\text{SConv}_{11}}$, and $y_{\text{SConv}_{21}}$ denote the output features of directional convolutions ($\nearrow$, $\rightarrow$, $\searrow$) with kernel sizes of $1\times k$ and $k\times 1$. $y_{\operatorname{mix}}^{\prime}$ denotes the output features of the multi-scale module, corresponding to $y_{gL}^{\prime}$ and $y_{lH}^{\prime}$ in Equation (5), respectively.

The SPatchGAN [35] discriminator, depicted in Figure 2 (f), improves discrimination via multi-scale feature extraction and statistical computation. It takes an image $x$ (real or generated) and outputs $\mathcal{D}_{m,n}(x)$, where $m$ is the scale index and $n$ is the statistical feature index. The structure consists of an initial feature extraction block and $M$ scale modules, each containing a downsampling block, an adaptation block, a statistical feature computation block, and $N$ MLPs. A simplified version with two scale modules is provided in Figure 2 (f). Feature maps are generated, reused at higher scales, and adjusted by the adaptation block. After extraction, image features go through downsampling, adaptation, statistical computation, and MLPs to produce scalar outputs.

Unlike traditional PatchGAN [16], which shares convolutional weights, SPatchGAN uses global statistical features for more stable and accurate discrimination, overcoming local receptive field limitations. Each feature is processed by an independent MLP, improving accuracy. Statistical matching is more reliable at lower scales, though harder at higher scales, where it approximates the input distribution. SPatchGAN aims to match the distribution of statistical features, not the input image distribution. We adopt key parameter settings from the SPatchGAN discriminator.

The loss function measures the discrepancy between predictions and ground truth, guiding optimization. In infrared image colorization, a well-designed composite loss function improves translation quality. Based on this, a composite loss function for spectral infrared image colorization is developed:

cGAN Loss: The traditional GAN model maps random noise to a specific data distribution, but the process is unstable and the output is uncontrollable. Conditional GAN (cGAN) introduces constraints on both the generator and discriminator, providing additional guidance during learning [16]. The cGAN loss function is as follows:

where, $\mathrm{I}_{x}$ and $\mathrm{I}_{y}$ denote the input spectral infrared image and its corresponding ground truth, respectively. The generator G seeks to minimize the objective function, while the discriminator D aims to maximize it.

Pixel Loss: In image reconstruction tasks, $\mathcal{L}_{1}$ loss is commonly used as it penalizes pixel-level errors, aiding convergence. In this work, $\mathcal{L}_{1}$ spatial pixel loss is employed to improve image detail recovery during colorization.

where, $\Theta$ represents the learnable network parameters, $I_{GT}$ and $G\left(I_{x}\right)$ represents the Ground truth and the generated colorized image, respectively.

To minimize the chroma and luminance discrepancies between the generated colorized images and the ground truth, we introduce SAM loss $\mathcal{L}_{\text{SAM}}$ [6], which measures channel similarity by computing the angle between spectral vectors. Based on pixel-space constraints, we incorporate a frequency-domain loss $\mathcal{L}_{\text{fft}}$ [36] to optimize low- and high-frequency components in both spatial and frequency domains, enhancing structure and details. Edge Loss $\mathcal{L}_{\text{edge}}$ [37] is introduced to penalize reconstruction errors in edge regions, improving edge detail recovery.

Objective Function: The combined loss model of $\mathcal{L}_{\mathrm{cGAN}}$, $\mathcal{L}_{\text{pixel}}$, $\mathcal{L}_{\text{SAM}}$, $\mathcal{L}_{\text{fft}}$, and $\mathcal{L}_{\text{edge}}$ is adopted as the initial configuration of the proposed method. Building upon this setup and inspired by the high-performing loss functions in infrared image colorization tasks reported by Kuang et al. [6], perceptual loss $\mathcal{L}_{\text{per}}$, total variation loss $\mathcal{L}_{\text{tv}}$, and structural similarity loss $\mathcal{L}_{\text{SSIM}}$ are incorporated as additional terms to enhance the realism and quality of the generated images. The objective function of the proposed method is defined as follows:

where $\lambda$ denotes the hyperparameter controlling the weighting of each loss function. Empirically, the values are set as $\lambda_{\text{cGAN}}=1$, $\lambda_{\text{pix}}=50$, $\lambda_{\text{SAM}}=0.1$, $\lambda_{\text{edge}}=0.5$, and $\lambda_{\text{per}}=\lambda_{\text{tv}}=\lambda_{\text{ssim}}=\lambda_{\text{fft}}=1$ to ensure that all loss terms remain within the same order of magnitude. This hybrid loss function is designed to optimize network performance through spatial, spectral, and frequency domain reconstruction accuracy.

(a) The KAIST dataset¹¹1https://github.com/SoonminHwang/rgbt-ped-detection is a multispectral pedestrian detection dataset containing 95,000 paired thermal and RGB images from day and night scenes in campus, road, and urban environments [38]. Due to low contrast and noise, thermal images pose detection challenges, which can be mitigated by colorization to enhance detail, contrast, and detection accuracy.

(b) The HSI ROAD dataset²²2NUST-Machine-Intelligence-Laboratory/hsi_road is a real-world hyperspectral road segmentation dataset with 28 spectral bands, consisting of 3,799 pairs of RGB and near-infrared images [39]. Unlike typical urban RGB datasets, it includes diverse road surfaces—such as asphalt, concrete, soil, and sand—under rural and natural conditions, enhancing diversity and representativeness for segmentation tasks.

(c) The IHSR dataset is a self-collected long-wave infrared hyperspectral remote sensing dataset captured in Hengdian Town, Zhejiang, China, covering diverse land-cover types including mountains, urban areas, rivers, parks, highways, and farmland. It offers 110 spectral bands spanning 8.0–11.3 $\mu$m at a 1.0 m spatial resolution, with dimensions of $443\times 17189\times 110$. Corresponding labeled RGB images are $443\times 17189\times 3$.

Experiments were conducted on the KAIST, HSI ROAD, and IHSR datasets. For KAIST, nighttime campus, road, and urban scenes were used, with thermal-RGB pairs resized to $500\times 400$ and center-cropped to $360\times 288$. HSI ROAD images, originally $192\times 384$, were resized to $512\times 450$, each containing RGB and near-infrared hyperspectral pairs. For IHSR, pixel-level registration in ENVI was followed by selecting 20 non-overlapping regions for testing and the rest for training. Due to limited spatial size, overlapping block sampling (stride 24) was used to extract $354\times 354$ training patches.

All datasets were augmented with random cropping to $256\times 256$, rotation, and scaling. Models were trained for 60 epochs with an initial learning rate of $1\times 10^{-4}$, decaying linearly after epoch 30. A batch size of 1 and the Adam optimizer [40] were used. Experiments were run on Ubuntu 18.04 with PyTorch 2.0.1 and an RTX 3090 GPU.

Due to the absence of corresponding daytime visible-light labeled images in the KAIST dataset, semantic segmentation and object detection are used to evaluate the texture and color naturalness of the reconstructed images. For the HSI ROAD and IHSR datasets, network performance is evaluated using four image quality metrics (PQI): PSNR, SSIM [41], NIQE [42], and UIQI [43]. NIQE measures image naturalness by quantifying deviations from high-quality natural image models, while UIQI evaluates luminance, contrast, and structural similarity to the ground truth. PSNR and SSIM focus on content quality, with ideal values of $+\infty$ and $1$, respectively. NIQE and UIQI assess color quality, with ideal values of $0$ and $1$.

In addition, a no-reference metric called Colorful is introduced, with an ideal value of 1, to subjectively assess the color richness of generated images. This indicator captures critical details affecting the visual experience and is based on the evaluation model proposed in [44]. To further evaluate the color distribution differences between the generated images and ground truth, this paper proposes a novel metric named ColorJSD. Unlike pixel-level metrics such as NIQE and UIQI, ColorJSD focuses on the similarity of overall color distributions based on probabilistic divergence. The expression of ColorJSD is defined as follows:

where, $P$ and $Q$ represent the color distribution histograms of the generated and ground truth images, respectively, and $M=\frac{P+Q}{2}$ denotes their average distribution. $D_{KL}(P\|Q)$ refers to the Kullback–Leibler (KL) divergence, calculated as follows:

The ideal value of ColorJSD is 0, indicating that the color distribution of the generated image closely matches that of the ground truth.

To validate the superiority of our method, eight representative infrared image colorization approaches were compared, including pix2pix [16], TICC-GAN [6], ToDayGAN [17], FRAGAN_P [10], LKAT-GAN [24], DDGAN [7], MUGAN [4], and MornGAN [8]. Among them, pix2pix and ToDayGAN are general image-to-image translation frameworks, commonly applied to infrared colorization tasks. The others are specifically designed for single-band infrared images. We trained and evaluated all models using their official PyTorch implementations and original parameter settings on the same dataset.

(1) Experimental Results on the KAIST Multispectral Dataset

In the KAIST dataset test, we convert nighttime thermal infrared (NTIR) images into daytime RGB images. Lacking pixel-level aligned RGB labels, we use the Segformer model [45], trained on the Cityscape dataset [46], for semantic segmentation to assess the feature quality and semantic consistency of the converted images. As shown in Table I and Figure 5, our method significantly outperforms others in segmenting small targets (e.g., pedestrians, vehicles), even slightly surpassing the reference nighttime visible images (NVC). This advantage is especially clear in the “sky” category, where low illumination hampers segmentation in NTIR images.

Traditional methods (e.g., pix2pix [16], TodayGAN [17], DDGAN [7], MUGAN [4]) struggle with NTIR’s low contrast and complex structures, resulting in poor segmentation of small targets. While MornGAN [8] performs relatively well, it still misrepresents “building” and “vehicle” categories. In contrast, TICC-GAN [6], FRAGAN_P [10], LKAT-GAN [24], and our method achieve better reconstruction and segmentation, with our approach excelling in both small-target detail and color realism. For example, in Scene 1, our model accurately reconstructs small vehicles and distinguishes “tree trunks” and service lanes. It achieves the highest mIoU, surpassing NVC and exceeding the next-best method, LKAT-GAN [24], by 15.9%. This gain is due to our multi-stage Transformer combined with wavelet transform, which enhances feature extraction and reconstruction, leading to improved segmentation accuracy.

To evaluate the realism of object features in generated images, we use YOLOv7 [19], trained on the MS COCO dataset [20], for object detection. Results are presented in Table II and Figure 6. Methods like Pix2pix [16], TodayGAN [17], DDGAN [7], and MUGAN [4] struggle with complex structures, often producing blurred building edges due to semantic confusion. In contrast, TICC-GAN [6], FRAGAN_P [10], LKAT-GAN [24], and MornGAN [8] improve edge clarity through advanced loss functions and generator designs.

As illustrated in the first two subfigures of Figure 6, most methods fail to reconstruct small targets (e.g., vehicles, pedestrians) accurately. Our method, however, preserves fine details, sharp edges, and realistic colors. Under low-light conditions, YOLOv7 detects only some pedestrians in NVC images, while our method captures full crowd poses and reduces missed detections. Although our recall is slightly lower than MornGAN’s, we achieve higher precision, leading to an 8.2% gain in mean Average Precision (mAP). This confirms our model’s strengths in feature extraction, detail preservation, and colorization.

Additionally, as shown in Table II, although the input infrared images exhibit low Noise, their Colorful is relatively weak. After colorization, this metric is significantly improved, enhancing color representation and subjective perception, which demonstrates the necessity of the colorization process.

(2) Experimental Results on the HSI ROAD and IHSR Hyperspectral Datasets

Compared to the KAIST dataset, the HSI ROAD and IHSR hyperspectral datasets enable better performance in image colorization. HSI ROAD’s 28 spectral bands provide richer textures, helping most methods better reconstruct small object contours like vehicles, pedestrians, and buildings. As illustrated in Figure 7, Pix2pix [16], TodayGAN [17], and DDGAN [7] suffer from geometric distortions and missing details (e.g., Pix2pix fails to restore vehicle outlines; DDGAN omits building edges). While other methods retain global structure, they lack detail fidelity. In contrast, our method best preserves fine details such as utility poles, wires, and building edges. Segformer-based  [45] semantic segmentation further validates this: segmentation results from our generated images closely match the ground truth and outperform those from raw infrared inputs, reflecting higher reconstruction quality.

Quantitative results in Table III confirm our advantage, with the highest PSNR, SSIM, UIQI, and lowest NIQE on HSI ROAD. Notably, our PSNR surpasses MUGAN [4] by 3.9%, and NIQE is optimized by 16.6% compared to TodayGAN [17], indicating more natural and visually realistic outputs.

Compared to the HSI ROAD dataset, the IHSR dataset presents greater challenges due to its large-scale coverage, low spatial resolution (1.0 m), and the presence of 110 thermal infrared spectral bands. These characteristics demand careful modeling of spectral correlations during the colorization process. Despite the limited spatial resolution, the rich spectral information enhances semantic discrimination. As shown in Figure 8, our method achieves superior texture restoration and edge clarity compared to Pix2pix [16], TICC-GAN [6], and MUGAN [4]. It also outperforms FRAGAN_P [10] and DDGAN [7] in detail sharpness and color fidelity, benefiting from targeted network optimizations. Although LKAT-GAN improves the generator architecture, it performs poorly on the IHSR dataset. This suggests that enhancing the generator alone, without adequately accounting for spectral correlations, leads to information loss during the infrared-to-color image cross-domain colorization process, thereby degrading reconstruction quality. As unsupervised methods, TodayGAN and MornGAN do not require labeled data; however, their texture and color restoration on the IHSR dataset are significantly inferior to supervised methods, highlighting the irreplaceable advantages of supervised learning. As shown in Table III, our method achieves the best performance on the IHSR dataset, surpassing the second-best method (FRAGAN_P) with improvements of 12.6% in PSNR, 19.1% in SSIM, and 13.3% in UIQI. Furthermore, it demonstrates notable advantages in the Colorful and ColorJSD metrics, especially in ColorJSD. These results suggest that our approach more effectively extracts deep spatial features and fully leverages spectral correlations, enabling the reconstruction of more natural and vivid colors.

Stage Number Analysis: A multi-stage learning strategy is introduced by cascading multiple STformers to achieve coarse-to-fine reconstruction optimization. The effect of the number of stages, $N_{s}$, on performance is analyzed (see Table IV). Results indicate that performance improves with higher $N_{s}$, but computational cost (parameters and FLOPs) increases linearly. While $N_{s}=4$ offers slightly better performance than $N_{s}=3$, it significantly increases computational cost. Thus, $N_{s}=3$ is chosen for MTSIC, balancing performance and efficiency, and used as the default in subsequent experiments. As illustrated in Table IV, and compared with Table II, even with $N_{s}=1$, the proposed method outperforms several single-band methods and has fewer parameters than high-performance models like LKAT-GAN (83.69M) and FRAGAN_P (52.45M), highlighting the effectiveness and superiority of the proposed framework.

The Transformer-based STformer network includes two key modules: SARB and MSWB. For fair evaluation, only one variable was modified in each experiment while keeping other parameters consistent. Detailed analyses of SARB and MSWB follow.

Complexity Analysis of SMSA: According to [47], G-MSA computes correlations by treating all spatial tokens as queries and keys, with the number of tokens given by $n=H\times W$, resulting in high computational cost. In W-MSA [48], the feature map is divided into non-overlapping $M\times M$ windows, and self-attention is computed within each window. This reduces the complexity but limits the ability to model global dependencies. SW-MSA addresses this limitation by alternating between regular and shifted windows to enhance long-range dependency modeling. However, due to the sparse spatial information in spectral images, its efficiency is lower than that of methods focusing on spectral correlation.

SMSA treats each spectral feature map as a token and computes attention along the spectral dimension, effectively capturing spectral dependencies. The computational complexity of W-MSA, SW-MSA, and SMSA scales linearly with spatial size, i.e., $O(HW)$, which is significantly lower than the quadratic complexity $O(HW^{2})$ of G-MSA. As summarized in Table IV, although SMSA exhibits slightly higher complexity than W-MSA and SW-MSA, it achieves better reconstruction performance, demonstrating a more favorable trade-off between efficiency and accuracy. This indicates that SMSA can more effectively exploit spectral features and is better suited for spectral image colorization than mechanisms focusing solely on spatial dimensions.

Analysis of STformer: The overall architecture of the MTSIC generator is shown in Figure 2(a), where ConvNeXt[32] is used for pretraining to extract spatial features. To establish a baseline, STformer is removed, leaving only the ConvNeXt backbone. To evaluate STformer’s effectiveness, ResNet [49] and UNet++ [50] replace it, while the rest of the architecture remains unchanged. Performance comparisons, presented in Table V, show that removing or replacing STformer results in performance degradation, emphasizing its ability to integrate spatial and spectral features. As illustrated in Figure 9, without STformer, small objects are often missed, and boundary ambiguity arises, further confirming its superiority in feature representation and detail preservation.

Analysis of the MSWB: Table V presents the impact of progressively removing sub-modules from MSWB. ”Global” and ”Local” represent the global and local branches, while ”DWT-L” and ”DWT-H” denote low- and high-frequency features from DWT decomposition. ”SFFM” refers to the spatial-frequency feature fusion module. To validate SFFM’s effectiveness, we compared performance with SFFM removed, and frequency-domain and spatial-domain features fused by concatenation (”w/o SFFM, with Concat”) or element-wise addition (”w/o SFFM, with Add”). Note that spatial-scale features are downsampled according to the wavelet transform scaling ratio to achieve fusion between spatial and frequency domains. Moreover, if any branch of SFFM is removed, its corresponding SFFM alignment module is also eliminated and replaced with an Add operation.

Figure 10 illustrates the impact of removing sub-modules, showing a performance drop in all cases, emphasizing the importance of each component. Replacing SFFM with Concat or Add caused significant degradation on the KAIST dataset due to the semantic gap between frequency-domain and spatial-domain features, which simple fusion methods fail to align effectively. The proposed fusion strategy, however, better captures semantic correlations, improving semantic consistency and segmentation accuracy in cross-domain image colorization.

Figure 11 presents feature maps for each submodule, demonstrating their roles in enhancing feature representation. The Local branch captures fine details (e.g., vehicles), while the Global branch targets larger regions. DWT-H highlights local edge information, and DWT-L improves global perception. However, the Local branch is less effective for complex textures (e.g., buildings). Removing any branch causes structural distortion in reconstructed images and reduces segmentation performance. Compared to removing the MSWB module (“w/o MSWB”), incorporating it (“Ours”) enhances frequency-domain energy and significantly improves spatial visualization, as seen in Figure 11.

Loss Analysis: As described in Section 3, we propose a composite loss function combining $\mathcal{L}_{\text{SAM}}$, $\mathcal{L}_{\text{fft}}$, and $\mathcal{L}_{\text{edge}}$ , improving upon the strategy of Kuang et al. [6]. To evaluate the contribution of each component, we conducted three ablation studies on the KAIST dataset, each excluding one loss term while keeping the network architecture and training data unchanged. The results in Table V and Figure 12 demonstrate that removing $\mathcal{L}_{\text{fft}}$ leads to the most significant performance degradation, highlighting the importance of spatial-frequency optimization. Omitting $\mathcal{L}_{\text{SAM}}$ results in a slight decline in Precision and Recall, suggesting its role in enhancing spectral consistency through angular similarity; moreover, including $\mathcal{L}_{\text{SAM}}$ results in more natural-looking images. Although $\mathcal{L}_{\text{edge}}$ has limited impact on detection performance, it aids in preserving edge structures and semantic consistency. In Figure 12, contours with and without edge loss are overlaid on the input image. The version with edge loss (“Ours”) better preserves contours and and mitigates boundary artifacts.