HSIMamba: Hyperpsectral Imaging Efficient Feature Learning with Bidirectional State Space for Classification

The integration of deep learning into HSI classification has heralded a new era of analytical capabilities, utilizing architectures such as Autoencoders (AEs), CNNs, Recurrent Neural Networks (RNNs), Generative Adversarial Networks (GANs), Capsule Networks (CapsNets), and Graph Convolutional Networks (GCNs) [34, 33, 19, 36, 21, 11]. These approaches have significantly advanced the extraction and classification of features from HSIs, capitalizing on their complex spatial-spectral composition.

CNNs, in particular, have demonstrated profound effectiveness in HSI classification by adeptly capturing local spatial features critical for accurate analysis. Pioneering works by Sharma et al.[24] and Ran et al.[22] highlight CNNs’ versatility in processing HSIs. Sharma et al. introduced a 2-D CNN model that emphasizes local spatial contexts, while Ran et al. developed a dual CNN framework to analyze spectral and spatial information in tandem, augmented with attention mechanisms to refine classification accuracy.

Moreover, the fusion of CNNs with other deep learning structures has yielded significant improvements in HSI classification efficacy. Chen et al. [4] innovatively combined AEs and CNNs, using PCA for dimensional reduction before deep feature extraction, illustrating the synergistic potential of these models. Liu et al.’s exploration of semi-supervised CNN approaches further demonstrates the power of leveraging unlabeled data to bolster model performance.

Additional contributions from RNNs, GANs, CapsNets, and GCNs enrich the HSI classification methodology. Hang et al.’s use of cascaded RNNs [11] for capturing the spectral band sequences, Zhu et al.’s GANs that incorporate PCA components [36], and the spatial-spectral capsule networks by Paoletti et al. [21] showcase the breadth of CNN applications in HSI classification, highlighting their adaptability and efficiency.

Nonetheless, each approach has its inherent challenges. For instance, RNNs might falter in capturing long-term dependencies due to their sequential processing nature. While CNNs have achieved notable successes, their primary focus on spatial information can sometimes overlook the spectral sequence’s critical role in enhancing classification accuracy. Moreover, despite GCNs’ promising outcomes, common issues like computational demand and redundancy in model architectures persist, posing challenges to optimal HSI classification.

Transformers, originally conceived for natural language processing (NLP), have made a significant leap into computer vision, demonstrating their prowess in capturing long-range dependencies within data through self-attention mechanisms. This shift towards utilizing vision transformers (ViTs)  [30]has opened new avenues in remote sensing image analysis, including HSI classification. Notably, the SpectralFormer framework [12] stands out as a seminal transformer-based model tailored for HSI, processing inputs in a pixel-wise or patch-wise manner without conventional preprocessing. Its architecture facilitates enhanced spectral information extraction via cross-layer skip connections, setting a benchmark for transformer applications in this field.

The advent of transformer models has revolutionized visual representation learning, with Vision Transformers (ViTs)[30] marking a significant departure from traditional methodologies by integrating self-attention mechanisms for comprehensive global context embedding. This innovation has paved the way for transformers’ application in HSI classification, exemplified by the SpectralFormer[12], which adopts a transformer-based structure to enhance spectral information extraction without conventional preprocessing. Additionally, the HSI-BERT model [10] employs bidirectional transformer encoders, emphasizing global dependency modeling.

Moreover, the integration of convolutional neural networks (CNNs) with transformer architecture has led to the development of hybrid models. The hyperspectral image transformer (HiT) classification method, introduced by Yang et al. [32]), exemplifies this by embedding convolutions within the transformer framework to enrich it with local spatial contextual insights. This method comprises two pivotal modules: one for generating spatial-spectral local information through spectral adaptive 3D convolution layers and another, the Conv-Permutator, for separately capturing spatial-spectral representations along different dimensions. Further contributions include the multiscale convolutional transformer by Jia et al.  [15], which adeptly integrates spatial-spectral information within the transformer network, and the spectral-spatial feature tokenization transformer (SSFTT) by Sun et al. [26], renowned for its proficiency in generating comprehensive spectral-spatial and semantic features.

The advent of both pure and hybrid attention-based models has significantly advanced the field of hyperspectral image classification. However, the computational intensity of these models presents a notable challenge, hindering further advancements in the field.

Despite the groundbreaking advancements heralded by these models, their computational demand poses a significant challenge, catalyzing the search for more computationally efficient alternatives.

In response to the computational challenges posed by transformers, SSMs, particularly the Mamba model [8], have emerged as a compelling alternative. Characterized by their linear scalability with sequence length, SSMs offer a streamlined approach for modeling long-range dependencies, a crucial feature for the complex data structures encountered in HSI classification  [5].

The Mamba model [7], in particular, represents a leap forward in this domain, demonstrating exceptional performance across various benchmarks with the added benefit of linear scalability. Its integration into computer vision, replacing traditional attention mechanisms with a scalable SSM-based backbone, suggests a paradigm shift towards more efficient processing methods for high-resolution imagery and fine-grained representation analysis.

Recent applications of SSMs in visual tasks have showcased their adaptability, ranging from capturing long-range temporal dependencies in video classification to facilitating high-resolution image generation. The exploration of hybrid CNN-SSM architectures further illustrates the potential of SSMs in addressing long-range dependencies in complex visual data, including hyperspectral imagery [31, 14, 20].

The integration of CNN, transformers, and the innovative use of Mamba in the realm of HSI classification signify a transformative phase in remote sensing analysis, offering sophisticated, efficient solutions for the challenges inherent in processing complex hyperspectral data. As the field continues to evolve, the synergy between deep learning architectures and their innovative applications in hyperspectral imaging promises to unlock new horizons in remote sensing technology, driving the development of models that are not only more sophisticated but also computationally efficient.

We introduce HyperBiRNet, our pioneering model for the analysis and classification of hyperspectral images. Central to our model are the transformation parameters $A$ and $B$, which are ingeniously crafted to exploit the unique spectral characteristics of hyperspectral data. These parameters enable a reversed bidirectional processing approach, enriching the model’s ability to capture and integrate spectral information comprehensively.

Figure 1 illustrates the architecture of the proposed method. The input hyperspectral data $x$ are first reshaped and normalized using a Layer Normalization (LayerNorm) technique. The normalised data are then projected into hidden dimensions using two distinct linear layers, producing $x_{\text{proj}}$ for the forward path and $z_{\text{proj}}$ for the backward path. The backward projection $z_{\text{proj}}$ is reversed along the spectral dimension using the tilt.flip function, resulting in $z_{\text{proj\_reversed}}$.

Both $x_{\text{proj}}$ and $z_{\text{proj\_reversed}}$ are processed through their respective convolutional layers with a subsequent application of the SiLU (sigmoid linear unit) activation function in the backward direction. A delta parameter is also incorporated into both paths, and a hyperbolic tangent (tanh) activation function is employed to refine the convolutional output. The resulting tensors from the forward and backward convolutional paths are averaged along the spectral dimension to reduce the data. Then they are linearly transformed and summed in elemental order to generate the combined spectral characteristic output $y_{\text{combined}}$.

The spectral output $y_{\text{combined}}$ is subsequently mapped to the spatial domain and passes through a linear projection layer. This process prepares the data for the final classification stage, where the classifier determines the class labels for the hyperspectral image data.

The HSIMamba BLock, shown in Fig.1, is an advanced neural network component specifically designed for the intricacies of hyperspectral image data, consisting of images with multiple spectral bands and a three-dimensional spatial structure. the proposed block leverages the rich, multidimensional nature of hyperspectral data.

State updates for each direction are modulated by a delta parameter $\Delta$, which adapts the transformation matrices $A$ and $B$ for discrete-time processing:

The output states from both directions are reduced (e.g., by averaging) and combined to form the final hidden representation $h$:

The combined hidden state $h_{\text{combined}}$ is projected to the output dimension $Y$ through linear transformations $\text{Linear}_{\text{forward}}$ and $\text{Linear}_{\text{backward}}$, and the results are summed to produce the output of the block:

Finally, the output $Y$ is passed through the subsequent spatial processing block before going to the classification block to predict the class labels for the hyperspectral image data.

where,

• •

  $x(t)$: Input data at spectral band $t$

• •

  $x_{\text{proj}},z_{\text{proj}}$: Projected inputs for forward and backward processing

• •

  $x_{\text{forward}},x_{\text{backward}}$: Hidden states after convolution and activation

• •

  $\Delta_{\text{expanded}}$: Delta parameter expanded for broadcasting

• •

  $h_{\text{forward}},h_{\text{backward}}$: Updated hidden states after applying $A$ and $B$

• •

  $h_{\text{combined}}$: Combined hidden state from forward and backward paths

• •

  $\text{reduce}()$: Reduction operation (e.g., mean across the spectral dimension)

• •

  $Y$: Final output after linear transformation

• •

  $\text{Conv1d}()$: 1-dimensional convolutional operation

• •

  $\text{Linear}_{\text{forward}},\text{Linear}_{\text{backward}}$: Linear transformations for output projection

• •

  $f()$: Activation function (e.g., SiLU)

In this subsection, we examine the efficiency aspects of our proposed HSIMamba. To establish the computational benefits of our method, we present a quantitative comparison with existing models like CNNs and Transformers, commonly used in hyperspectral image processing.

Algorithm 1 illustrates the HSIMamba operation procedure. This block incorporates 1D convolutional layers that adeptly capture both the spatial and the spectral dependencies inherent in the data. By utilizing non-linear activation functions like SiLU and tanh, the HSIVimBlock effectively enhances the feature representation capabilities.

Specifically, the operation of HSIVimBlock in Algorithm 1. The input token sequence $\mathbf{X}_{\text{norm}}$ is the first normalized by the normalization layer, the normalized sequence is linearly projected on $x$ and $z$ with dimension size ${E}$ and then $x$ is processed in the forward and backward directions. The outputs of these bidirectional pathways are then integrated through gating mechanisms, producing a combined feature representation, denoted as ${y}_{\text{combined}}$. This comprehensive feature set subsequently undergoes spatial feature extraction, culminating in an output that encapsulates the essential characteristics of the input hyperspectral image data, which can be used for further classification.

Our HSIMamba offer a substantial improvement in computational efficiency without compromising the model’s performance on hyperspectral image classification tasks. The dual-path processing framework ensures comprehensive feature extraction while maintaining a lower computational footprint than its Transformer and CNN counterparts. In the following section, a comprehensive experiments are conducted to verify the proposed architecture and its efficiency.