Spectral Spatial Neighborhood Attention Transformer for Hyperspectral Image Classification

Abstract

Hyperspectral image (HSI) can provide rich spectral information, which can be helpful for accurate classification in many applications. However, the hyperspectral image classification task has challenges, including limited labeled data, data redundancy, data sparsity, and imbalanced class samples. Over time, various methods have been proposed to solve the above-mentioned problems. To mitigate the issue mentioned above in this paper, we present the neighborhood attention transformer with a channel-wise shift technique for hyperspectral image classification. The neighborhood attention transformer leverages the power of the attention mechanism to capture the spatial relationship between neighboring pixels and extract discriminative features. The channel-wise shift techniques empower the model to adjust the spectral characteristics of each channel adaptively, enhancing its ability to handle the spectral variations present in hyperspectral data. To validate the effectiveness of the proposed model, we conduct comprehensive experiments on the publically available dataset. The results demonstrate that our model consistently outperforms other state-of-the-art methods. The overall accuracy of the proposed model reached $99.41\%,$ $99.93\%,$ $98.35\%$ $99.59\%$ four datasets.

Résumé

L’image hyperspectrale (HSI) peut fournir des informations spectrales riches, ce qui peut être utile pour une classification précise dans de nombreuses applications. Cependant, la classification d’images hyperspectrales présente des défis, notamment en ce qui concerne le nombre limité des données étiquetées, la redondance des données, une rareté des données et un déséquilibre dans le nombre d’échantillons par classe. Au fil du temps, diverses méthodes ont été proposées pour résoudre les problèmes susmentionnés. Dans cet article, nous présentons le transformateur d’attention de voisinage avec une technique de décalage par canal pour la classification d’images hyperspectrales. Le transformateur d’attention de voisinage tire parti de la puissance du mécanisme d’attention pour capturer la relation spatiale entre les pixels voisins et extraire des entités discriminantes. Les techniques de décalage par canal permettent au modèle d’ajuster les caractéristiques spectrales de chaque canal de manière adaptative, améliorant ainsi sa capacité à gérer les variations spectrales présentes dans les données hyperspectrales. Pour valider l’efficacité du modèle proposé, nous avons mené des expériences exhaustives sur l’ensemble de données accessible au public. Les résultats montrent que notre modèle surpasse constamment les autres méthodes de la litératire. La précision globale du modèle proposé a atteint $99.41\%,$ $99.93\%,$ $98.35\%$ $99.59\%$ dans le cas de quatre ensembles de données.

Keywords:

• Attention Module
• Convolutional Neural Network
• Hyperspectral image Classification
• Vision Transformer

Introduction

Hyperspectral sensors capture Hyperspectral image data. The hyperspectral image data consist of hundreds of narrow bands with a lot of information. Hyperspectral image classification (HSI) is a part of earth observation extensively used in agriculture, physics, astronomy, chemical imaging, and environmental sciences (J.M. Bioucas-Dias et al., 2013). Hyperspectral image classification (HSI) data capture spectral information from numerous contiguous spectral bands of surface objects. In the hyperspectral image classification, there are three main challenges.

• First, in the spectral dimension of hyperspectral image data, hundreds of bands exist and have a lot of information. Still, the bands values and information are frequently redundant, which requires a lot of computational costs.

• Secondly, in hyperspectral images, pixels are mixed, interfering with the classification since one pixel usually corresponds to a different object category.

• Third, manually labeling HSI samples is costly and time-consuming, leading to limited availability of labeled samples. Numerous approaches have been proposed in the last decade to address these challenges. This work proposes to solve the problem of limited training samples in HSI data.

Machine learning techniques such as the Bayesian approach (Sahin et al., 2018), multinomial logistics regression (Haut et al., 2017), support vector machine (SVM) (Ye et al., 2022), and many more methods have been developed for analyzing hyperspectral image data. These approaches primarily focused on spectral features while disregarding spatial information, which could yield suboptimal outcomes. Furthermore, researchers have proposed classification methods that rely on feature extraction or dimensionality reduction (Villa et al., 2011) and linear discriminative analysis (Ye et al., 2018). Researchers have proposed different techniques (Zhou et al., 2015) to extract spectral-spatial features, improve the representation of hyperspectral image data, and increase accuracy (Zhang and Lin 2010). The hyperspectral image field recently experienced the deep learning model (Zhang et al., 2018). It achieved significant results due to the deep learning model’s remarkable ability to extract spectral-spatial features. (Özdemir et al., 2014) first introduced a stacked autoencoder for classifying hyperspectral images. (Liu et al., 2017) This research examined the effectiveness of a deep belief network (DBN)-based model in extracting deep spectral feature maps. This approach involves an initial unsupervised pretraining step on unlabeled samples, which is subsequently followed by supervised fine-tuning using labeled samples. The aforementioned models have highly trainable parameters and face challenges regarding spatial information loss, as they require a dimensional input format. Convolutional neural networks made a breakthrough in classification accuracy (Ahmad et al., 2022; Gao et al., 2020; Guo and Zhu 2019; Lee and Kwon 2017; Yu et al., 2020). Compared with a fully connected network, a convolutional neural network (CNN) effectively utilizes the local connection and shared parameters to extract 2D spatial features. Furthermore, (Yang et al., 2016) employed a two-channel convolutional neural network (CNN) to extract spatial and spectral features for hyperspectral image classification (HSI) classification independently. Since hyperspectral data can be represented as a 3D data cube, using 3D kernels to extract spectral-spatial features is intuitive. While the deep learning model mentioned above has made significant improvements in HSI classification, the pursuit of achieving higher classification accuracy with fewer labeled samples remains a crucial objective in HSI classification research (Wang et al., 2021)

This is because convolutional neural networks struggle with modeling long-range dependencies and capturing global context information effectively (Tan et al., 2021). On the other hand, the transformer model utilizes comprehensive contextual information across a wide range by treating the input image as consecutive patches. Initially introduced for machine translation, transformers have emerged as the state-of-the-art approach in various natural language processing (NLP) tasks (Vaswani et al., 2017). In hyperspectral image classification (He et al., 2020), hyperspectral image classification first employed BERT to capture the global dependencies between pixels. (Yu et al., 2022) proposed a multilevel spectral-spatial transformer, improving the classification accuracy by fusing the multilevel features. Sun et al. (2022) proposed a spectral-spatial feature tokenization transformer for hyperspectral image classification. To extract low-level features, the author employed 3D and 2D convolutional neural networks, while high-level semantic features were extracted using a Gaussian weighted tokenizer with a multi-head self-attention transformer. Moreover, these models employ a multi-head self-attention mechanism (MHSA). In hyperspectral image classification, using such transformers in remote sensing is challenging. The author proposed a local and global attention mechanism to extract discriminating features. (Ouyang et al., 2023) the author proposed a hybridFormer using a convolutional neural network and a transformer network. The author proposed a spectral-spatial attention mechanism instead of multihead self-attention. These transformer models required an insufficient number of training samples to extract more discriminating features. Motivated by the transformer attention mechanism, we proposed a neighborhood attention mechanism for hyperspectral image classification. The main contributions to this work are as follows:

• The channel-wise shift technique is applied to recalibrate the channel feature information by emphasizing informative spectral bands while simultaneously attenuating the less valuable ones.

• To utilize the capabilities of convolutional neural networks (CNN), we combine the 3D and 2D convolutional neural network models to capture low-level features.

• The proposed model utilizes the neighborhood attention mechanism to obtain long-range dependencies instead of self-attention, which is computationally too expensive to improve the generalization capability of the model.

The remainder of the paper is as follows Section II provides the details of the proposed model, Section III is about Dataset and Performance evaluation, Section IV describes the result and discussions Section V is conclusion.

Proposed Methodology

In this section, we describe our proposed model shown in Figure 1. The proposed model consists of three main components: channel shift mechanism, spectral-spatial feature extraction, and neighborhood attention transformer.

Figure 1. Proposed neighborhood attention transformer.

Principle Component Analysis (PCA) Process

We adopt the principle component analysis (PCA) step in our proposed work. Some important features are extracted by PCA operation. Typically, in hyperspectral image classification, there are limitations in the number of training samples. This can lead to overfitting issues when training complex model. To address these challenges, we employ PCA techniques in addition, it reduces the dimension of spectral bands and maintains the spatial information as much as possible. It measures the importance of each direction by comparing the variance of data in projection space. The greater variance of the data the more information it contains. In precise terms, Principle Component Analysis optimizes the intrinsic information within after reducing its dimensionality. It evaluates the significance of the projected direction by quantifying data variance. The mathematical process can be succinctly summarized as follows: consider X, which comprises column vectors, each representing a feature vector. By applying algebraic linear transformation to this matrix, the function can be as follows: (1) $$\underset{x}{\mathit{\text{min}}}\mathit{\text{tr}} (X^T\mathit{\text{AX}}), s.t.X^T X=I$$(1)

Where tr and T represent the trace and transpose of the matrix, and A is a covariance matrix. Therefore the output data of PCA can be represented by $Y=X^{T}B$, and the optimal matrix X is composed of the eigenvectors corresponding to the largest n eigenvalues in front of the covariance matrix as column vectors, thus reducing the original dimension of B to the n dimension.

Channel-wise Shift

This work uses the channel-wise shift technique to enhance feature extraction capability and improve classification accuracy. Before incorporating feature information into the convolution module, essential features undergo pre-processing via PCA and channel shift operation. Consequently, the spectral bands in HSI data, following PCA, are organized in descending order of importance. The bands that preserve more spectral data contribute substantially to subsequent feature extraction. The channel-wise shift scheme involves moving the relatively more critical spectral bands at the margins, requiring fewer computations for information retrieval Figure 2 shows the channel wise shift operation.

Figure 2. Channel wise shift scheme.

This approach enhances the count of spatial feature extraction operations for informative channels and guarantees their inclusion in the central region of efficient receptive fields. As a result, important ­spectral bands are positioned in the middle of all channels enabling more convolution operations. Let k be a channel at ($i,j$) position. Suppose we need to move $x_k$ the band into the margin position $\begin{array}{l}X=(x_1,x_2,\dots \dots \dots ,x_i,x_{i+1},\dots \dots \dots ,x_{k-1},x_k)\\ \to X=(x_1,x_2,\dots \dots \dots ,x_i,x_k,x_{i+1},\dots \dots \dots ,x_{k-1})\end{array}$. (2) $$\begin{array}{l}X=(x_1,x_2,\dots \dots ,x_i,x_{i+1},\dots \dots ,x_{k-j-1},x_{k-j},\dots ,x_k)\\ \to X=(x_1,x_2,\dots \dots ,x_i,x_{k-j},\dots \dots ,x_k,x_{i+1},\dots ,x_{k-j-1})\end{array}$$(2) (3) $$\left\{\begin{array}{c}X(:,:,B-2\times i-1)\to X(:,:,i)\text{\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i}\in [0,N/2)\\ X(:,:,2\times i-B)\to X(:,:,i)i\in \left[N/2,N\right]\end{array}\right.$$(3)

The linear operation of channel shift technique can be follow as Equation (3)(3) $$\left\{\begin{array}{c}X(:,:,B-2\times i-1)\to X(:,:,i)\text{\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i}\in [0,N/2)\\ X(:,:,2\times i-B)\to X(:,:,i)i\in \left[N/2,N\right]\end{array}\right.$$(3) .

Feature Extraction

Hyperspectral image data is represented $X\in {\mathbb{R}}^{h\times w\times b}$ where h represents the height of the image w represents is width of the image and b represents is hyperspectral bands. The entire pixel within the region of interest is categorized into p classes, which are represented by $L=(l_1,l_2,l_3,\dots \dots ..l_p)$. To harness the feature extraction abilities of CNNs, we implemented a hierarchical architecture consisting of 3D and 2D CNNs as the backbone network. The 3D adjacent patch $Q\in {\mathbb{R}}^{(w\times w)\times b}$ where w is the small window size of a small 3D cube.

The 2D convolutional layer focuses on extracting the spatial correlation of each channel in given HSI image data. In the 3D convolutional layer, the correlation between different channels is also used to improve the ability of feature representation to obtain spectral and spatial feature maps. To be precise, the 2D convolutional layer is used to extract spatial features, but it cannot obtain significant features from spectral bands. Conversely, the 3D convolutional layer can extract spectral spatial features. Therefore, independently using 2D and 3D convolutional layers is not a good option. For the first layer of the 3D convolution filter and kernel parameter set ($8\times 3\times 3\times 7$), ($16\times 3\times 3\times 5$), ($32\times 3\times 3\times 3$) means eight filters with 3D kernel size ($3\times 3\times 7$) dimension. After completing the 3D convolution operation, we reshape the output of the layer into three dimensions by concatenating the third and fourth dimensions. Finally, we use a 2D convolution layer with the ($3\times 3$) kernel to reduce the number of spectral bands. The feature map generated by the 3D layer has different characteristics, so fusing the 2D convolutional layer with the 3D convolutional neural network layer improves accuracy by involving more complementary information. The mixed-use of 3D and 2D convolutional layers can fully utilize the spectral and spatial information to obtain more discriminative features. As depicted in the Figure 3 a standard 3D convolution operation is illustrated. It becomes evident that the leftmost and rightmost spectral bands receive fewer 2D convolution operations when only a single 3D kernel transverses their surfaces. In contrast, spectral bands positioned toward the undergo a higher number of 2D convolutions, equivalent to the dimensionality of the third dimension of the 3D kernel.

Figure 3. 3D convolution along spectral dimension grey frames represent spectral bands, dark black frames are the filters across spectral bands and red frames represents the filters.

Neighborhood Transformer Encoder

Neighborhood attention (Hassani et al., 2023) can effectively capture the data’s local dependencies and spatial relationships. This mechanism is beneficial for image analysis, where neighboring pixels often contain relevant information for understanding the overall object. Localized self-attention that incorporates inductive biases similar to convolutional operations. These inductive biases mimic the behavior of convolutional operations, which are commonly used in advanced model Swin Transformers (Liu et al., 2021). By incorporating these inductive biases, neighborhood attention eliminates the need for additional computational overhead, such as pixel shift, explored in Swin Transformer. Neighborhood attention constrains the receptive field of the query token to a specific size of neighboring pixels. The aims to establish a local neighborhood window where smaller neighboring regions are given higher priority with local attention. It often improved control over the receptive field, effectively balancing the properties of translational invariance and equivariance compared to another vision transformer Figure 4 shows the neighborhood attention mechanism. Let us have an input image of the spatial location of a neighborhood pixel at ($a,b$) with the lth feature maps i.e., $(a,b,l)$ within the neighbor window is expressed by $\tau (a,b,l)$ which consist of a finite set of indices belonging to pixels close to the location $(a,b)$ of lth feature maps.

Figure 4. Neighborhood attention.

The small window size $S\times S$a neighborhood of $\tau (a,b,l)$ is calculated as Euclidean distance $\parallel \tau (a,b,l)\parallel =S^2$ The attention on a single pixel at location ($a,b$) with the lth feature maps can be expressed in terms of linear projection Q, K, and V of extracted features Y and relative positional biases B($a,b$). The size query is ($1\times Q$) and both keys and values are ($N\times Q$) for all patch matrix. (4) $$q=W_{q}Y,k=W_{k}Y,v=W_{v}Y$$(4)

The dot product between queries and keys and apply a softmax function to calculate the attention map. (5) $$C=\mathit{\text{softmax}}(q_{(a,b,l)}{k}^T+B_{(a,b,l)}/\sqrt{d})$$(5)

Where $\sqrt{d}$ is a scaling factor. It is vital to notice the distinction between self-attention and neighborhood attention. Self-attention allows each token to interact with all other tokens, while neighborhood restricts the receptive field of each token to its nearest area. As a result, the concept of neighborhood attention offers a unique advantage by inherently limiting each pixel to its adjacent region, without any computational overhead. This need for pixel transition to represent interdependence across different windows. It is crucial to highlight that if the neighborhood size of the pixel equals or greater than the feature map size, the outcomes of the self-attention and neighborhood attention will closely resemble those of the original input feature maps.

In summary, the PCA-reduced cube is embedded into the channel shift operation, where the high information bands are placed in the margin to extract more spectral features. After this process, the features will be passed through a features extraction module consisting of a 3D and 2D convolutional neural network (CNN). The neighborhood attention embeds the output of the feature extractor with two successive convolution layers, with the stride yielding the spatial size of one-fourth of the hyperspectral input data. The neighborhood attention employs overlapping convolution; the model consists of two levels, where the first level has three attention blocks while the second level uses four blocks. The encoder block consists of an attention module and multilayer perceptron (MLP) with layer normalization and residual connection. In each encoder block layer, normalization is added before each attention module and an MLP layer, and residual connections are added after each attention module and MLP layer.

Dataset Description and Parameter Analysis

Dataset Description

For the experiment, we use four publicly available hyperspectral datasets including the Xuzhou dataset, Salinas Dataset, Gulf Port Dataset, and Botswana. The Xuzhou dataset acquires a HYPEX spectral camera over the Xuzhou peri-urban site. The spatial dimension of this dataset is $500\times 260$ pixels with a 0.73 m/pixel spatial resolution, and the spectral bands range is 415 to 2508 after removing the noisy band remaining is 436. This dataset has 9 category labeled classes Table 1 shows the detail of this dataset. The second dataset is the Salinas dataset acquired by (AVIRIS) sensor. The spatial dimension of this dataset is $512\times 217$ pixels and the wavelength range is from 400 nm to 2500 nm with a 3.7 m/pixel and 10 nm spatial and spectral resolution, and the spectral bands 224 after removing the noisy band remaining is 204. This dataset has 16 robust labeled classes. Table 2 shows the details of the dataset.

Table 1. Xuzhou dataset labeled samples.

No.	Class	Color	Train	Samples	False color	Ground measurement
1	Barland1		78	26396
2	Lakes		12	4027
3	Coals		8	2783
4	Cement		15	5214
5	Crops-1		39	13184
6	Trees		7	2436
7	Bareland2		20	6990
8	Crops-2		14	4777
9	Red-tiles		9	3070
Total           200   68877

Table 2. Salinas dataset labeled sample.

No.	Class	Color	Train	Samples	False color	Ground measurement
1	Brocoli_g_w_1		7	2009
2	Brocoli_g_w_2		14	3726
3	Fallow		7	1976
4	Fallow_rough_plow		6	1394
5	Fallow_smooth		10	2678
6	Stuble		15	3959
7	Celery		13	3579
8	Graphes_untrained		42	11271
9	Soil_vinyard_develop		23	6203
10	Corn_senesced_g_w		12	3278
11	Lettuce_romaine_4kw		4	1068
12	Lettuce_romaine_5kw		7	1927
13	Lettuce_romaine_6kw		3	916
14	Lettuce_romaine_7kw		4	1070
15	Vinyard _ untrained		27	7268
16	Vineyard_verticle_trellis		6	1807
	Total		200	54129

The third dataset is the Mississippi GulfPort (GP) dataset acquired from the University of Southern Mississippi Gulf Park campus. The dataset consists of 72 spectral bands with the spatial resolution of $185\times 89$ pixels with wavelength range is 375 nm to 1050 nm with a bandwidth of 10 nm. The dataset has 6 land cover labeled classes, the details of this dataset are shown in Table 3. The fourth dataset is Botswana acquired by Hyperion sensor in the Okavango Delta area of Botswana. The spatial resolution of this dataset is $1476\times 256$ pixels with the wavelength from 400 to 2500nm in which the spatial resolution is 30 m and spectral resolution is 10 nm. The dataset contains 145 spectral bands with a spectral resolution is 10 nm. This dataset contains 3248 labeled samples grouped with 14 land cover categories, the details of this dataset including training and testing dataset size in Table 4.

Table 3. GulfPort dataset labeled samples.

No.	Class	Color	Train	Samples	False color	Ground measurement
1	Trees		76	2330
2	Shadows		18	527
3	Grass		32	970
4	Dead Grass		14	438
5	Asphalt		54	1657
6	Dirt		6	175
Total			200	6097

Table 4. Botswana dataset labeled samples.

No	Class	Color	Train	Samples	False color	Ground measurement
1	Exposed soil		5	95
2	Mixed mopane		16	268
3	Short mopane		12	181
4	Acacia grassland		20	305
5	Shurbland		15	248
6	woodland		19	314
7	Island interior		12	203
8	Firescar		16	259
9	Riparian		16	269
10	Reeds		16	269
11	Floodplain-2		14	215
12	Floodplain-1		15	251
13	Hippo grass		6	101
14	Water		16	270
Total			200	3248

Experimental Configuration and Setup

All the experiments were performed on NVIDIA RTX 3060 GPU memory with 64GB RAM using PYTHON 3.8 environment, the initial learning rate is set to 0.001 weight decay 0.00001 with Adam optimizer, Batch size is 64, Epochs set to 100. We chose 200 samples randomly from all classes of four datasets to train the model and the remaining samples were used to test the model. All of the experiments were conducted ten times and average results were recorded. The details of the train and test samples are given in Tables 1-4.

Selection of PCA

HSI contains much redundant information in the spectral bands. Therefore, extracting sufficient spectral information from the image is very challenging. The comparative experiments are conducted without dimensionality reduction, principal component analysis (PCA), and linear discriminative analysis (LDA) pre-processing methods on the Botswana dataset, and the results are shown in Table 5.

Table 5. Dimensionality reduction results on botswana dataset.

Method	OA (%)	AA (%)	Kappa (%)
Without Dimensionality Reduction	92.10(±0.54)	93.05(±0.00)	91.44(±0.59)
With LDA	99.07 (±0.06)	99.12 (±0.08)	99.00 (±0.06)
With PCA	99.59(±0.15)	99.60(±0.15)	99.56(±0.17)

Comparative experiment results show that the principle component analysis (PCA) is a very effective technique and achieves the best classification results; in this context, we adopt PCA.

Influence of Patch Size

Input cubes of a specific size cantered around the pixels are selected to leverage the spectral-spatial information in HSI. Optimal experimental results are shown in Figure 5.

Figure 5. OA comparison at different patch sizes.

OA increases and then decreases when the patch size is on the Gulfport dataset because a larger patch size increases information but introduces a noise factor. When the patch size increase the accuracy decreases for all dataset we chose $9\times 9$ patch size.

Selection of Attention Kernel Size

To constrain the reception field from the image for attention mechanism the kernel size is used it is called attention kernel. Ablation experiments are done on Botswana, GP and Xuzhou dataset with different kernel sizes ranging from $3\times 3$ to $9\times 9$ the results are shown in Table 6. For this experiment, we set $7\times 7$ kernel size.

Table 6. Performance on Botswana, GP and Xuzhou dataset with different kernel sizes.

Kernel Size	Botswana(OA)	GP(OA)	Xuzhou(OA)
$3\times 3$	99.38(±0.52)	97.50(±0.30)	98.25(±0.15)
$5\times 5$	99.47(±0.55)	98.08(±0.15)	99.02(±0.11)
$7\times 7$	99.59(±0.15)	98.35(±0.13)	99.41(±0.07)
$9\times 9$	97.58(±0.39)	97.08(±0.34)	98.90(±0.12)

Result and Analysis

We evaluate the performance of this model by using three widely used metrics: Overall accuracy (OA), Average Accuracy (AA), and Kappa coefficient (K). For comparison, we select some backbone methods, including SSSAN (Zhang et al., 2022), HiT (Yang et al., 2022), SSTN (Zhong et al., 2022), CT Mixer (Zhang et al., 2022), HybridFormer (E. et al. et al., 2023), Transnet (Chen, Lu et al., 2021) and Transfuse (Liu et al., 2021). All the comparison models are transformer-based models consisting of different attention modules. The classification result on four datasets with comparison methods is shown in Tables 7–10.

Table 7. Classification result (%) on Xuzhou dataset.

Class	SSSAN	HiT	SSTN	CT Mixer	HybridFormer	TransUnet	Transfuse	Proposed
1	95.93(±1.49)	94.65(±0.84)	93.84(±2.25)	97.69(±1.59)	95.18(±0.52)	94.12(±0.26)	95.02(±1.22)	99.83(±0.06)
2	96.97(±0.33)	89.90(±9.00)	97.42(±0.88)	97.84(±0.70)	94.75(±0.46)	95.02(±1.2)	95.65(±2.36)	99.62(±0.32)
3	84.37(±0.59)	79.38(±0.36)	61.42(±19.3)	85.40(±2.12)	76.70(±7.26)	86.22(±2.96)	89.19(±6.35)	98.45(±1.55)
4	97.75(±0.91)	65.77(±19.9)	90.85(±0.54)	96.40(±1.34)	93.28(±1.32)	95.27(±1.36)	94.17(±4.14)	99.87(±0.11)
5	96.59(±0.25)	92.04(±2.20)	99.10(±0.11)	86.20(±12.79)	94.10(±0.52)	98.65(±5.30)	97.14(±6.10)	99.78(±0.01)
6	98.99(±0.06)	64.73(±8.54)	45.88(±27.5)	88.78(±4.50)	90.86(±1.35)	93.65(±2.30)	92.56(±2.67)	99.83(±0.17)
7	98.43(±1.00)	75.91(±1.70)	99.86(±0.05)	96.89(±2.59)	70.99(±25.57)	86.25(±6.30)	96.28(±1.96)	99.61(±0.13)
8	99.46(±0.30)	64.68(±27.6)	98.33(±0.11)	99.02(±0.87)	95.79(±3.95)	89.61(±0.89)	98.69(±2.58)	99.82(±0.00)
9	97.07(±0.34)	82.96(±7.00)	94.51(±0.06)	95.39(±0.42)	91.50(±6.37)	96.55(±5.36)	95.36(±3.26)	99.64(±0.35)
OA	96.44(±0.56)	85.51(±5.50)	92.77(±0.89)	94.50(±2.27)	91.33(±2.65)	93.02(±0.22)	95.64(±1.78)	99.41(±0.07)
AA	96.17(±0.18)	78.89(±8.58)	86.80(±4.92)	93.73(±1.97)	89.24(±2.56)	92.81(±2.15)	94.89(±0.22)	99.33(±0.01)
K (%)	95.51(±0.70)	81.28(±7.24)	90.85(±1.18)	92.95(±2.97)	89.00(±3.38)	92.25(±2.2)	94.96(±1.96)	99.25(±0.09)

Table 8. Classification result (%) on Salinas dataset.

Class	SSSAN	HiT	SSTN	CT Mixer	HybridFormer	TransUnet	Transfuse	Proposed
1	99.70(±0.29)	99.52(±0.07)	97.95(±2.04)	99.87(±0.12)	93.10(±4.38)	95.36(±0.25)	96.68(±0.08)	100(±0.00)
2	100(±0.00)	100(±0.00)	100(±0.00)	100(±0.00)	100(±0.00)	99.96(±1.30)	98.98(±0.04)	100(±0.00)
3	100(±0.00)	97.84(±1.09)	61.65(±25.5)	99.89(±0.10)	99.61(±0.17)	96.89(±2.05)	95.67(±0.03)	100(±0.00)
4	98.84(±0.21)	85.42(±6.08)	79.94(±15.38)	99.92(±0.07)	88.62(±7.41)	89.36(±1.65)	90.21(±0.58)	99.57(±0)
5	96.13(±0.14)	96.28(±1.16)	92.78(±0.61)	95.22(±0.16)	94.86(±0.89)	93.64(±3.05)	92.59(±1.25)	100(±0.00)
6	99.91(±0.08)	99.72(±0.01)	99.98(±0.01)	100(±0.00)	99.98(±0.01)	98.37(±0.36)	98.69(±2.05)	99.92(±0.07)
7	99.76(±0.15)	99.91(±0.01)	99.92(±0.05)	99.87(±0.04)	98.47(±0.32)	97.98(±0.98)	98.25(±3.00)	99.94(±0.06)
8	84.36(±1.95)	85.47(±3.17)	87.20(±4.62)	93.70(±0.20)	87.16(±4.77)	92.78(±0.88)	93.19(±1.98)	99.71(±0.28)
9	99.99(±0.00)	100(±0.00)	100(±0.00)	100(±0.00)	99.87(±0.12)	98.78(±0.57)	97.98(±6.35)	99.98(±0.02)
10	98.13(±1.31)	96.26(±0.12)	96.80(±1.40)	97.71(±1.48)	97.76(±1.07)	95.69(±0.69)	96.99(±5.25)	100(±0.00)
11	100(±0.00)	99.01(±0.89)	99.01(±1.26)	99.85(±0.04)	98.87(±0.18)	98.69(±0.58)	98.99(±2.08)	99.90(±0.09)
12	99.94(±0.05)	94.60(±4.97)	47.29(±25.30)	98.46(±1.43)	95.83(±1.25)	89.79(±0.48)	90.54(±4.05)	100(±0.00)
13	89.97(±7.61)	67.63(14.29)	25.61(±13.70)	95.45(±0.16)	82.47(±1.86)	69.51(±0.32)	81.69(±1.07)	100(±0.00)
14	88.08(±3.28)	97.23(±1.36)	78.09(±20.30)	83.48(±12.41)	85.22(±8.77)	85.61(±0.81)	86.96(±0.05)	99.90(±0.09)
15	85.33(±1.87)	77.59(±2.52)	77.20(±10.90)	87.45(±0.75)	69.17(±8.48)	82.96(±1.05)	87.59(±1.89)	99.98(±0.01)
16	91.81(±1.13)	96.72(±1.94)	71.83(±22.83)	97.66(±0.61)	94.33(±2.72)	95.89(±1.00)	89.51(±3.08)	100(±0.00)
OA	93.72(±0.20)	92.13(±0.61)	86.93(±0.26)	96.07(±0.37)	91.16(±0.28)	93.65(±0.05)	94.98(±0.09)	99.93(±0.02)
AA	95.75(±0.11)	93.32(±0.90)	81.23(±0.37)	96.78(±0.89)	92.83(±0.09)	92.57(±0.08)	93.40(±0.08)	99.9(±0.03)
Kappa	93.02(±0.22)	91.24(±0.67)	85.41(±0.32)	95.62(±0.41)	90.15(±0.35)	92.96(±0.00)	93.69(±0.10)	99.92(±0.01)

Table 9. Classification result (%) on GulfPort dataset.

Class	SSSAN	HiT	SSTN	CT Mixer	HybridFormer	TransUnet	Transfuse	Proposed
1	100(±0.00)	64.49(±12.2)	99.11(±0.88)	86.09(±12.72)	94.37(±0.88)	95.59(±1.25)	96.63(±0.28)	100(±0.00)
2	99.65(±0.21)	97.59(±0.71)	99.93(±0.06)	99.90(±0.09)	78.65(±20.22)	96.39(±1.89)	94.99(±3.69)	99.93(±0.06)
3	93.04(±0.82)	66.03(±7.54)	73.70(±1.53)	82.31(±2.83)	73.34(±8.49)	89.63(±0.96)	90.29(±2.96)	99.55(±0.44)
4	96.32(±1.43)	79.74(±8.42)	98.24(±0.47)	96.85(±0.37)	96.53(±3.46)	94.97(±2.09)	95.98(±1.28)	100(±0.00)
5	96.66(±0.39)	82.45(±9.70)	83.13(±1.37)	89.31(±5.98)	84.90(±2.74)	90.89(±3.60)	91.99(±1.96)	94.97(±4.71)
6	98.15(±0.15)	93.83(±1.68)	97.89(±0.06)	98.93(±0.31)	98.20(±0.06)	95.29(±6.09)	95.21(±3.69)	99.50(±0.31)
OA	97.82(±0.25)	88.78(±1.59)	95.52(±0.13)	96.47(±1.23)	89.57(±5.31)	94.58(±1.09)	95.99(±0.28)	98.35(±0.13)
AA	97.30(±0.09)	8069(±3.96)	92.00(±0.25)	92.23(±3.68)	87.66(±3.88)	93.79(±0.09)	94.18(±0.08)	97.00(±0.25)
Kappa	97.07(±0.34)	84.90(±2.18)	93.93(±0.18)	95.21(±1.69)	85.98(±7.09)	92.98(±3.49)	94.08(±0.04)	97.76(±0.18)

Table 10. Classification result (%) on Botswana dataset.

Class	SSSAN	HiT	SSTN	CT Mixer	HybridFormer	TransUnet	Transfuse	Proposed
1	99.80(±0.19)	99.80(±0.19)	100(±0.00)	100(±0.00)	99.80(±0.19)	98.36(±0.18)	96.39(±0.01)	95.16(±4.84)
2	98.94(±1.05)	98.94(±1.05)	100(±0.00)	100(±0.00)	85.26(±3.15)	90.69(±1.08)	91.05(±0.52)	100(±0.00)
3	88.13(±0)	87.92(±9.95)	86.65(±13.34)	73.30(±0)	95.55(±0.63)	96.58(±2.08)	95.69(±2.06)	100(±0.00)
4	100(±0.00)	99.00(±0)	100(±0.00)	100(±0.00)	100(±0.00)	99.38(±1.03)	99.10(±1.00)	99.37(±0.62)
5	97.02(±1.78)	93.05(±1.78)	92.46(±1.58)	98.01(±1.98)	93.84(±3.37)	94.58(±0.89)	95.37(±0.08)	100(±0.00)
6	97.02(±0.59)	92.65(±2.18)	61.70(±18.45)	92.46(±1.58)	85.51(±10.51)	86.19(±3.06)	90.09(±0.98)	100(±0.00)
7	100(±0.00)	100(±0.00)	100(±0.00)	100(±0.00)	99.79(±0.20)	98.69(±5.08)	98.27(±1.08)	100(±0.00)
8	100(±0.00)	98.94(±1.05)	99.73(±0.26)	100(±0.00)	97.89(±1.57)	98.29(±0.01)	99.28(±0.63)	100(±0.00)
9	100(±0.00)	98.47(±1.52)	100(±0.00)	99.66(±0.33)	94.57(±2.03)	95.88(±0.08)	94.98(±3.68)	97.87(±2.13)
10	98.71(±1.28)	98.49(±1.07)	98.28(±0.42)	99.57(±0.42)	90.12(±4.72)	96.02(±2.69)	97.59(±5.08)	100(±0.00)
11	93.70(±4.54)	93.35(±2.79)	89.86(±2.79)	92.83(±2.27)	94.75(±4.19)	95.99(±0.02)	96.31(±6.38)	100(±0.00)
12	95.88(±1.17)	87.64(±0.58)	65.29(±0.58)	100(±0.00)	85.88(±11.17)	89.36(±1.69)	91.81(±4.08)	100(±0.00)
13	100(±0.00)	97.41(±2.19)	99.00(±0.99)	99.40(±0.59)	99.40(±0.19)	95.67(±0.09)	94.89(±0.19)	100(±0.00)
14	98.31(±1.68)	88.76(±2.24)	84.83(±1.68)	100(±0.00)	93.82(±0.56)	94.99(±1.97)	93.89(±6.03)	100(±0.00)
OA	97.57(±0.68)	95.58(±0.96)	91.61(±2.70)	96.35(±0.29)	94.53(±1.72)	95.89(±0.02)	95.59(±1.02)	99.59(±0.15)
AA	97.68(±0.56)	95.32(±0.81)	91.27(±2.46)	96.80(±0.22)	94.01(±1.85)	95.04(±0.08)	95.33(±0.96)	99.60(±0.15)
Kappa	97.36(±0.74)	95.21(±1.04)	90.91(±2.93)	96.05(±0.31)	94.07(±1.86)	94.58(±0.69)	95.65(±0.06)	99.56(±0.17)

Figures 6–9 show the classification maps generated by each model on the Xuzhou, Salinas, GulfPort, and Botswana datasets. Among these models proposed models show the best classification results with less noise in the classification map. It can be seen proposed models achieved the best classification results with overall accuracy (OA), average accuracy (AA), and kappa (K) reaching $99.41\%,$ $99.33\%$ and $99.25\%$. However, Hyperspectral Image Transformer Model (HiT) is not capable of obtaining long-range dependencies features due to the use of 3D Convolutional layers. The spectral-spatial self-attention Transformer network obtains better results due to the use of an attention mechanism that is not computationally costly to extract high-level features with small training samples. The proposed model with the Xuzhou dataset improves $2.97\%,$ $13.9\%,$ $6.64\%,$ $4.91\%,$ $8.08\%$, $6.39\%,$ $3.77\%$ overall (OA) compared to SSSAN, HiT, SSTN, CT Mixer, HybridFormer, transUnet and transfuse models. The classification accuracy of classes varies due to an imbalance of data and different class similarities with ground objects. CT Mixer performs well and captures high-level feature with the use of a convolutional neural network with a transformer network CT Mixer consists of a Local-Global multihead attention mechanism to extract more discriminate features. OA performance on the Xuzhou dataset is very well.

Figure 6. Classification maps on Xuzhou dataset (a) SSSAN, (b) HiT, (c) SSTN, (d) CT Mixer, (e) HybridFormer, (f) TransUnet, (g) Transfuse, (h) Proposed.

Figure 7. Classification map on Salinas dataset. (a) SSSAN, (b) HiT, (c) SSTN, (d) CT Mixer, (e) HybridFormer, (f) TransUnet, (g) Transfuse, (h) Proposed.

Figure 8. Classification maps on GulfPort dataset. (a) SSSAN, (b) HiT, (c) SSTN, (d) CT Mixer, (e) HybridFormer, (f) TransUnet, (g) Transfuse, (h) Proposed.

Figure 9. Classification maps on Botswana dataset. (a) SSSAN, (b) HiT, (c) SSTN, (d) CT Mixer, (e) HybridFormer, (f) TransUnet, (g) Transfuse, (h) Proposed.

The proposed model achieves the highest accuracy in all classes with small training samples because of the compact attention module and better selection of kernel size. Different models using 200 training samples generate the classification map on the Xuzhou dataset as shown in Figure 6 (a)–(h). The classification maps generated by the proposed model exhibit lower level of noise in comparison to other models. The quality of SSSAN model and proposed model almost looks better compared to other models.

The classification result on the Salinas dataset is shown in Table 8. The proposed model achieves the best result with no misclassification. Both the CT Mixer and the proposed model nearly achieve good results in terms of OA. In some classes, other models also perform well such as class two, and class nine. The proposed model improves $4.15\%,$ $6.58\%,$ $18.67\%,$ $3.12\%,$ $7.07\%$, $7.33\%,$ $6.5\%$ on average accuracy (AA) compared to SSSAN, HiT, SSTN, CT Mixer, HybridFormer, transUnet and transfuse. In addition, in some classes, SSSAN, HiT, and CT Mixer achieved the best classification results. Classification maps are also shown in Figure 7 noise in the CT Mixer model and the proposed model is nearly acceptable.

The classification result of the Gulfport dataset is shown in Table 9. The proposed model improves $0.69\%,$ $12.86\%,$ $3.83\%,$ $2.55\%,$ $11.78\%$, $4.78\%,$ $3.68\%$ on the Kappa coefficient compared to SSSAN, HiT, SSTN, CT Mixer, HybridFormer, transUnet and transfuse models. In some classes such as classes one, two, and five achieved the best results from SSSAN and SSTN models. The average accuracy of SSSAN is better than the proposed model. The classification map is shown in Figure 8 the Ground measurement shows that the proposed model has less noise.

The classification results on the Botswana dataset are shown in Table 10. As we can see, the proposed model achieved more satisfactory results than other state-of-the-art models. The SSSAN model and the proposed model almost achieved the best classification results.

The proposed model improve $2.02\%,$ $4.01\%,$ $7.98\%,$ $3.24\%,$ $5.06\%$ overall accuracy compared to other state-of-the-art models. Classification maps obtained from all models are shown in Figure 9. The proposed model classification map is nearly to the ground measurement image, proposed model has less noise effect on ground objects.

Visualization

To show the representative ability of the proposed model, we visualized the extracted features and feature maps in 2-D space via the t-SNE (van der Maaten and Hinton 2008) as shown in Figure 10, samples from the same class are clustered into one group. In contrast, samples from different classes become easily separable from each other’s. The proposed model can learn the abstract representation of spectral-spatial features well.

Figure 10. Visualization of 2D spectral spatial features on different dataset via t-SNE. (a) Xuzhou, (b) Salinas, (c) GP, and (d) Botswana dataset.

Ablation Experiments

The proposed model ablation experiments are carried out on different modules to improve the effectiveness further. Two modules are used in the proposed model: convolutional neural network (CNN) and transformer network with neighborhood attention. Table 11 shows the ablation classification results regarding OA, AA, and Kappa coefficients. With the addition of the Neighborhood attention mechanism, accuracy is improved compared to the CNN model. Only the convolutional neural network is not able to extract long-range dependency information with the addition of attention mechanism accuracy $4.55\%,$ $4.02\%,$ and $4.54\%$ improvement. The parameters of the Proposed attention technique a very few.

Table 11. Ablation experiments in terms of OA, AA, and Kappa with different modules.

Models	OA	AA	Kappa
Without Chan Shift	97.55(±0.61)	97.74(±0.59)	97.34(±0.65)
CNN	95.04(±0.66)	95.58(±0.84)	95.02(±0.71)
Transformer	91.43(±0.23)	91.46(±0.03)	90.71(±0.25)
CNN + Transformer	99.59(±0.15)	99.60(±0.15)	99.56(±0.17)

Time Cost Comparison

A comparison of training and testing for HybridFormer, HiT, SSSAN, CT Mixer, TransUnet, transfuse and proposed model is shown in Table 12. It can be seen that the proposed model relatively faster than the other models except for SSSAN, CT Mixer, TransUnet and transfuse. Therefore, proposed model can effectively reduce the computation time and improve classification accuracy. The proposed model take relatively long time due to the use of 3D convolutional layer to extract spectral spatial features.

Table 12. Training and testing time of different methods on Botswana dataset.

Model	Train time (s)	Test time (s)
HybridFormer	167(s)	4.18(s)
HiT	357(s)	7.11(s)
SSSAN	46.37(s)	1.01(s)
CT Mixer	56.73(s)	1.17(s)
TransUnet	256.35(s)	5.9(s)
Transfuse	312.09(s)	6.1(s)
Proposed	102.20(s)	1.22(s)

Impact of Training Samples

Further, evaluate the performance of different models with varying training samples, i.e., 200, 300, 500, and 1000 for the Botswana dataset. Figure 11 shows the OA values of different models with varying training sample sizes. Overall, the proposed model consistently performs well another one on all training samples. The poor performance was given by SSTN and HybridFormer models. When the larger training samples provided the proposed model, SSSAN, CT Mixer, and HiT performed closely. However, when a small training sample is used, the robustness (Zhang and Lin 2010) of the proposed model can be observed by comparing the more significant differences among other models. This is mainly because we use the neighborhood attention mechanism to extract more discriminate features.

Figure 11. OA achieved by different methods with varying training samples.

Conclusion

In this paper, we propose a neighborhood attention transformer with a channel strategy for hyperspectral image classification to solve the problem of limited training samples. Initially, the model uses a 3D and 2D convolutional neural network to extract local spectral spatial features. Then, extracting more abstract features using the neighborhood attention mechanism is proposed. The conducted experiments on four HSI datasets validate the proposed model’s effectiveness and exhibit its superiority over state-of-the-art models. The suggested model demonstrates superior adaptability to small sample image classification compared to other methods, resulting in more consistent and accurate classification outcomes. Furthermore, ablation studies were conducted on different modules demonstrates its significant impact in enhancing the model’s performance. This research thoroughly examines the spatial and spatial-spectral information, but neglects to account for the impact of spectral dimension information. The subsequent phase will comprehensively incorporate the multi-dimensional data from hyperspectral images into the learning and classification process. Additionally, it will integrate an adaptive feature extraction module to enhance the optimization of information across various dimensions. Furthermore, there is a growing preference for lightweight network models in the future. Therefore, the focus should be on developing highly adaptable and widespread lightweight models.

Acknowledgments

The author would like to thank potential supervisors, editors, and reviewers for their advice and comments.

Disclosure Statement

The author reported no conflict of interest.

Data Availability Statement

The data use for this research available on http://www.ehu.eus/ccwintco/index.php?title = Hyperspectral_Remote_Sensing_Scenes

Additional information

Funding

The work was supported by National Natural science foundation of china under Grant 62271171.