A hybrid convolution transformer for hyperspectral image classification

ABSTRACT

Hyperspectral images play a crucial role in remote sensing applications surveillance, environment and precision agriculture, containing abundant object information. However, they often face challenges such as limited labelled data and imbalanced classes. In recent years, convolutional neural networks (CNNs) have shown impressive performance in computer vision tasks, including hyperspectral image classification. The emergence of transformers has also attracted attention for hyperspectral image analysis due to their promising capabilities. Nevertheless, transformers typically demand a substantial amount of training data, making their application challenging in scenarios with limited labelled samples. To overcome this limitation, we propose a hybrid convolution transformer framework. Our method uses a vision transformer and a residual 3D convolutional neural network model. It also uses a sequence aggregation layer to avoid overfitting issues that come up when there isn’t enough training data. Our proposed residual channel attention module captures richer spatial-spectral complementary information and maintains spectral details during the feature extraction process. We conducted experiments on three benchmark datasets. The proposed model achieved state of the art performance$99.75\mathrm{\%}$, $99.46\mathrm{\%}$and $99.95\mathrm{\%}$ in terms of overall accuracy (OA) using only $5\mathrm{\%}$,$5\mathrm{\%}$and $10\mathrm{\%}$labelled training samples respectively. Which is better than other state of the art methods.

KEYWORDS:

• Convolutional neural network
• hyperspectral image classification
• vision transformer
• remote sensing data

Introduction

Hyperspectral image data is acquired by hyperspectral sensors. Hyperspectral has abundant spatial and spectral information. The hyperspectral image is widely used in mineralogy (J. Wang et al., 2012), environment (Bioucas-Dias et al., 2013), surveillance (Uzkent et al., 2017), military (Ardouin et al., 2007), agriculture monitoring (De Petris et al., 2023; Pirotti et al., 2014) and other domains (Liao et al., 2023; Schimleck et al., 2022; Vaishnavi et al., 2022). Hyperspectral data contain a lot of spectral information about surface objects. As remote sensing technology advances, the spatial resolution of hyperspectral imaging data has significantly improved, which has widely enhanced the ability of the hyperspectral image dataset to express different objects. By investigating the spectral information, it was found that data redundancy exists in bands. To reduce the data redundancy, some methods were used, such as linear discriminative analysis (Bandos et al., 2009), independent analysis (Villa et al., 2011), principle component analysis (Licciardi et al., 2011), and some other methods. In the early stages, machine-learning techniques were used for hyperspectral image classification, such as support vector machines (Melgani & Bruzzone, 2004), random forests (J. M. Haut et al., 2017), logistic regression (Ham et al., 2005), kernel-based methods (J. Haut et al., 2017), and k-means clustering (Camps-Valls & Bruzzone, 2005). However, these traditional techniques can classify hyperspectral images effectively when relying solely on spectral information; the feature learning process does not fully leverage the spatial pixel correlation present in the data. Thus, these methods lead to misclassification and cannot give satisfactory classification results. Ma & Chang, (2021) proposed methods that utilise active learning and iterative training sampling to enhance spectral spatial classification.

With the development of deep learning (LeCun et al., 2015), approaches in the field of agriculture (Ganatra & Patel, 2021; Santos et al., 2020) have made progress in the last few years. Deep learning models have the ability to extract relevant features as compared to manual engineering models. Due to its local connectivity and weight-sharing ability, the convolutional neural network (CNN) has strong feature extraction abilities. For the moment, SAE (Özdemir et al., 2014) and DBN (Yao et al., 2016) were introduced for the classification of satellite images and land-use classification tasks. These methods are based on one-dimensional (1-D) sample training data. However, alteration of 1-D training data leads to loss of spatial information. As a solution, researchers have explored various network structures based on convolutional neural networks for hyperspectral image classification. An improved classification method was proposed by (W. Hu et al., 2015). This method uses a 1-D convolutional neural network with five convolution layers. Also, this method takes spectral information as input and extracts spectral features (Zhao & Du, 2016). A 2D-convolutional neural network model was introduced to extract spatial features from the first few principle component analyses after dimensionality reduction. Yang et al., (2017) proposed a dual branch spectral spatial network using a 1-D and 2-D convolutional neural network to extract spatial spectral features. From this method, joint spectral spatial features are extracted. For effective spectral spatial feature extraction Chen et al., (2016) introduce the 3D convolutional neural network method. Roy et al., (2019) proposed a hybrid network with a 3D and 2D convolutional neural network model to effectively extract spectral spatial features. This method could reduce the computational complexity. Recently, the residual model (Zhong et al., 2017) was adopted for hyperspectral image classification to increase the number of layers and extract more discriminative features. In W. Wang et al., (2018), an end-to-end dense convolutional model was proposed to extract spectral spatial features. Apart from the convolutional neural network, other network categories have shown promising performance in the hyperspectral image classification. It leverages convolution and pooling layers to effectively learn deep features from the data. In addition, feedforward and recurrent neural networks (Hang et al., 2019) are used for hyperspectral image classification (Wu & Prasad, 2017). proposed the model combination of a convolutional neural network and a recurrent neural network (RNN). The first convolutional neural network extracts former features, and then the recurrent neural network is used to extract further contextual information. Meanwhile, alternative network architectures have been explored for hyperspectral image classification, including those based on generative adversarial networks (Zhu et al., 2018), capsule networks (Paoletti et al., 2018), and graph networks (Hong et al., 2020). These underlying networks effectively capture the interdependencies among pixels and demonstrate a high level of robustness. To some extent, the convolutional neural network-based methods mentioned above have limited ability to fully capture the correlation information between the spectral and spatial domains in hyperspectral image data.

Recently, vision transformers (ViT) (Dosovitskiy et al., 2020) have gained great attention in the field of computer vision due to their long-range dependencies. The attention module is used to describe the global context of the image through positional encoding. Researchers have conducted work in the field of hyperspectral image data (He et al., 2021). The spectral spatial transformer model was proposed, and the VGGNet structure was used to extract the spectral spatial feature. Qing et al., (2021) use a spectral attention module combined with a multihead transformer module to effectively capture the continuous spectral relationship (Hong et al., 2021). A new model was developed called spectralFormer, which is used to learn spectral information from group-wise neighbouring bands and, at last, uses a cross-layer transformer encoder. Sun et al., (2022) proposed the spectral spatial feature tokenization transformer (SSFTT) model to capture semantic features with the tokenization process. Zhang et al., (2022) proposed a convolution transformer mixer to extract local global information through a combination of a convolutional neural network and a transformer. While the aforementioned methods can capture global information to some extent, they lack the ability to capture local information effectively.

The process of labelling hyperspectral image data takes much labour and is time-consuming, leading to a scarcity of labelled samples available for classification tasks. Consequently, the challenge lies in obtaining a better classification result while using as few labelled samples as possible. This topic has been a focal point of research, aiming to optimise classification performance under limited data scenarios. Small sample hyperspectral classification can be approached through various strategies, including generative adversarial networks (GANs) (Zhong et al., 2019), semi-supervised learning, and network optimization methods (Feng et al., 2019). The backbone of network optimisation is residual connection, aiming to enhance feature fusion and reuse capabilities within the model. Indeed, network optimization holds significant research significance, especially in the context of “small sample data”. Building a network with a more reasonable structure can prove to be an effective solution to address the challenges of limited labelled data.

To address the limitations of previous methods, we propose a hybrid convolutional transformer that uses a convolutional neural network and a vision transformer to extract joint spectral spatial features with both local and global information at various scales from small training samples. Attention mechanisms are added to both the convolutional neural network and transformer model. By employing this model, the hyperspectral image classification task can effectively leverage both local and global information, leading to improved network representation. The main contributions are as follows:

• An effective approach is proposed for HSI classification, aiming to leverage both local and global information through utilization of a powerful residual 3D convolutional neural network model.

• To mitigate the loss of spectral features, a residual channel attention mechanism is proposed after the depth wise convolution layer. This mechanism effectively extracts spectral spatial joint features, ensuring the preservation of valuable information.

• Introducing a sequence aggregation layer helps mitigate overfitting caused by limited training samples. This layer helps to alleviate overfitting issues by aggregating information from the sequence and reducing model complexity.

The remaining of the article as follows: section II describes the proposed Methodology section III describes dataset and experimental evaluation, section IV result and discussion, section V conclusion.

Proposed methodology

Figure 1. Shows the proposed model for HSI classification composed of three parts the residual 3D Convolutional Neural Network for local feature extraction, residual channel attention module and Vision Transformer module. Let $X_{\mathit{hsi}}\in {\mathbb{R}}^{\mathit{H}\times \mathit{W}\times \mathit{C}}$represent original HSI data where H, W and C represent height width and channel respectively. Suppose that$X_{\mathit{hsi}}$contain M labelled pixel $\mathit{X}=\{x_1,x_2\dots ,x_M\}\in {\mathbb{R}}^{1\times 1\times \mathit{C}}$and their corresponding one hot label $\mathit{Y}=\{y_1,y_2,\dots ,y_M\}\in {\mathbb{R}}^{1\times 1\times \mathit{K}}$where K represents number of classes. The spatial size of the centre pixel represent $\mathit{S}\times \mathit{S}$can be defined a spectral spatial vector. After the data notation, all hyperspectral image (HSI) data was randomly divided into training and testing. The training data is used to optimize the proposed model and obtain the best-trained model tested on the test dataset. After the completion of training and testing, the evaluation of the model is performed using three evaluation matrices called overall accuracy, average accuracy, and Kappa coefficient. Finally, the model classifies the pixels to generate a classification map. First, we apply the Principle Component Analysis (PCA) algorithm to reduce the noise and redundancy in input data while preserving the spatial dimension. Second, we proposed a residual 3D convolutional neural network model to jointly learn spectral spatial features. Third, we proposed a residual channel attention module to preserve important spectral information in bands. Finally, the vision transformer with the sequence-aggregation layer is used to further extract more representative and discriminative semantic features and perform classification.

Figure 1. Illustration of proposed residual 3D-CNN vision Transformer.

Residual 3D-CNN module

Let the hyperspectral data cube is $X_{\mathit{hsi}}\in {\mathbb{R}}^{\mathit{H}\times \mathit{W}\times \mathit{C}}$where H represents the height, W represents the width and C represents the spectral bands. The input data first pre-processed by Principle Component Algorithm in spectral dimension, which reduced the redundancy of input data. The spectral bands after Principle Component Analysis (PCA) reduced from C to B, B has a great change on the computational complexity. Using the both spectral spatial information, a small region size of $\mathit{L}\times \mathit{L}$cantered on the pixel location (i, j). In 3D convolution model, the convolution operation involves applying a 3D convolutional kernel. This operation act on not only spatial dimension but also capture the correlation information between multiple spectral bands. The input dimension is $\mathit{L}\times \mathit{L}\times \mathit{B}$and the N 3D convolution kernels, the kernel size is $\mathit{M}\times \mathit{N}\times \mathit{D}$used for convolution operation. The output feature maps is 4D tensor and dimension of this is $(\mathit{L}-\mathit{M}+1)\times (\mathit{L}-\mathit{M}+1)\times (\mathit{B}-\mathit{D}+1)\times \mathit{D}$ the number of bands across spectral dimension is fix and the final result is 4D tensor size of $(\mathit{L}-\mathit{M}+1)\times (\mathit{L}-\mathit{M}+1)\times (\mathit{B}-\mathit{D}+1)\times \mathit{N}$ 3D kernel better utilized the spectral features. Hence, the 3D convolutional Neural Network approach are better suited for hyperspectral image classification that involve abundant spectral information. During the computation of 3D convolution, the activation value at position $(\mathit{a},\mathit{b},\mathit{c})$ on the jth feature map in the ith layer is obtained. The mathematical formula of 3D convolution given in equation:

(1) $x_{ij}^{abc}=\mathit{\theta }\left(b_{\mathit{ij}}+\sum _l\sum _{\mathit{r}=1}^d\sum _{\mathit{p}=1}^h\sum _{\mathit{q}=1}^w{w}_{ijm}^{pqr}\times x_{i-1,l}^{a+b,y+p,c+r}\right)$(1)

Where $x_{ij}^{abc}$ means that the output variable at position$(\mathit{a},\mathit{b},\mathit{c})$. $\mathit{\theta }$ is activation function $w_{\mathit{ij}}$ is depth of the kernel, is a bias. In summary, the proposed residual 3D-CNN model uses three-dimension convolutional layer with different kernel sizes to extract spectral spatial discriminative features. $3\mathit{D}\_\mathit{CONV}1=8\times 3\times 3\times 7$ Where the kernel size is $3\times 3\times 7$ and the filter size is 8. $3\mathit{D}\_\mathit{CONV}2=16\times 3\times 3\times 5$ the kernel size is $3\times 3\times 5$ and output filter size is 16. $3\mathit{D}\_\mathit{CONV}3=32\times 3\times 3\times 3$ where kernel size is $3\times 3\times 3$and output filter size is 32. The model weights are initialized randomly and then optimized using the Adam optimizer along with back propagation and softmax loss function. During the training mini batch, size is set to 64 and training perform for 100 epochs without applying batchnormlization and augmentation techniques.

We compare the proposed 3D-CNN with the conventional 3D-CNN structure. In the traditional approach, the spectral information is extracted from network using 3D convolutional kernel operation. The extracted features are then directly forward to classification network. When the conventional network becomes too deep, it often suffers from the gradient degradation problem, leading to insufficient extraction of deep spectral features. However, the residual 3D-CNN which effectively addresses the issue of gradient degradation as shown in Figure 2. Additionally, this approach enables the network to be further deepened, ensuring the maximum extraction of semantic information by the model.

Figure 2. Proposed 3D residual CNN Structure.

Attention module

The attention mechanism, which is extensively employed in deep learning tasks such as image recognition, image segmentation and Natural Language processing (NLP), plays a vital role in selecting pertinent information from a large volume of data. It focuses on highlighting the most significant information for the given task.

The channel attention module is specifically designed to extract spatial spectral information by integrating both depth wise convolution and channel attention mechanism. The channel attention component allows the network to adaptively prioritize and emphasize certain channels based on their relevance to the given task. This component for the limitation of depth wise convolution, which applies a single filter to each input channel and ignores channel specific information. This module implemented in this approach leverages the residual structure to establish an identity mapping between original features and the features obtained after convolution performance by preserving important information from the original features throughout the processing stages.

As shown in Figure 3. The efficient channel attention module that builds upon the squeeze and excitation network (J. Hu et al., 2018). The main approach employed by ECA-NET involve leveraging local-cross channel interaction without dimensionality reduction. This technique effectively reduces the number of network parameter while still allowing for the management of output feature channels with varying weights. As a result, ECA-NET successfully extract crucial features from image while minimizing model complexity. For ECA-NET input dimension $\mathit{X}\in {\mathbb{R}}^{\mathit{H}\times \mathit{W}\times \mathit{C}}$with their dimension. First, apply Global average pooling layer operation to reduce the parameter count and incorporate the spatial information of the input. A reshape operation is applied to obtain matrix $1\times 1\times \mathit{C}$after that second matrix obtained by one dimension convolution operation $1\times 1\times \mathit{C}$and the last matrix obtained by sigmoid function. Finally, the output feature map is multiply with the input.

Figure 3. Illustration of ECA model.

Vision transformer

Transformer is an innovative deep learning model that leverages the self-attention mechanism and feed-forward neural network. Unlike the convolutional layers typically employed in hyperspectral image classification, the transformer layer calculates feature representation based solely on the self-attention mechanism. This enables the model to capture rich and robust feature representation. A trainable deep learning model can construct by stacking multiple transformer layers. This deep model consists of several parts: image tokenization, positional embedding, classification token, transformer encoder, and classification head.

Image tokenization

In typical transformer model, a sequence of vector called as token is provided as input. When applying transformer to image determining the order of tokens is not straightforward. The vision transformer addresses this challenge by dividing the image into non-overlapping square patches, following a raster scan order. The sequence of patches,$x_p{\in }^{H\times (P^2\mathit{C})}$ with patch size P are flattened into one dimension vector. The simple patching and embedding have some limitations like loss of information.

Positional embedding

Positional embedding is a technique used to incorporate spatial information into the sequence of tokens in a transformer model. Since the model lacks inherent knowledge about the spatial relationship between tokens, providing additional information to represent this relationship can be beneficial. This is commonly achieved by either using a learned embedding or assigning weights to tokens based on two sine waves with high frequencies. By incorporating positional embeddings, the model can learn and recognise the positional relationships that exist between these tokens.

Transformer encoder

Multiple stacked encoding layers compose the transformer encoder. Each encoder layer consists of two sub-layers of multi-head self-attention and a multiple-layer perceptron head. Prior to each sublayer, layer normalization is applied, which helps to normalize the input. After each sublayer, a residual connection is added, allowing the output to flow to the next sublayer. This residual connection preserves the information from the previous sublayer and aids in the flow of gradients during training.

Classification head

In vision transformers, an additional learnable token, often referred to as the class token, is included in the sequence of embedded patches. This token represents the class parameter of the entire image. After passing through the transformer encoder, the state-of-class token can be utilised for classification purposes. The class token plays a crucial role in accumulating information about the sequence through self-attention. It learns to capture important features and contextual information relevant to the classification task.

Sequence aggregation

In the proposed work sequence, aggregation (Hassani et al., 2021) was chosen instead of average and global pooling to flatten the data. In some scenarios, it is preferable to choose these techniques. This sequence aggregation is an attention-based technique to pool the sequence data. To preserve the output, sequence information is based on the understanding that it contains relevant information from various parts of the input data. By retaining this information, we can potentially enhance the performance of the model without adding any extra parameters. This operation involves applying a mapping to the output sequence. The output of the sequence aggregation transformer encoder is batch size, sequence length, and total embedding dimension. The output of the sequence aggregation transformer encoder is a linear layer that applies softmax operations. This generates a weight for each input token. After flattening the output, it was sent through the classification head. The sequence pool enables the network to assign weights to the sequential embedding generated by the transformer encoder, allowing for the correlation between input data.

Dataset description and parameter analysis

Study area

The Xuzhou dataset was acquired using an airborne HYSPEX hyperspectral camera, covering the Xuzhou peri-urban site in November 2014. It consists of a spatial size of $500\times 260$ pixel with an impressive highly spatial resolution of 0.73 m/pixel. For the experiments, 436 spectral bands were utilized, carefully selected after removing the noisy bands ranging from 415 nm to 2508 nm. The scene represent a peri-urban area and its characteristics by nine distinct categories, including crops, vegetation, man-made structured and coal fields. The presence of such diverse and complex mixed categories makes this dataset particularly challenging for classification tasks. Given the very high spatial resolution and the diverse nature of the categories, this dataset poses a significant challenge for hyperspectral image classification and analysis. The mineral classification is conducted on Xuzhou dataset. The total number of training and testing samples can be found in Table 1.

Table 1. Xuzhou dataset labelled samples.

The second Dataset is Salinas dataset was acquired by Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) sensor over the Salinas Valley in California, USA. This Dataset encompasses a spatial dimension of $512\times 217$pixel with a spatial resolution of 3.7 m/pixel. The spectral range spans from 400 to 2500 nm, consisting of 204 spectral bands after removal of 20 noisy bands. Within the dataset, there are 16 distinct classes manually labelled, including various land covers and agricultural crops. Alongside the unlabelled pixels, the dataset is enriched with valuable information that can aid in Hyperspectral image classification. The number of training and testing samples used in this experiments can be found in Table 2. due to its detailed spectral information and diverse land categories, the Salinas dataset presents a fascinating and demanding scenario for hyperspectral image analysis and classification algorithms.

Table 2. Salinas dataset labelled sample.

The dataset used in this study was acquired by the National Aeronautics and Space Administrator (NASA) Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) instruments over Kennedy space centre (KSC), located in Florida, USA, on 23 March 1996. The dataset comprises 224 spectral bands and has a spatial resolution of$512\times 614$pixels, with a corresponding spatial resolution of 1.8 m/pixels. The spectral range spans from 400 to 2500 nm with a spectral resolution of 10 nm. The classification task is very challenging due to the presence of 13 distinct land cover types, including water and mixed classes. To optimize the data for classification water absorption and low signal to noise ratio (SNR) bands has been removed, leaving 176 relevant bands for classification. In Table 3 detailed information regarding the number of Training and Testing samples for each class can be found. The dataset diverse spectral and spatial characteristics, as well as the abundance of labelled pixels, present an excellent opportunity for evaluating and refining hyperspectral image classification models.

Table 3. KSC dataset with Labelled Samples.

Experimental configuration and setup

We conduct experiment on NVIDIA 3060 GPU with 64 GB RAM. The code implementation environment is PYTHON 3.8 with TensorFlow library. We evaluate the three performance matrices OA, AA and Kappa coefficient. For the training and testing of model, we select $10\mathrm{\%}$ for KSC Dataset for training and$90\mathrm{\%}$for testing, for Xuzhou and Salinas dataset $5\mathrm{\%}$ for training and $95\mathrm{\%}$ for testing randomly selected samples. The reason to choose small training sample is to solve the problem of small training sample for hyperspectral image data. We choose batch size, Adam optimizer with learning rate and epoch set 64, 0.001 and 100 for all experiment.

Selection of PCA component

Extracting spectral information from Hyperspectral image data is a challenging task due to the presence of redundant and noisy data in the spectral channels. To address this issue comparative experiment were conducted using the Principle Component Analysis (PCA) on three datasets. The number of principle components was selected: 10, 20, 30, 40, and 50. The experimental results shown in Table 4 show that the 30-principle component gives optimal accuracy for three datasets.

Table 4. OA (%) on different PCAs component.

Dataset	10	20	30	40	50
KSC	99.82	99.89	99.95	99.82	99.76
XUZHOU	99.59	99.56	99.46	99.45	99.44
SA	99.45	99.72	99.75	99.74	99.75

Influence of patch size

In the experiment, different spatial patch sizes are selected around the centre pixel, which can also affect classification accuracy. This experiment with different spatial sizes was carried out, as shown in Table 5.

Table 5. OA (%) result on different patch size.

Dataset	$9\times 9$	$11\times 11$	$13\times 13$	$15\times 15$
KSC	99.85	99.70	99.95	99.87
XUZHOU	99.35	99.33	99.46	99.37
SA	99.62	99.78	99.75	99.80

As we can see in table for the KSC dataset when spatial size increase the accuracy stop improving and for Pavia university same scenario. For Salinas dataset when patch size increase accuracy increase and decrease but for optimal result we set $13\times 13$ patch size.

Result & discussions

To compare the classification result of proposed model comparative experiments are conducted on three dataset KSC, Salinas and Xuzhou. The comparative methods 2D convolutional neural network (2DCNN) (Zhao & Du, 2016), 3D convolutional neural network (3DCNN) (Hamida et al., 2018), spectral spatial residual network (SSRN) (Zhong et al., 2017), rethinking spatial dimension vision transformer (RVT) (Heo et al., 2021), vision transformer (VIT) (Dosovitskiy et al., 2020), convolution transformer mixer (CT Mixer) (Zhang et al., 2022), and spectral spatial feature tokenization transformer (SSFTT) (Sun et al., 2022) are chosen. Classification result compared with other state of the art methods are shown in Tables 6–8. All the experimental settings we discussed in Experimental configuration part.

Table 6. Classification result (%) on Salinas Dataset.

Class	2DCNN	3DCNN	SSRN	RVT	VIT	CT Mixer	SSFTT	Proposed
1	100	100	100	100	100	100	100	100
2	99.97	100	100	100	100	100	100	100
3	100	100	100	100	99.94	100	100	100
4	99.54	99.92	100	99.62	99.84	100	92.44	99.77
5	96.26	97.24	98.27	99.37	99.79	93.12	100	99.84
6	99.97	99.62	100	100	100	100	99.09	100
7	99.97	99.94	100	100	100	100	100	100
8	97.98	98.58	97.31	97.83	98.86	93.13	100	99.98
9	100	100	100	99.93	100	100	100	100
10	99.38	99.19	100	99.74	99.72	99.93	99.74	99.90
11	100	100	99.80	99.90	99.79	100	99.80	100
12	100	99.01	100	100	100	100	99.18	99.94
13	100	100	100	100	100	100	99.08	99.88
14	98.81	98.81	92.22	99.70	99.68	97.73	99.90	97.63
15	95.75	95.82	99.97	92.71	97.85	100	99.89	98.84
16	99.12	99.06	99.24	100	100	100	99.41	99.65
OA	98.71	98.83	99.16	98.49	99.43	98.18	99.64	99.75
AA	99.17	99.20	99.17	99.30	99.71	98.99	99.28	99.71
Kappa	98.57	98.70	99.07	98.32	99.36	97.97	99.59	99.65

Classification result

The performance indexes are employed to assess the accuracy of the model in this study. These indexes include overall accuracy (OA), average accuracy (AA), and Kappa coefficient (Kappa). The overall accuracy (OA) gauges the proportion of correctly classified samples achieved by the model. It provides an overall assessment of the model’s effectiveness in distinguishing between different classes. The average accuracy (AA) calculates the average precision across all land objects. The kappa coefficient (Kappa) is a valuable accuracy metric derived from the confusion matrix. It quantifies the percentage of error reduced by the model compared to a completely random classification.

By considering these three metrics, the study ensures a thorough evaluation of the model’s performance and ability to accurately distinguish various lands objects. To ensure the effectiveness and reliability of the model we conduct 5 consecutive experiments. The average performance indices along their standard deviation for each model on the three datasets are presented in Tables 6–8. In Figures 4–6 we show the classification map generated by each model on Salinas, Xuzhou and KSC datasets. Among these models proposed model demonstrated superior performance yielding more detailed and accurate classification result.

Table 7. Classification result (%) on xuzhou dataset.

Class	2DCNN	3DCNN	SSRN	RVT	VIT	CT Mixer	SSFTT	Proposed
1	99.15	98.70	99.70	97.33	97.92	99.72	99.23	99.32
2	99.32	98.95	99.86	98.98	98.90	99.68	99.24	99.97
3	98.86	97.73	98.94	96.63	96.97	95.31	99.01	99.43
4	99.71	99.77	98.02	96.30	97.69	100	96.60	99.81
5	99.05	98.45	95.68	99.08	99.39	99.36	99.64	99.64
6	99.65	99.52	95.89	97.79	97.45	98.53	98.57	99.61
7	97.77	99.80	98.49	99.39	99.48	98.90	99.96	99.48
8	100	99.58	99.97	98.72	99.47	99.07	99.93	99.86
9	99.69	99.62	99.58	97.42	97.42	92.14	98.28	97.80
OA	99.13	98.95	98.54	97.98	98.43	98.98	99.16	99.46
AA	99.24	99.12	98.46	97.96	98.30	98.08	98.94	99.44
Kappa	98.90	98.67	98.14	97.98	98.01	98.71	98.94	99.31

Figure 4. Classification maps for Salinas Dataset (a) 2DCNN (b) 3DCNN (c) SSRN (d) RVT (e) ViT (f) CT mixer (g) SSFTT (h) Proposed.

It can be seen that proposed model achieved best, with overall accuracy (OA) reaching $99.75\mathrm{\%}$,$99.46\mathrm{\%}$ and $99.95\mathrm{\%}$ on three dataset. It can be seen from Salinas Dataset 2D CNN not suitable for small training samples. For Xuzhou dataset, spectral-spatial residual network (SSRN) and CT Mixer not obtain satisfactory results. For the KSC dataset 2D

and 3D model is Not provide a better result on small training samples with the use of 2D and 3D kernel size. In order to classification result, proposed model under the condition with small training samples using residual connection is more suitable for classification results. In order to evaluate the classification performance, Figures 4–6 show the classification maps obtained from the entire compared and proposed model on three-hyperspectral dataset using 10% and 5% training samples. The presence of noise in the classification maps is inversely related to the classification accuracy and this relationship is noteworthy. The proposed method exhibits a lower number of noise point compared to other methods, indicating its effectiveness in enhancing the classification performance Of HSI. The performance of the proposed model surpasses that the state of the art methods.

Table 8. Classification result (%) on KSC dataset.

Class	2DCNN	3DCNN	SSRN	RVT	VIT	CT Mixer	SSFTT	Proposed
1	90.36	100	99.41	100	100	100	100	100
2	62.55	89.49	100	98.17	99.54	100	100	100
3	79.13	95.65	99.56	93.47	100	100	98.69	100
4	65.63	71.80	91.62	71.36	76.65	100	100	100
5	79.31	62.06	100	77.93	73.10	100	100	100
6	92.23	82.52	100	79.61	94.17	98.05	100	100
7	82.97	94.68	100	100	100	100	100	100
8	98.96	97.93	100	98.71	98.96	99.74	100	99.48
9	92.73	100	100	99.14	99.78	100	100	100
10	99.72	100	100	100	100	100	100	100
11	96.81	97.34	100	100	100	100	100	100
12	77.48	100	100	99.77	100	100	100	100
13	78.05	100	100	100	100	100	100	100
OA	85.67	95.50	99.48	96.41	97.65	99.89	99.93	99.95
AA	84.30	91.65	99.27	93.70	95.55	99.83	99.89	99.96
Kappa	84.18	94.98	99.43	96.00	97.38	99.88	99.92	99.95

Figure 5. Classification maps for KSC dataset (a) 2DCNN (b) 3DCNN (c) SSRN (d) RVT (e) ViT (f) CT mixer (g) SSFTT (h) proposed.

Figure 6. Classification maps for xuzhou dataset (a) 2DCNN (b) 3DCNN (c) SSRN (d) RVT (e) ViT (f) CT mixer (g) SSFTT (h) Proposed.

Discussions

This paper proposed a new model Hybrid Convolution Transformer for Hyperspectral image classification using 3D residual Convolutional neural network with Re- Attention module and vision transformer. That model can extract feature from small training samples and handle the long range dependencies successfully. One can see from Tables 7–8 that Convolutional neural network model and transformer model including 2DCNN, 3DCNN, SSRN, ViT, CT Mixer and SSFTT consider for comparison. All experiment results exhibit the proposed model can achieve best performance on all dataset. For hyperspectral image dataset with more redundant spectral bands and lower spatial resolution, spectral features demonstrate superiority over spatial features. On the other hand, for dataset with fewer redundant spectral bands and higher spatial resolution the scenario be opposite. The proposed model with Salinas Dataset improves$1.04\mathrm{\%}$, $0.92\mathrm{\%}$, $0.59\mathrm{\%}$, $0.14\mathrm{\%}$, $0.32\mathrm{\%}$, $2.08\mathrm{\%}$and $0.11\mathrm{\%}$ on OA compared to 2DCNN, 3DCNN, SSRN, RVT, VIT, CT Mixer and SSFTT. At that time the proposed method achieve $0.54\mathrm{\%}$, $0.51\mathrm{\%}$, $0.54\mathrm{\%}$, $0.15\mathrm{\%}$, $1.04\mathrm{\%}$, $0.43\mathrm{\%}$ on AA and $1.08\mathrm{\%}$ $0.95\mathrm{\%}$, $0.58\mathrm{\%}$, $0.09\mathrm{\%}$, $0.29\mathrm{\%}$, $2.24\mathrm{\%}$and $0.06\mathrm{\%}$on Kappa. The classification accuracy of different classes is different because of imbalance of training samples and different classes have similarity with ground object. From Table 7 Xuzhou Dataset with proposed method improved accuracy compare to stat of the art methods. Proposed method obtain$0.33\mathrm{\%}$, $0.51\mathrm{\%}$, $0.92\mathrm{\%}$,$1.48\mathrm{\%}$, $1.03\mathrm{\%}$, $0.48\mathrm{\%}$ and $0.3\mathrm{\%}$ OA accuracy. As well$0.2\mathrm{\%}$, $0.32\mathrm{\%}$, $0.98\mathrm{\%}$, $1.48\mathrm{\%}$, $1.14\mathrm{\%}$, $1.36\mathrm{\%}$ and $0.5\mathrm{\%}$ AA accuracy and $0.41\mathrm{\%}$, $0.64\mathrm{\%}$, $1.17\mathrm{\%}$, $1.33\mathrm{\%}$, $1.3\mathrm{\%}$, $0.6\mathrm{\%}$ and $0.37\mathrm{\%}$ improve on Kappa. Table 8 shows the KSC classification result. The proposed model improve accuracy compared to other methods. Proposed method obtained $14.28\mathrm{\%}$, $4.45\mathrm{\%}$, $0.47\mathrm{\%}$, $2.92\mathrm{\%}$, $2.3\mathrm{\%}$, $8.51\mathrm{\%}$ and $0.02\mathrm{\%}$ OA accuracy. At mean time $15.66\mathrm{\%}$,$8.31\mathrm{\%}$, $0.69\mathrm{\%}$, $4.19\mathrm{\%}$, $4.41\mathrm{\%}$, $14.55\mathrm{\%}$, $0.07\mathrm{\%}$ AA accuracy obtain and$15.77\mathrm{\%}$, $4.97\mathrm{\%}$, $3.26\mathrm{\%}$, $2.57\mathrm{\%}$, $9.49\mathrm{\%}$, $0.03\mathrm{\%}$Kappa accuracy obtain. All the methods obtain satisfactory result with limited training samples but there is inconsistency with class accuracy taking the example of Xuzhou dataset 3DCNN, SSRN and CT Mixer overall accuracy (OA) is not consistent. In terms of overall accuracy the proposed model exhibit the superior results. Despite the slightly decrease in accuracy for certain classes (<100%), the overall performance of the model remains consistently high. In conclusion, the evaluation of the proposed model on three dataset reveals some key findings: the proposed model achieves highest accuracy across all three datasets with accuracy for each class surpassing 90%. This demonstrate that the network optimization strategy not only enhance the accuracy of small samples classes but also maintains stability for other classes. Besides the proposed model performs well and the model utilizes 3D residual CNN for feature extraction and incorporate with attention module for spectral enhancement and Vision transformer for extracting high-level features these architectural components play a crucial role to extracting discriminative features for image classification.

Ablation Study

For the experiment four set of ablation study are conduct between Residual 3D-CNN, attention module and Vision transformer on KSC dataset. The Overall accuracy result show in Table 9. From the table one can see that overall accuracy is 97.80% when using 3D-CNN and attention module. However, for using 3D-CNN and vision transformer the overall accuracy increase$2.11\mathrm{\%}$. Vision transformer can Capture subtle spectral feature and produce competitive Result as compared to CNN based model.

Table 9. OA (%) of ablation Study.

Component			OA (%)
3D-CNN	Attention	ViT	//
√	√	×	97.80
√	×	√	99.91
×		√	99.77
√	√	√	99.95

However, when we apply attention module the overall accuracy (OA) increases simultaneously. Without attention module overall accuracy (OA) is $99.91\mathrm{\%}$ and with attention mechanism OA is $99.96\mathrm{\%}$.

Impact of training samples

Evaluating the performance of models under different sample size is a crucial aspect of assessing model quality. Deep learning model typically demand a substantial amount of training data to achieve advanced learning capabilities. Unfortunately, obtaining a large number of samples in real life scenarios can be challenging and often limited. This scarcity of training data poses a significant challenge for deep learning models and highlights the importance of developing techniques that can perform effectively with smaller sample sizes. Figure 7 shows the test results of various methods on KSC dataset at different training ratios.

Figure 7. OA of 2–10% of training sample on KSC dataset.

As expected, as a training ratio increases, the accuracy of different hyperspectral image classification methods also raises. However, regardless of whether the training ratio is high or low, traditional methods such as 2DCNN and 3DCNN methods consistently yield lower accuracies compared to more recently proposed methods such as SSRN.

When the training ratio is high (e.g. 10%), there is little difference in the experimental results among methods. However, RVT and SSRN exhibits a significant drop in performance when training ratio is decreases. On the other hand, when the training ratio is less than 5% the proposed model demonstrate superior classification performance, indicating that hybrid convolution transformer model can effectively extract discriminative information even with limited training samples. The observation indirectly suggest that proposed model is capable of handling small sample scenarios effectively.

Conclusion

In order to effectively leverage both spectral and spatial features from limited training samples, we have introduced a hybrid convolutional transformer network for HSI. Our approach combines the residual 3DCNN model with channel attention module to extract joint spectral spatial features. The integration of vision transformer further enhance the model performance. In order to address the issue of overfitting we have incorporate sequence aggregation layer. Experimental result demonstrate the superior performance of our proposed model compared to other state of the art methods. For the future, we aim to refine the model architecture and develop a lightweight network for improved efficiency.

Acknowledgments

Author would like to thanks potential supervisor, editor and reviewer for their advice and comment.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This research was funded by National Natural sciences foundation of china under Grant 62271171.