TrmGLU-Net: transformer-augmented global-local U-Net for hyperspectral image classification with limited training samples

ABSTRACT

In recent years, deep learning methods have been widely used for the classification of hyperspectral images. However, their limited availability under the condition of small samples remains a serious issue. Moreover, the current mainstream approaches based on convolutional neural networks do well in local feature extraction but are also restricted by its limited receptive field. Hence, these models are unable to capture long-distance dependencies both on spatial and spectral dimension. To address above issues, this paper proposes a global-local U-Net augmented by transformers (TrmGLU-Net). First, whole hyperspectral images are input to the model for end-to-end training to capture the contextual information. Then, a transformer-augmented U-Net is designed with alternating transformers and convolutional layers to perceive both global and local information. Finally, a superpixel-based label expansion method is proposed to expand the labels and improve the performance under the condition of small samples. Extensive experiments on four hyperspectral scenes demonstrate that TrmGLU-Net has better performance than other advanced patch-level and image-level methods with limited training samples. The relevant code will be opened at https://github.com/sssssyf/TrmGLU-Net

KEYWORDS:

• Hyperspectral image classification
• deep learning
• convolutional neural network
• self-attention
• U-Net
• superpixel segmentation

Introduction

Hyperspectral remote sensing images have the combined characteristics of an image and a spectrum, and they contain both spectral and spatial information of surface objects. Different objects have unique spectral characteristics (S. Li et al., 2019). Because of this correlation, hyperspectral image (HSI) has been widely used in various geological tasks, such as recognition of tree species, precise extraction of water boundaries, and statistical analyses of land use (Y. Zhong et al., 2018). Such applications are inseparable from HSI classification methods. Although rich spectral and spatial information offers great possibilities for fine-scale differentiation between objects, it also poses great challenges to classification tasks. Early HSI classification research focused on mining spectral features and classification algorithms. During that time, machine learning algorithms, such as support vector machines (SVMs) (Melgani & Bruzzone, 2004), random forests (L. Zhang et al., 2018), linear discriminant analysis (C. Li et al., 2011), and neural networks, were successively applied to solving HSI classification problems. Facing limited computing resources, researchers have explored different methods of data dimensionality reduction (Agarwal et al., 2007; Luo et al., 2020; Villa et al., 2010) and band selection (Cai et al., 2020; B. Fang et al., 2020; Q. Wang et al., 2020; Zeng et al., 2019) to deal with the high dimensionality of hyperspectral images. Although early studies made improvements to the classification of hyperspectral images, the following three problems still must be solved for HSI classification: the limited availability of labelled training samples, which leads to overfitting, high dimensionality of HSI data and the high correlation between adjacent bands, which further aggravates overfitting, and limited capability to distinguish different objects by spectral characteristics alone because they may have the same spectral characteristics.

The existence of these issues severely limits the classification accuracy. To improve HSI classification accuracy, researchers have incorporated spatial information into classification. Many methods use spatial information, and they can be broadly classified into three types. The first is represented by methods of spatial feature extraction, such as extended morphological profiles (EMPs) (Benediktsson et al., 2005), Gabor (Jia et al., 2019), and local binary patterns (W. Li et al., 2015). These methods account for the information of the neighboring pixels of sample points in feature extraction and input the extracted features into the classifier to complete the classification process. The second type considers the distance between samples. The closer the samples, the higher the probability that they belong to the same class. Such correlations are used to constrain the classifier during classification to improve accuracy. The third type focuses on the problem of classification noise in the initial classification result of hyperspectral images, which are mainly denoised by using morphological filtering and other methods. The introduction of spatial information has improved the accuracy of HSI classification. However, the limited availability of labelled training samples continues to restrict its development and application. To this end, researchers are exploring semi-supervised learning methods, such as label propagation (J. Zhang et al., 2020), transductive SVM (Bruzzone et al., 2005), collaborative training (Wan et al., 2015), and active learning (Z. Wang et al., 2017). This activity has resulted in the rapid development of HSI classification methods, giving rise to spatial – spectral classification and semi-supervised classification. The methods are effective in improving the classification accuracy of hyperspectral images while addressing the availability issue of labelled samples to some extent. However, classification performance with these methods relies heavily on expert experience, and it usually requires manual designs of complex feature extraction rules as well as the setting of different hyperparameters for different data.

In recent years, as computing capabilities and data numbers continue to grow, data-driven methods represented by deep learning have gained great popularity in many tasks, including image recognition (K. He et al., 2016), object detection (Ren et al., 2017), semantic segmentation (Shelhamer et al., 2017), and three-dimensional (3D) reconstruction (Yao et al., 2019). Deep learning can automatically extract features from data for downstream tasks. Hence, researchers have introduced deep learning to HSI classification to develop more versatile classification methods. Commonly used deep learning models include autoencoders (Xing et al., 2016), deep belief networks (Y. Chen et al., 2015), convolutional neural networks (CNNs) (Hu et al., 2015), and recurrent neural networks (RNNs) (Mou et al., 2017). Among these, CNNs have shown good performance in dealing with high-dimensional images. Thus, those methods have received wide attention in HSI classification. Inspired by spatial-spectral classification, researchers take a certain sample point as the center and slice out a local image patch of a certain size from the hyperspectral image as the feature of that sample, which is then input to a two-dimensional (2D) CNN (Haut et al., 2019; B. Liu et al., 2018; L. Zhang et al., 2018), 3D CNN (Y. Chen et al., 2016), or other models for classification. To further improve classification accuracy, more recent deep learning-based methods are used, such as residual learning (Liu, Yu, Zhang, et al., 2021), DenseNet (Huang et al., 2017), and attention mechanisms (Kuiliang Gao, Yu, et al., 2021; Xue et al., 2021). Considering that different bands of hyperspectral images can be used as time-series data, RNNs, long short-term memory models, and others have been proposed for the classification of hyperspectral images. Besides, the transformer-based model has made some progress in HSI classification at present. A bidirectional encoder representations from transformers for HSI classification task (HSI-BERT) was early on thinking about using the transformer to capture the global dependence among pixels (J. He et al., 2020). A spatial-spectral transformer (SST) was proposed to utilize the transformer to solve the problem of extract dependency of long-distance sequential spectra (X. He et al., 2021). The spectral and spatial transformer block replacing convolution was developed in a novel spectral – spatial transformer network (SSTN) to extract features, which achieved superior performance (Z. Zhong et al., 2022). The SpectralFormer (Hong et al., 2022) rethought the issue of HSI classification from a sequential perspective to purely utilize the transformer to finish the task.

Although these methods can effectively lower the difficulty of training deep learning models with hyperspectral images, the limited availability of labelled training samples for HSI classification remains a problem. Besides, these explorations and attempts still belong to the methods that use local image patches as input. Therefore, numerous works have been proposed to solve the problem of small samples HSI classification in recent years. For instance, Data augmentation can alleviate sample scarcity, which allows the model to be more fully trained for HSI classification when there are not enough labels (Nalepa et al., 2020a). Transfer learning methods have been widely explored to enable the model to have a higher generalization ability for target data by training on the source data, so the model can quickly converge on the target domain and alleviate the dependence of training samples (Yifan Sun, Bing Liu, Xuchu Yu, Anzhu Yu, Kuiliang Gao et al., 2022). And the practice has verified the effectiveness of transfer learning both on convergence speed and improvement with limited samples (J. He et al., 2020; Lee et al., 2022; Nalepa et al., 2020b). Few-shot learning specializes in solving the problem with less labels, which is also explored to improve the model’s performance in HSI classification of small samples (Liu, Yu, Yu, et al., 2019). As a prevalent and effective scheme of few-shot learning, meta-learning mainly makes model had a capacity of learning to learn and has been explored to improve the performance of small samples HSI classification (Kuiliang Gao et al., 2020, 2022; Kuiliang; Gao, Liu, et al., 2021). Besides, due to the massive growth of labelled samples, except for the supervised learning paradigm, semi-supervised learning and unsupervised learning are defined according to the participation of unlabelled samples during the training. Among them, semi-supervised learning methods can combine unlabelled samples and labelled samples for joint learning, represented by generative adversarial network (GAN) (L. Zhu et al., 2018) and graph convolution network (GCN) (Hong et al., 2021). And the strategy of pseudo-label is also a representative method for semi-supervised learning and explored widely to utilize unlabelled samples to expand the number of training data (B. Fang et al., 2020). Unsupervised learning is explored to enable the network to learn representation of feature for HSI classification merely with unlabelled samples. The reconstruction-based methods with encoder-decoder architecture are representative and have been widely utilized to accomplish spectral-spatial feature learning (Mei et al., 2019; S. Zhang et al., 2022). And a novel unsupervised learning framework is constructed to track the spectral variation information to extract the spectrum motion feature (Sun, Liu, Yu, Yu, Gao, & Ding, 2022). As an influential branch of unsupervised learning, self-supervised learning aims to construct loss function by using some attributes of data instead of labelled samples, which involves contrastive learning and generative learning. Contrastive learning utilizes a special contrastive loss function to cluster the positive samples and alienate the negative samples, which enables networks to learn robust capacity of feature learning without labels (Hou et al., 2022; B. Liu et al., 2021; M. Zhu et al., 2022). Generative learning usually enables networks to be trained by generating samples, and a relevant method is proposed to enable network to learn spectral-spatial feature through recovering the information of masked samples (Xue et al., 2022). Numerous works above mentioned are significant exploration for the problem of HSI classification with limited training samples, which also effectively improves the performance of deep models under the condition of small samples (Sun et al., 2022). While almost all of these methods are designed for patch-level methods, how to improve the classification performance of image-level classification methods with limited training samples is still worth exploring.

Second, deep learning methods that use local image patches as input have been successfully used for HSI classification tasks (Sun et al., 2023). However, these methods suffer from two major problems: one is that a model having local image patches as input fails to perceive the contextual information of the whole scene; and the other is that these highly overlapping image patches generate a lot of redundant computations, which reduces classification efficiency. To address these problems, semantic segmentation models have been introduced to the classification of hyperspectral images. Specifically, whole hyperspectral images are input to the segmentation model, and the classification result for the whole scene is produced. By this approach, contextual information is utilized to improve classification accuracy while unnecessary computations are reduced. Classical semantic segmentation models include fully convolution networks (FCNs) (Shelhamer et al., 2017), U-Net (Ronneberger et al., 2015), SegNet (Badrinarayanan et al., 2017), and DeepLab (L. C. Chen et al., 2018). These models usually require a large number of densely labelled samples to optimize thousands of network parameters in a semantic segmentation task. Simultaneously, the global learning framework specializes in capturing contextual information is gradually applied in remote sensing field to obtain better performance (M. Zhu et al., 2022). However, the labelled samples tend to be sparse in HSI classification task. More importantly, the number of these labelled samples is usually small, which leads to poor performance when applying these classical segmentation models directly. For this reason, DSSNet (Pan et al., 2020) was proposed for HSI classification, which is a segmentation network containing four convolutional layers. To improve the classification accuracy by utilizing the contextual information in HSI more fully, researchers have developed segmentation models while considering the characteristics of HSI, such as the deep fully convolutional network-based spatial distribution prediction (Jiao et al., 2017), the spectral-spatial fully convolutional networks (SSFCN) (Xu et al., 2020), fast patch-free global learning (FPGA) framework based fully convolutional network (Zheng et al., 2020), the fully convolutional network with channel and spatial attention (FCN-CSA) (Jiang et al., 2021), PBiNet (B. Liu & Yu, 2021), FOctConvPA (Yifan Sun, Bing Liu, Xuchu Yu, Anzhu Yu, Zhixiang Xue et al., 2022) and the FullyContNet-Pyramid (D. Wang et al., 2022).

Owing to the use of local connections for reducing the number of parameters, CNNs tend to have restricted receptive fields and cannot perceive long time dependencies. This may be improved by increasing the convolutional kernel or using dilated convolution, but that can result in blurred feature boundaries. A transformer regards images as sequential one-dimensional data and learns features using the self-attention mechanism. Thus, the transformer structure has a global receptive field. Hence, it can be used to perceive contextual information in hyperspectral images.

However, except for the contextual information, the local details are also crucial because the purpose of HSI classification is to assign a class label to each pixel. To achieve the purpose of modelling both contextual information and local details, this paper proposes a global-local U-Net augmented by transformers (TrmGLU-Net) for small-sample classification of hyperspectral images. Specifically, the hyperspectral image is input as a whole. Then, alternating transformers and convolutional layers are used to perceive both its contextual and local information to improve the accuracy of HSI classification. This also allows the characteristic of transformer’s global receptive field to work better on image rather than just local patches.

The originality of this paper is shown from the following aspects:

• TrmGLU-Net is proposed, which takes a whole hyperspectral image as input and performs end-to-end training. TrmGLU-Net comprises alternating transformers and convolutional layers. Skip connections are used between encoders and decoders, which allow the model to improve classification accuracy by making full use of contextual and local information.

• A superpixel-based label expansion method is proposed to improve the performance of image-level methods under the condition of small samples. With this method, the original image is partitioned into segments, and the results of superpixel segmentation are used to expand labels to effectively increase supervision information and obtain higher classification accuracy.

• The validity of the proposed method for small sample classification of hyperspectral images is verified using four sets of hyperspectral images with artificially labelled samples. Quantitative and qualitative experiments suggest that the combination of TrmGLU-Net and the superpixel-based label expansion method can obtain better classification results than those of semi-supervised methods, providing better adaptability when only a small number of labelled samples are available for each class of objects.

The remainder of this paper is organized as follows. Section 2. introduces the proposed classification method. Section 3. describes the validation of the proposed method for small-sample classification and its adaptability to sample size through classification experiments on four sets of hyperspectral images. Section 4. concludes the paper.

TrmGLU-Net and superpixel-based label expansion

The proposed method for HSI classification combines TrmGLU-Net and superpixel-based label expansion. The following subsections introduces the architecture of TrmGLU-Net, then its components and the method for label expansion based on superpixel segmentation.

TrmGLU-Net architecture

The architecture of the proposed TrmGLU-Net is illustrated in Figure 1. The size of the model input is $B\times H\times W$, where B, $H$ and $W$ refer to the band number, height and width of the whole image, respectively. To fully utilize whole HSI data information, an embedding layer composed of one convolution layer is deployed to projected the HSI data into the standard dimension (64). The embedding layer guarantees all spectral information can be modelled and simultaneously avoids the huge computational consumption. The network is very concise and adopts the encoder – decoder scheme of U-Net in general. What differs is that only two alternating transformers and convolutional layers are used for encoding and decoding to respond to the small sample size. First, feature transformation is performed on the input image using a conventional 2D convolution with a step size of one. During the first stage of encoding, the feature map is input into the transformer layer. Then, the output features are aggregated through two 3$\times$3 convolutional layers, where the step size of the first convolutional layer is set to one, and the step size of the second convolutional layer is set to two. After the second convolution layer, the dimension of the feature space is doubled with the number of feature map channels. Next, the second stage of encoding begins. In this process, the original input is used to concatenate with the downsampled features, thus helpfully supplementing the missing details. Finally, the features output from the second stage of encoding are fed to the decoder through two transformers. The decoder and the encoder are similar in structure. However, the encoder downsamples the feature map through a convolutional layer with a step of two to aggregate features, whereas the decoder upsamples the feature map by using a deconvolution operation with a step of two. The following subsections introduce the components of TrmGLU-Net and the superpixel-based label expansion method.

Figure 1. TrmGLU-Net architecture.

Transformer

Transformer has self-attention mechanisms, allowing them to have global receptive fields and capture long-distance dependencies. However, if we want to use transformers to process the feature map in Figure 1, it is first necessary to convert the feature map into a sequential form. The transformer-based method originally used to solve the image problem simply originally used to solve an image problem was simply to cut the image into fixed-size patches (Dosovitskiy et al., 2020). It is obviously not conducive to high-resolution visual tasks, especially hyperspectral image classification related to dense prediction task. For hyperspectral image classification, it is necessary to establish the pixel-level dependencies when using transformer, but it will lead high computational complexity. The original transformer does not have high computational efficiency, and both contextual and local information must be used for HSI classification. Considering these facts, a window-based multi-head attention (W-MSA) model (B. Liu et al., 2021) was adopted in this study.

As shown in Figure 2, the feature map is first divided into $M\times M$ non-overlapping windows along the spatial dimension, and then it is evenly divided into $k$ non-overlapping parts along the channel dimension. Thus, we can obtain a total number of $\frac{H\times W}{M\times M}\times k$ independent windows. Next, the feature map in each window is expanded into a 2D matrix upon which the self-attention operation is performed. The $k$ denotes the number of heads for the transformer. For the feature map, $X$, the transformer can be expressed as:

(1) $X_k=\{X_k^1,X_k^2,\cdots \hspace{0.17em}{X}_k^N\},N=\frac{H\times W}{M\times M}$(1)

(2) $Y_k^i=Attention(X_k^i{W}_k^Q,X_k^i{W}_k^K,X_k^i{W}_k^V)$(2)

(3) $\widehat{Y_k}=\{Y_k^1,Y_k^2,\cdots \hspace{0.17em}{Y}_k^N\}$(3)

Figure 2. Schematic of the W-MSA model.

where $X_k^i$ refers to the $i$-th window for the $k$-th head. $W_k^Q$,$W_k^K$,$W_k^V$ denote the queries, keys, and values for the $k$-th head, respectively. In the W-MSA model, the attention calculations are assigned for inside of each window, so different windows can be treated as different batches when training. The attention calculations can be formulated as:

(4) $Attention(Q,\hspace{0.17em}K,\hspace{0.17em}V)=softmax(\frac{Q{K}^T}{k}+B)V$(4)

where $B$ is the relative position-encoding, softmax denotes the softmax normalized function. Like the original transformer, a fully connected layer is utilized to enhance the nonlinearity of the features following W-MSA. It should be noted that each window in the feature map must be flattened to a 2D matrix before being input to the transformer; it is then reverted to a feature map after the transformer before being input to the convolution layer. If$M\times M=H\times W$, that is, $N$ = 1, the model can be called global multi-head attention model, which attempts to establish the dependency pixel by pixel in the whole image. Compared with global multi-head attention models, the W-MSA model in Figure 2 can significantly reduce computational efforts from $O(H^2\times W^2\times k)$ to $O(M^2\times H\times W\times k)$. Besides, as the size of the feature map decreases with the deepening of the network, the receptive field of window actually increases, which attends better to contextual and local information. Therefore, it can be used to effectively improve the accuracy of HSI classification.

Convolutional layers

TrmGLU-Net employs three types of 3$\times$3 convolutions. A convolution layer with a step size of one is used to extract local features. A convolution layer with a step size of two is used to reduce the spatial dimension of the feature map for feature aggregation. An inverse convolution layer (ConvT2D) with a step size of two is used to upsample the feature map. The convolution operation can be described with the following equation.

In the $j$-th feature map of the $i$-th convolutional layer, the value of $v_{ij}^{xy}$ at ($x,y$) can be obtained using the following equation:

(5) $v_{ij}^{xy}=f(b_{ij}+\sum _{k=1}^m\sum _{p=0}^{P_i-1}\sum _{q=0}^{Q_i-1}{w}_{ijk}^{pq}{v}_{(i-1)k}^{(x+p)(y+q)})$(5)

where $P_i$ and $Q_i$ denote kernel sizes, $m$is the number of feature maps of the$i=1$ layer, $v_{(i-1)k}^{(x+p)(y+q)}$ is the value of the $k$-th feature map in the $i-1$ layer at $(x+p,y+q)$, $w_{ijk}^{pq}$ is the convolution kernel connected to the $k$-th feature map of the layer $i-1$, $b_{ij}$ denotes the bias, $f(\cdot )$ is the activation function. The bidirectional encoder representations from transformers uses Gaussian error linear units (GELUs) as the activation function. Therefore, the model in this paper also adopts GELUs as the activation function. ConvT2D involves the process where the feature map is first interpolated before the convolution operation.

Superpixel-based label expansion

Figure 3 shows that, in the image-level HSI classification task, only a small number of labelled samples are available for training, which are also sparsely distributed. Such weak supervision is likely to cause noise in the classification results and blurred boundaries between different classes. Considering that the sample points within the superpixel are very likely to have consistent labels, we propose a superpixel-based label expansion method. Specifically, superpixel segmentation is performed on the hyperspectral image, and the result of superpixel segmentation is used as a mask. For a superpixel containing labelled samples, the classes of the samples within that superpixel are labelled with the classes of known samples to greatly increase the number of labelled samples. The expansion of labels will significantly strengthen the supervision information of image-level task and thus the network can be trained more adequately.

Figure 3. Schematic of the superpixel-based label expansion method.

The accuracy of superpixel segmentation is the key to the result of the proposed label expansion method. Therefore, the segmentation results of simple linear iterative clustering (SLIC) are used to expand a small number of labels. The SLIC algorithm is simple and efficient in that it only requires the input of the number of superpixels, $K$, to obtain the segmentation results. The SLIC algorithm is as shown in Alg.1.

Extensive evidence suggests that good superpixel segmentation results can be obtained with 10 iterations for most image data. Therefore, the number of iterations in SLIC is set to 10 for later experiments. It is noted that the superpixel-based label expansion here is different from the common superpixel segmentation post-processing method, in fact, it utilizes the result of superpixel segmentation to extend the labelled sample before model training. Therefore, the superpixel-based label expansion is actually not a post-processing, and it can be understood as a pseudo-label technology to solve the problem of insufficient samples.

Experimental results and analysis

The validity of the proposed method was tested by classifying three HSI datasets. In terms of hardware settings, Intel Core Intel(R) Xeon(R) Gold 6152 central processing unit, Nvidia A100 PCIE GPU, 40-GB GPU memory, and 128-GB RAM were used. The algorithms were all implemented in Python and PyTorch. The overall accuracy (OA), average accuracy (AA), and Cohen’s kappa $(K)$ were selected as the evaluation criterion. All trials will run 10 times with different samples, and the average results were report to smoothen errors as far as possible. Besides, an additional two-tailed Wilcoxon test was conducted to show if the experimental conclusion were statistically significant (i.e. reporting the $p$-value).

Experimental data

To test the validity of the proposed method, four HSI scenes were used for classification (i.e. a University of Pavia scene, an Indian Pines scene, a Salinas scene and a Houston scene). The scenes were obtained by the Reflective Optics Spectrographic Imaging System (ROSIS-03), the Airborne Visible/Infrared Imaging Spectrometer (AVIRIS), again AVIRIS, and the ITRES CASI-1500 sensor, respectively. To quantitatively evaluate the performance of different classification algorithms, some regions of the images were manually labelled. Specifically, data from the University of Pavia scene were classified into 9 classes with a total of 42,776 samples; data from the Indian Pines scene were classified into 16 classes with a total of 10,776 samples; data from the Salinas scene were classified into 16 classes with a total of 54,129 samples; and data from the Houston scene were classified into 15 classes with a total of 15,029 samples. The properties of these HSIs are listed in Table 1. To verify the efficacy of the proposed method for small sample classification, samples were randomly selected from each class as training samples (i.e. 20, 10, 10 and 20 samples from the University of Pavia data, the Indian Pines data, the Salinas data, and the Houston data, respectively). The remaining samples were used as test samples. For the four HSIs, the number of samples used for training was 180, 160, and 160, and 300 respectively. The proportion of training samples to the total number of samples was 0.4%, 1.5%, 0.3% and 1.9%, respectively.

Table 1. The properties of the three HSIs.

	University of Pavia	Indian Pines	Salinas	Houston
Image size	610 × 340	145 × 145	512 × 217	349 × 1905
Number of bands	103	200	204	144
Spectral range	0.43–0.86 μm	0.4–2.5 μm	0.4–2.5 μm	0.38–1.05 μm
Spatial resolution	1.3 m	20 m	3.7 m	2.5 m
Sensor	ROSIS-03	AVIRIS	AVIRIS	ITRES CASI-1500
Number of feature classes	9	16	16	15
Sample size	42,776	10,776	54,129	15029

Parameter settings and analysis

The Adam optimization algorithm was used for network training with a learning rate of 0.00005 and 150 epochs. The expansion of superpixel tags directly affects the accuracy of classification, and SLIC requires prior input of the number of superpixels. Therefore, how to determine the number of segments becomes a key problem. To make this process more rational, we utilize a soft turning mechanism. Concretely, we firstly define a space of hard segments number $\mathrm{\Theta }=\{100,500,1000,2000,5000,10000\}$ to test the effect of different number of superpixels on the classification accuracy and observe the variation trend. Each experiment was repeated 10 times. Figure 4 presents the box plots of overall accuracy with different numbers of segments used for the four HSIs under the defined space. The experimental results in Figure 4 show that both too few (100, 500) and too many (5000, 10000) superpixels lead to a decline in classification accuracy, except for the Houston dataset. The lower the number of superpixels and the higher the number of image points contained within each superpixel, the higher the probability of introducing noise in the label, thus causing a decrease in classification accuracy. On the other hand, the higher the number of superpixels, the lower the number of image points contained in each superpixel. Thus, the expanded labels were more accurate, but the number of expanded labels was smaller. Hence, there was a slight decrease in classification accuracy. As for the Houston scene, maybe it is too complex and contains a large number of linear objects, so it is not suitable to contain too many points in each superpixel. Secondly, we further narrow the range of optimal values according to the results of the previous step. We narrow the range to (500, 5000) for the first three scenes and to (5000, 13000) for the Houston scene by taking one value as the new experimental number of segments every 100 intervals within this range. In order to facilitate intuitive comparison, we fix the training sample and the variation trend of classification performance with number of segments is shown in Figure 5. Thus, the optimal range is further compressed. Taking the University of Pavia scene as an example, its range is narrowed to (1200, 1300). Next, in order to improve accuracy and give consideration to efficiency, the interval is further reduced to 5 to determine the final optimal range. Therefore, the final optimal range of number of segments for four scenes are determined to (1200, 1300), (1100, 1300), (530, 565) and (11010, 12030), respectively. And the minimal of each range is selected as the final number of segments for each scene.

Figure 4. Overall accuracy with different number of segments:(a) University of Pavia; (b) Indian Pines; (c) Salinas (d) Houston.

Figure 5. The variation trend of the overall accuracy with different number of segments.

Ablation study

To demonstrate the enhancing effect of transformers and the effectiveness of superpixel-based label expansion, an ablation study was designed as follows. First, the transformers were removed from TrmGLU-Net to obtain GLU-Net for classification experiments, which were conducted by using TrmGLU-Net. Finally, superpixel-based label expansion was incorporated into the classification with TrmGLU-Net. Figure 6 presents the box plot distributions of classification accuracy with different methods. The label “Augment” refers to TrmGLU-Net, which was trained with labels expanded by superpixel segmentation. First, by comparing GLU-Net and TrmGLU-Net, it can be seen that after the transformers were removed, there was a significant decrease in the mean value of classification accuracy for the 10 experiments $(p<0.005)$. This result indicates that the introduction of transformer layer in GLU-Net promotes feature extraction, thus improving the classification accuracy of the model. We also compared the TrmGLU-Net trained with the original labels and the TrmGLU-Net trained with labels expanded based on superpixel segmentation. It was found that label expansion based on the results of superpixel segmentation significantly improve classification accuracy $(p<0.005)$, showing the effectiveness of the proposed method.

Figure 6. Overall accuracy with three methods: (a) University of Pavia; (b) Indian Pines; (c) Salinas (d) Houston.

Besides, as the TrmGLU-Net is proposed for small sample classification, we consider inputting data with high dimensionality may cause overfitting. Besides, the 3-channel input is a standard form for conventional Transformer model. Therefore, original hyperspectral images are transformed through principal component analysis (PCA), and the first three principal components are taken as input. The comparative results are shown as in Table 2. As we can observe, the input after PCA cannot improve the performance by reducing the dimensionality of data. And the step of PCA even weakens the performance to an extent because of potential information loss. By contrast, the full data as input can obtain higher overall accuracy on all HSI data, which shows the necessity of keeping full-dimension information. Therefore, we retain the setting of inputting the full-dimension HSI data.

Table 2. Ablation results with principal component analysis (PCA) (%).

Scheme	PCA	Dataset
		PU	IP	SA	HO
A	√	87.83	81.74	93.60	87.51
B	×	91.47	82.95	94.32	89.97

Comparison of methods

Tables 3–6 show the per-class accuracy, overall accuracy, average accuracy, and Cohen’s kappa of different methods on the four HSIs. The experiment was repeated 10 times for each method. The mean and standard deviation for the 10 experiments are listed in the tables, and the $p$-value of two-tailed Wilcoxon test are also reported. Combined with the kernel method, SVM (Melgani & Bruzzone, 2004) can address the overfitting issue caused by small sample sizes to a certain extent, and it has been widely used for HSI classification. Therefore, the SVM with spectral features as the direct input was chosen as the benchmark. Then, the classical EMP (Benediktsson et al., 2005) was selected as a representative method for manual feature extraction. Given the large number of patch-level deep learning methods, we selected three popular methods (i.e. 2D-CNN (Yue et al., 2016), 3D-CNN (Y. Chen et al., 2016) and the double-branch multi attention mechanism network (DBAM) (Ma et al., 2019)) for comparison. Additionally, deep few-shot learning (DFSL) (Liu, Yu, Yu, et al., 2019) and convolutional neural network with model-agnostic meta-learning algorithm (CNN-MAML) (Kuiliang Gao, Liu, et al., 2021) were also selected, which were both specifically designed for small-sample classification. The DFSL uses multiple previously collected HSIs to pretrain the model, and the trained network can be applied to process target HSIs as a feature extractor. The CNN-MAML optimizes the CNN model on plenties of different tasks to make it more general, so the model can better adapt to target tasks with only small samples. The last method was the latest image-level method FCN-Pyramid (D. Wang et al., 2022) with an attention mechanism, which also takes the whole hyperspectral image as model input and directly outputs the classification results of the whole scene. As shown in the table, the SVM, which used only spectral features, had the lowest classification accuracy for the four HSIs, whereas EMPs with spatial information achieved higher classification accuracy. In addition, the classification accuracy was improved by using deep learning methods based on local image blocks (e.g. 2D-CNN, 3D-CNN and DBAM), improving the accuracy. A significant increase was observed when using DBAM. A high classification accuracy was obtained with both DFSL and CNN-MAML on the University of Pavia, Salinas and Houston scenes, whereas their performance was relatively poor on the Indian Pines scene, which had lower resolution. By contrast, FCN-Pyramid only performs well on Salinas and Indian Pines scenes consisting mainly of planar objects, and it did not offer significant advantages over DBAM and DFSL owing to the small number of labelled samples. For all four HSIs, the highest overall accuracy was obtained by using the method proposed in this paper $(p<0.005)$. For example, on the University of Pavia scene, the average of the overall accuracy scores of TrmGLU-Net+Aug was more than 7.52% higher than that of SVM and 3.57% higher than that of FCN-Pyramid, which is also an image-level classification method. The advantage of TrmGLU-Net+Aug became more obvious when classifying the more challenging Indian Pines scene. Its accuracy was nearly 23.9% higher than that of SVM and 5.03% higher than that of FCN-Pyramid.

Table 3. Classification results with different methods on the University of Pavia dataset (%).

No.	Traditional		Patch-Level					Image-Level
	SVM	EMPs	2D-CNN	3D-CNN	DBAM	DFSL	CNN-MAML	FCN-Pyramid	TrmGLU-Net+Aug
1	94.20 ± 2.98	96.16 ± 2.13	95.28 ± 4.31	93.22 ± 2.33	88.20 ± 0.03	92.57 ± 1.88	94.79 ± 6.02	95.80 ± 2.32	92.09 ± 1.65
2	91.30 ± 1.68	93.32 ± 1.53	95.20 ± 4.35	90.55 ± 1.61	91.70 ± 0.03	97.14 ± 0.93	97.10 ± 0.83	94.80 ± 1.50	90.75 ± 2.90
3	56.49 ± 5.30	61.98 ± 11.45	69.84 ± 12.74	55.99 ± 7.15	82.79 ± 0.14	76.36 ± 3.97	63.21 ± 10.49	67.40 ± 12.85	89.84 ± 2.37
4	68.65 ± 10.81	85.71 ± 11.85	86.45 ± 10.30	88.94 ± 9.55	94.71 ± 0.09	91.97 ± 10.033	81.46 ± 19.02	85.26 ± 13.63	80.47 ± 1.79
5	93.95 ± 2.62	94.64 ± 2.49	98.33 ± 1.38	94.85 ± 4.22	98.60 ± 0.02	100.0 ± 0.0	99.97 ± 0.04	93.70 ± 14.48	99.73 ± 0.20
6	55.79 ± 12.09	56.14 ± 11.17	66.76 ± 17.26	57.55 ± 6.97	80.64 ± 0.16	65.92 ± 11.84	78.51 ± 9.61	63.69 ± 16.28	98.77 ± 1.18
7	48.75 ± 6.44	67.70 ± 7.75	74.25 ± 18.16	60.21 ± 8.31	79.87 ± 0.19	70.51 ± 7.68	76.77 ± 6.7	63.91 ± 11.33	99.12 ± 0.41
8	76.52 ± 5.57	82.86 ± 3.76	79.38 ± 9.83	78.99 ± 5.65	69.52 ± 0.05	78.31 ± 3.19	84.58 ± 6.06	68.11 ± 8.78	91.87 ± 3.04
9	99.89 ± 0.08	99.94 ± 0.05	96.45 ± 2.53	89.16 ± 4.31	91.83 ± 0.08	95.55 ± 10.65	99.24 ± 0.54	89.87 ± 12.77	77.61 ± 0.81
OA	81.92 ± 1.12	85.80 ± 1.72	82.81 ± 3.96	83.44 ± 1.24	86.63 ± 0.04	88.96 ± 1.42	87.55 ± 3.56	85.87 ± 2.79	91.47 ±1.46
AA	76.17 ± 0.89	82.05 ± 2.07	84.66 ± 3.05	78.83 ± 1.82	86.43 ± 0.05	85.37 ± 2.10	86.18 ± 2.62	80.28 ± 5.29	91.14 ±0.79
K	70.60 ± 1.85	76.65 ± 4.68	79.24 ± 4.80	75.73 ± 2.12	82.18 ± 0.06	82.52 ± 4.65	83.98 ± 4.34	76.54 ± 6.24	88.89 ±1.85

Table 4. Classification results with different methods on the Indian Pines dataset (%).

No.	Traditional		Patch-Level					Image-Level
	SVM	EMPs	2D-CNN	3D-CNN	DBAM	DFSL	CNN-MAML	FCN-Pyramid	TrmGLU-Net+Aug
1	29.16 ± 6.20	42.01 ± 15.48	68.14 ± 15.25	37.43 ± 9.62	69.23 ± 0.23	53.02 ± 8.08	95.18 ± 3.33	63.15 ± 16.53	59.14 ± 6.45
2	45.67 ± 8.35	53.18 ± 7.98	73.3 ± 12.01	44.83 ± 8.74	73.51 ± 0.15	57.06 ± 7.47	77.08 ± 11.43	68.98 ± 6.16	85.82 ± 3.85
3	35.63 ± 5.90	44.19 ± 4.42	56.71 ± 13.15	33.10 ± 5.06	73.70 ± 0.19	49.44 ± 10.27	62.97 ± 23.27	57.78 ± 2.44	95.41 ± 1.88
4	32.17 ± 4.23	36.05 ± 6.25	56.23 ± 14.19	40.55 ± 9.24	63.99 ± 0.28	43.39 ± 8.75	67.53 ± 12.99	54.39 ± 7.67	89.59 ± 4.87
5	61.55 ± 11.74	66.90 ± 9.29	94.28 ± 11.02	49.72 ± 8.99	88.72 ± 0.11	85.50 ± 7.25	94.42 ± 4.74	94.74 ± 3.66	68.20 ± 4.07
6	87.40 ± 4.10	90.80 ± 2.41	97.92 ± 1.13	69.57 ± 5.42	94.48 ± 0.04	95.89 ± 2.12	98.57 ± 1.36	87.85 ± 4.88	74.45 ± 3.97
7	35.09 ± 12.30	39.10 ± 6.38	36.35 ± 9.56	31.38 ± 17.066	27.27 ± 0.13	52.90 ± 19.30	48.57 ± 18.04	47.12 ± 14.12	33.79 ± 4.60
8	94.21 ± 5.06	98.70 ± 1.37	100.00 ± 0.00	93.86 ± 4.11	98.36 ± 0.02	98.29 ± 1.66	99.33 ± 0.91	87.39 ± 4.46	82.54 ± 4.68
9	16.38 ± 3.99	29.69 ± 8.62	23.94 ± 14.83	15.76 ± 6.41	24.41 ± 0.13	30.46 ± 8.46	56.04 ± 29.63	19.01 ± 2.27	11.84 ± 1.77
10	38.23 ± 10.92	50.27 ± 8.16	65.06 ± 8.24	48.39 ± 6.07	73.89 ± 0.14	58.60 ± 9.65	73.72 ± 13.77	53.03 ± 4.29	83.80 ± 10.13
11	61.50 ± 7.94	71.46 ± 4.16	81.32 ± 8.27	71.17 ± 4.98	75.42 ± 0.09	76.32 ± 7.35	74.35 ± 12.54	90.63 ± 2.94	90.98 ± 1.22
12	28.07 ± 4.53	36.12 ± 4.19	58.04 ± 12.75	35.45 ± 7.05	67.95 ± 0.18	39.86 ± 4.04	83.01 ± 14.86	60.55 ± 6.89	87.17 ± 5.31
13	78.72 ± 8.28	89.63 ± 5.01	94.05 ± 5.73	71.84 ± 13.28	89.41 ± 0.10	86.74 ± 6.96	99.50 ± 0.78	87.03 ± 7.56	79.67 ± 3.76
14	90.81 ± 2.96	93.70 ± 2.62	98.06 ± 1.90	79.34 ± 3.24	92.98 ± 0.08	96.65 ± 2.86	96.44 ± 1.04	96.36 ± 2.04	96.21 ± 1.10
15	44.37 ± 13.47	61.06 ± 8.87	69.45 ± 13.17	51.84 ± 6.45	72.19 ± 0.11	57.02 ± 8.34	83.77 ± 6.23	66.93 ± 5.52	62.63 ± 3.72
16	92.98 ± 13.69	98.26 ± 1.83	78.43 ± 9.93	58.65 ± 14.74	87.39 ± 0.10	88.19 ± 7.58	88.69 ± 0.97	80.51 ± 6.05	88.34 ± 2.47
OA	59.05 ± 2.86	66.92 ± 2.04	74.24 ± 4.16	59.23 ± 1.97	76.82 ± 0.05	71.60 ± 2.33	76.18 ± 2.80	77.92 ± 1.23	82.95 ±1.38
AA	54.50 ± 1.88	62.57 ± 1.80	71.95 ± 3.82	52.05 ± 2.05	73.30 ± 0.07	66.83 ± 2.43	81.20 ± 3.06	69.72 ± 1.26	89.50 ±0.77
K	47.77 ± 3.21	58.43 ± 2.15	71.11 ± 4.46	50.03 ± 2.86	73.33 ± 0.06	63.31 ± 2.39	72.91 ± 2.92	70.76 ± 1.11	80.55 ±1.84

Table 5. Classification results with different methods on the Salinas dataset (%).

No.	Traditional		Patch-Level					Image-Level
	SVM	EMPs	2D-CNN	3D-CNN	DBAM	DFSL	CNN-MAML	FCN-Pyramid	TrmGLU-Net+Aug
1	91.65 ± 5.47	96.81 ± 2.81	93.48 ± 3.35	79.37 ± 15.48	99.99 ± 0.01	100.0 ± 0.0	100.00 ± 0.00	95.60 ± 1.22	99.99 ± 0.02
2	98.63 ± 0.46	98.90 ± 0.55	94.78 ± 0.47	94.91 ± 4.09	96.49 ± 0.05	99.93 ± 0.17	99.79 ± 00.18	98.88 ± 1.50	94.97 ± 2.05
3	87.69 ± 5.99	87.81 ± 3.28	91.91 ± 1.85	84.72 ± 5.10	93.86 ± 0.06	91.48 ± 6.03	96.1 ± 3.30	84.38 ± 8.46	98.94 ± 0.34
4	97.14 ± 1.51	96.56 ± 1.26	88.82 ± 4.49	91.90 ± 5.08	90.04 ± 0.05	98.92 ± 0.70	98.34 ± 0.98	81.26 ± 8.64	94.89 ± 1.30
5	95.22 ± 4.64	95.30 ± 4.16	94.16 ± 0.95	94.71 ± 2.78	94.03 ± 0.07	97.04 ± 2.86	97.97 ± 1.47	94.93 ± 3.91	98.07 ± 0.62
6	99.48 ± 0.41	99.92 ± 0.11	94.95 ± 0.07	98.41 ± 2.14	98.68 ± 0.01	100.0 ± 0.0	99.92 ± 0.15	98.84 ± 0.65	97.30 ± 0.73
7	96.27 ± 2.60	96.02 ± 2.56	94.47 ± 0.81	95.02 ± 1.53	96.02 ± 0.04	99.76 ± 0.29	99.99 ± 0.01	97.48 ± 2.82	86.90 ± 1.19
8	72.23 ± 8.92	77.28 ± 4.99	80.91 ± 7.18	82.46 ± 6.80	83.31 ± 0.07	81.15 ± 6.15	83.78 ± 2.26	89.50 ± 1.22	85.74 ± 2.73
9	98.29 ± 1.24	98.72 ± 0.91	93.77 ± 1.22	96.49 ± 1.50	98.65 ± 0.01	98.69 ± 1.12	98.83 ± 1.39	98.27 ± 0.86	99.94 ± 0.04
10	84.75 ± 5.50	84.05 ± 5.12	88.37 ± 2.75	80.08 ± 2.73	91.87 ± 0.05	95.94 ± 1.81	93.79 ± 3.63	88.96 ± 8.32	97.13 ± 0.80
11	70.89 ± 7.38	73.86 ± 6.18	89.78 ± 2.77	54.55 ± 7.62	91.28 ± 0.06	67.98 ± 5.96	95.37 ± 2.39	79.26 ± 5.45	94.13 ± 1.20
12	93.58 ± 3.46	93.87 ± 1.64	93.71 ± 1.58	93.89 ± 5.33	97.21 ± 0.05	98.98 ± 0.45	99.11 ± 0.59	96.03 ± 1.94	92.56 ± 1.29
13	89.40 ± 4.74	93.28 ± 3.28	94.85 ± 0.20	82.57 ± 14.11	91.28 ± 0.11	99.99 ± 0.03	99.01 ± 1.35	59.68 ± 6.12	95.69 ± 1.57
14	84.40 ± 7.60	87.15 ± 14.81	86.01 ± 6.45	73.66 ± 9.74	93.67 ± 0.05	98.24 ± 1.50	95.86 ± 2.84	73.11 ± 5.91	99.03 ± 0.31
15	52.28 ± 7.01	59.68 ± 5.03	61.45 ± 10.13	61.34 ± 8.01	71.45 ± 0.16	66.59 ± 4.30	68.33 ± 6.68	80.80 ± 4.65	97.18 ± 0.70
16	94.36 ± 3.70	89.56 ± 9.49	94.91 ± 0.22	82.66 ± 7.89	97.82 ± 0.03	98.96 ± 1.57	99.79 ± 0.42	68.23 ± 16.99	100.0 ± 0.0
OA	83.76 ± 2.53	86.06 ± 0.99	85.09 ± 2.78	84.45 ± 2.74	88.01 ± 0.05	89.95 ± 1.35	90.33 ± 1.46	89.75 ± 0.83	94.32 ±0.54
AA	87.89 ± 1.40	89.30 ± 1.15	89.77 ± 1.07	84.17 ± 2.14	92.85 ± 0.02	93.35 ± 0.86	94.37 ± 0.89	86.58 ± 1.24	95.78 ±0.31
K	80.71 ± 3.09	83.39 ± 1.24	84.01 ± 3.07	80.45 ± 3.68	86.69 ± 0.05	87.97 ± 1.57	89.81 ± 1.60	87.15 ± 1.11	93.70 ±0.59

Table 6. Classification results with different methods on the Houston dataset (%).

No.	Traditional		Patch-Level					Image-Level
	SVM	EMPs	2D-CNN	3D-CNN	DBAM	DFSL	CNN-MAML	FCN-Pyramid	TrmGLU-Net+Aug
1	92.38 ± 6.67	92.67 ± 6.69	85.05 ± 6.59	89.98 ± 5.29	90.5 ± 5.38	97.44 ± 0.75	91.10 ± 6.67	89.28 ± 5.07	93.72 ± 2.70
2	89.83 ± 5.68	90.77 ± 5.81	94.6 ± 5.04	92.11 ± 5.91	93.2 ± 6.48	98.51 ± 0.44	92.91 ± 5.09	78.32 ± 8.88	88.77 ± 2.62
3	98.55 ± 0.79	98.90 ± 0.67	99.99 ± 0.04	99.93 ± 0.15	99.89 ± 0.34	99.71 ± 0.18	100.00 ± 0.0	97.87 ± 3.48	99.58 ± 0.27
4	93.12 ± 2.22	93.68 ± 1.84	96.39 ± 6.58	98.61 ± 1.35	98.95 ± 0.72	93.15 ± 0.41	96.94 ± 4.46	92.68 ± 1.79	89.73 ± 0.99
5	96.71 ± 3.74	98.02 ± 0.89	94.91 ± 4.25	91.13 ± 3.80	96.66 ± 2.64	98.42 ± 0.48	96.03 ± 2.96	93.71 ± 6.19	99.44 ± 0.39
6	87.50 ± 3.62	90.33 ± 5.04	97.31 ± 3.54	97.63 ± 4.23	97.29 ± 2.99	99.36 ± 0.48	96.67 ± 9.98	79.93 ± 8.22	97.63 ± 1.34
7	76.32 ± 4.39	79.48 ± 5.56	79.86 ± 9.31	80.28 ± 9.08	86.49 ± 11.03	85.30 ± 1.57	84.96 ± 6.78	81.21 ± 5.57	90.15 ± 2.62
8	51.26 ± 8.94	52.75 ± 7.80	85.23 ± 11.27	85.33 ± 11.42	90.89 ± 12.54	71.33 ± 1.21	93.13 ± 6.23	48.72 ± 5.26	76.86 ± 2.30
9	76.10 ± 3.62	73.73 ± 3.12	69.51 ± 8.50	72.87 ± 11.96	80.59 ± 7.56	76.02 ± 1.28	76.53 ± 5.29	81.91 ± 5.05	85.41 ± 1.73
10	62.42 ± 11.21	70.03 ± 10.43	62.93 ± 9.92	67.09 ± 8.68	67.15 ± 8.61	85.92 ± 2.26	72.23 ± 8.59	63.81 ± 7.86	89.77 ± 7.02
11	62.46 ± 8.34	63.49 ± 4.74	67.41 ± 9.62	70.58 ± 9.08	82.65 ± 10.94	75.05 ± 3.37	79.68 ± 10.26	72.71 ± 12.05	82.32 ± 2.61
12	47.42 ± 9.25	56.53 ± 12.18	67.81 ± 10.77	71.20 ± 7.63	73.06 ± 10.49	77.26 ± 1.68	76.73 ± 10.70	76.90 ± 5.14	87.72 ± 1.85
13	34.47 ± 8.93	38.49 ± 6.09	71.06 ± 14.48	85.55 ± 9.80	82.83 ± 10.82	59.51 ± 3.01	83.6 ± 7.78	94.24 ± 1.74	98.33 ± 1.26
14	98.11 ± 1.25	98.49 ± 1.17	93.08 ± 5.20	93.67 ± 5.37	94.14 ± 5.74	99.73 ± 0.40	97.32 ± 2.89	99.90 ± 0.19	100.0 ± 0.00
15	98.79 ± 0.49	98.71 ± 0.47	96.01 ± 1.66	96.04 ± 2.95	94.89 ± 0.78	99.04 ± 0.51	95.74 ± 2.31	95.21 ± 7.33	93.19 ± 3.26
OA	76.68 ± 1.25	78.79 ± 1.37	81.28 ± 2.97	82.91 ± 2.07	85.88 ± 2.01	85.76 ± 0.50	86.67 ± 2.68	80.82 ± 1.92	89.97 ±0.98
AA	77.70 ± 0.99	79.74 ± 1.18	84.08 ± 2.64	86.13 ± 1.78	88.61 ± 1.65	86.72 ± 0.44	88.91 ± 2.11	83.09 ± 1.48	91.51 ±0.91
K	74.79 ± 1.34	77.06 ± 1.48	79.78 ± 3.21	81.53 ± 2.24	84.74 ± 2.17	84.61 ± 0.54	85.59 ± 2.09	79.30 ± 2.06	89.16 ±1.06

Figure 7–10 show the maps obtained with different classification algorithms. It can be seen from the maps that isolated noise points often occur in the maps obtained with pixel-based or patch-level HSI classification methods. In contrast, classification maps with better visual results were obtained by using image-level classification methods (e.g. FCN-Pyramid and TrmGLU-Net+Aug), though their misclassified samples were often distributed in blocks. The results of the qualitative evaluation in Figures 7–10 are consistent with the results of quantitative evaluation in Tables 2–5. In both cases, the proposed TrmGLU-Net+Aug obtained the best classification results, demonstrating its effectiveness.

Figure 7. Classification maps with different methods on the University of Pavia dataset. (a) Ground-truth map. (b) SVM. (c) EMPs. (d) 2D-CNN. (e) 3D-CNN. (f) DBAM. (g) DFSL+SVM. (h) CNN-MAML. (i) FCN-Pyramid. (j) TrmGLU-Net+Aug.

Figure 8. Classification maps with different methods on the Indian Pines dataset. (a) Ground-truth map. (b) SVM. (c) EMPs. (d) 2D-CNN. (e) 3D-CNN. (f) DBAM. (g) DFSL+SVM. (h) CNN-MAML. (i) FCN-Pyramid. (j) TrmGLU-Net+Aug.

Figure 9. Classification maps with different methods on the Salinas dataset. (a) Ground-truth map. (b) SVM. (c) EMPs. (d) 2D-CNN. (e) 3D-CNN. (f) DBAM. (g) DFSL+SVM. (h) CNN-MAML. (i) FCN-Pyramid. (j) TrmGLU-Net+Aug.

Figure 10. Classification maps with different methods on the Houston dataset. (a) Ground-truth map. (b) SVM. (c) EMPs. (d) 2D-CNN. (e) 3D-CNN. (f) DBAM. (g) DFSL+SVM. (h) CNN-MAML. (i) FCN-Pyramid. (j) TrmGLU-Net+Aug.

Conclusion

By utilizing the contextual information in a hyperspectral image, this paper obtained the classification results of the whole scene directly by inputting the whole image into the segmentation model. A non-local TrmGLU-Net is proposed for the small sample classification of hyperspectral images. This proposed network has a wider receptive field and can attend to both local and global features, which greatly improves the accuracy of HSI classification. To obtain high-accuracy classification with small sample sizes, a superpixel label expansion method is also proposed to increase the number of labelled samples to further improve classification accuracy. The classification performance of the proposed method on four popular HSI datasets is compared with that of other classification methods, including 3D-CNN, DBAM, DFSL, CNN-MAML and FCN-Pyramid. The results show that the proposed method achieves higher classification accuracy than the others.

However, the form of inputting the whole image will lead to tremendous hardware stress such as GPU memory. We will further propose corresponding strategies to address the issue in subsequent studies.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This research was funded by Natural Science Foundation of Henan Province [222300420387]