SSC-SFN: spectral-spatial non-local segment federated network for hyperspectral image classification with limited labeled samples

ABSTRACT

Hyperspectral image (HSI) classification methods based on deep learning (DL) have performed well in numerous investigations. Although many modified superpixel-wise neural networks are utilized to enhance spatial information, their ability to mine spectral information in graph structures is insufficient. Moreover, single classifier approaches are unable to extract adequate spatial and spectral information simultaneously. For the classification of large-scale research areas, many works have relied on the use of a large number of labeled samples, leading to low efficiency and weak generalization. To address these issues, an effective spectral-spatial HSI classification approach based on spectral-spatial non-local segment federated network (SSC-SFN) was developed in this study. In this framework, deconvolution is employed to recover the data size, while the lost spatial information is replaced by up-pooling. The spectral dimensional features are updated through the generation of non-Euclidean graph structures and the non-local segment smoothing technique. The convolutional neural network and graph convolutional network techniques are coupled to exploit the available spectral and spatial structure information fully. Extensive experimental results obtained using four public benchmark datasets show that the classification accuracy of SSC-SFN can exceed 90% for large-scale HSIs with limited samples.

KEYWORDS:

• Hyperspectral image
• segment anything
• convolutional neural network
• graph convolutional network
• federated network

1. Introduction

Hyperspectral images (HSIs) with dozens or even hundreds of bands contain abundant spectral and spatial information, which facilitates extensive, in-depth studies of the Earth's surface. Hence, land use and land cover classification (Anderson 1976), environmental monitoring (Galieni et al. 2022), Earth observation management (Park and Song 2020), target detection (Li et al. 2020), and urban area planning (Hänsch and Hellwich 2021) all involve HSI classification. Problems including salt-and-pepper noise, spatial heterogeneity, and the complexity of ground surfaces make it challenging to perform accurate HSI classification with a small number of labeled samples (Abdelmoneim, Soliman, and Moghazy 2020; Irteza et al. 2021; Mukherjee and Singh 2020). A convolutional neural network (CNN) can extract rich spectral features through subsampling, while a graph convolutional network (GCN) can extract discriminative and nonlinear spectral-spatial features. However, HSI classification methods based on deep learning (DL) are limited by the fixed-format convolution kernel, which ignores many geometric details of high-dimensional data. Specifically, the sliding window of operating rules in the CNN is unable to adaptively extract important features. The primary drawback of GCN is that spectral information is not given enough consideration and only spatial context information is mined through the graph structure (Chen, Li, and Dai 2022; Yang et al. 2022; Zhang et al. 2017). Meanwhile, spatial information is not properly utilized after data segmentation, and the generalization ability of this method is poor for large-scale data (Amirabbas et al. 2018; Cao and Wang 2017). Therefore, the development of a more efficient and general method for classifying large-scale complex background areas with limited samples is essential. A method for classifying large-scale complex data based on limited samples is also urgently needed.

Many DL-based image processing algorithms have been successfully extended to HSI classification in the field of remote sensing (Gong et al. 2021; Hong et al. 2020; Hong et al. 2022; Li and Ning 2020; Liu et al. 2018a; Masolele et al. 2022). CNN and GCN are two effective and popular classification algorithms, but CNN only considers the spectral information of HSI, while ignoring the equally important spatial structure information (Ahishali et al. 2021; Dong et al. 2022; Jiang et al. 2022; Ke et al. 2018; Ruan et al. 2022). In contrast, although GCN skillfully represents the spatial topological relationship between samples, the rich spectral information is not adequately mined (Li, Chen, and Cheng 2022; Lu et al. 2022; Ye et al. 2021; Zhang et al. 2022). Therefore, Li, Chen, and Cheng (2022) designed global attention blocks to optimize spectral and spatial features for enhanced learning. Wang et al. (2021) applied a GCN to extract discriminative and nonlinear spectral-spatial features from HSI, to replace linear mapping features, and gradually enhanced the feature representation capabilities. However, the adjacency matrix generates enormous computational expenses when GCN classifies a large-scale high-dimensional HSI, causing the gradient to explode or vanish (He et al. 2022b; Shahraki and Prasad 2018; Wang et al. 2021). Second, a constructed graph structure usually contains a variety of noise types associated with the complex background, which means that the edges (relationships) between nodes are sometimes not credible, and that the spatial neighborhood information obtained may be incorrect (He et al. 2022a; Qin et al. 2019; Sun et al. 2022). Finally, in DL-based HSI classification methods, massive numbers of labeled samples are often utilized for model training to achieve satisfactory results, but the collection of numerous labeled samples is expensive and time-consuming.

Inputting superpixels into the classifier rather than pixels greatly lessens the need for labeled samples (Gao et al. 2022; Liu et al. 2018b; Xu et al. 2019). As a consequence, a large number of HSI classification methods have been developed based on superpixel segmentation techniques to enhance classification efficiency, and have achieved competitive results (Achanta et al. 2012; Alkhatib and Velez-Reyes 2019; Liu et al. 2011; Zhang et al. 2019). In (Zhang, Su, and Shen 2019), combining multi-scale segmentation with classical classifiers made it possible to acquire separable, compact, anti-noise superpixels. Moreover, superpixel-level methods can exploit both the spatial features and spectral information of HSIs to achieve better results than pixel-level methods against complex background surfaces (Farooq et al. 2019; Xie et al. 2020; Zhao, Su, and Yan 2020). Many studies have shown that the rational use of superpixel techniques accelerates the convergence of deep networks and improves the generalizability of models (Jia et al. 2021; Liu et al. 2017; Mei et al. 2019a).

As described in previous work, the superpixel pooling convolutional neural network (SP-CNN) represents a breakthrough in the finite sample problem, as it combines downsampling and upsampling to obtain and recover spatial information. However, due to the instability of the model itself, SP-CNN is not suitable for regions with large-scale and complex backgrounds. Moreover, the hyperparameters of these networks are overdependent on transfer learning. In summary, this method addresses the issue of obtaining high-accuracy conclusions with limited samples despite the poor robustness of the deep network framework when learning from a small number of spatial-spectral features. Non-local superpixels are exploited in the pre-processing stage of the GCN. Initially, target pixels are updated using all pixel information in each superpixel. A second fractional update for each superpixel is performed after obtaining relatively uniform patches. In addition, based on a previous theoretical comparison of the CNN and GCN, the proposed network framework (Xie et al. 2021) is adjusted to combine the advantages of both network types. SP-CNN is used to complement missing spatial structure information. In the graph structure of the non-local superpixel GCN, the spectral node features in the non-Euclidean domain are fully extracted. Superpixel-level classification significantly reduces the requirement for a large number of sample numbers. Therefore, the proposed method can accurately classify large-scale HSIs with complex backgrounds.

• An advanced end-to-end dual-path deep network is proposed to learn small-scale features of both neatly arranged regions and large-scale irregular regions simultaneously.

• A novel segment denoising strategy is proposed for global smoothing inside each superpixel to obtain uniform patches, namely the non-local segment method.

• To improve the generalization ability for large-scale data, three federated classification strategies are proposed. The strategies of integrating spectral information and spatial context features are selected for different data.

2. Methodology

The deep-learning-based HSI classification framework proposed in this study consists of a superpixel pooling CNN, a GCN with superpixels as vertices, segment anything model, spectral attention models, spectral information fusion, and a fully connected network, as shown in Figure 1. Spectral information from the HSI, extracted through the federated network composed of a CNN and GCN, was input into the classifier by using different spectral fusion techniques, such as superpixel superposition, superpixel multiplication, and superpixel splicing. We used superpixels instead of pixels as vertices to improve the GCN. Furthermore, using superpixels as the final output of the federated network reduced the number of samples that needed to be classified, thereby greatly reducing the classification time.

Figure 1. Schematic of a double convolution federated network.

2.1. Superpixel pooling convolutional neural network

Data collected through remote sensing usually contains large amounts of unavoidable noise and shadow. Visual interpretation and traditional preprocessing methods cannot easily eliminate such interference, which reduces the purity of the data. Interestingly, the pivotal spectral features of the target are extracted through the convolution and pooling techniques in the CNN framework. The downsampling technique is useful for classification tasks. The outstanding performance of CNN in terms of HSI classification can be attributed to its multiple convolution and pooling steps. However, these processes do not solve the issue of complex background, as an HSI generally contains many mixed pixels. In addition, spatial structure information for the HSI is gradually lost when the number of network layers is increased excessively. In our experience, simple application of spectral information cannot support effective learning of complex samples, and the classification results obtained from the network are not ideal. To solve this issue, deconvolution and up-pooling are used to supplement the lost spatial information, followed by a search to obtain the missing details.

The output matrix of the convolution layer is transposed during downsampling, thereby reversing the recorded data. According to characteristic mapping and the size of the convolution kernel, deconvolution results are easily obtained. We expand the input and output matrices into column vectors X and Y, respectively. For a given convolution kernel K, the sparse matrix C can be derived, such that. (1) $\mathrm{Y}=\mathrm{CX}$(1) Through inversion of equation (1(1) $\mathrm{Y}=\mathrm{CX}$(1) ), the following deconvolution operation can be obtained: (2) $\mathrm{X}={\mathrm{C}}^{\mathrm{T}}\mathrm{Y}$(2) The size of the input matrix X is recovered by the deconvolution operation, but spectral information is lost. Traditional resampling techniques and linear interpolation methods can over-smooth the boundary information of the class. The proposed up-pooling procedure provides a solution to this problem. Unpooling is the reverse of the max pooling operation. Unpooling places 0 values at all positions in the unpooling layers except for the maximum feature. Thus, spatial structure information of the maximum feature is well preserved.

After the convolution and pooling operations, the CNN inputs the obtained one-dimensional vector into a fully connected layer to complete the activation of feature information learning for classification of each neural unit. However, the fully connected layer is classified at the pixel level. Therefore, a large number of labeled samples are needed to supply classifiers for training, but the acquisition of labeled samples is generally time-consuming and expensive. In addition, after continuous downsampling, spatial shape information for the target category is gradually lost. Moreover, the mixed geographic ontology undoubtedly reduces classification accuracy. To address these issues, we employ a superpixel pooling layer before the classifier. For this process, principal component analysis (PCA) and Convolution kernel (1 × 1) are used to reduce the dimensionality of the HSI. After parametric experiments, we reduced the dimension of Salinas dataset, WHU-Hi-HanChuan dataset, WHU-Hi-HongHu dataset, and WHU-Hi-LongKou dataset to 30, 50, 50, and 50 respectively, and fine-tune the data dimensions by 1 × 1 convolution kernel to anastomose the data output from GCN stream. Afterward, the HSI is segmented using entropy rate superpixel (ERS) and segment anything model (SAM), which were selected because of their fast implementation and effective recognition of edge information. According to the first law of geography, pixels within the same superpixel have a high probability of belonging to the same category; that is, homogeneity of the segmented superpixel is explainable. Therefore, it is reasonable to input each superpixel block as a single unit into the classifier. This process greatly reduces the number of samples in the input classifier. Moreover, the adaptation and texture features of the superpixel help highlight the spatial structure information of the HSI. The output feature map $\mathbb{Z}\in {\mathbb{R}}^{W\times H\times C}$ is the input superpixel pooling layer, where W represents the width of the HSI, H is the height of the HSI and C is the number of channels. An HSI with a size of W × H is divided into M superpixels, and each superpixel $S_i$ contains $n_i$ pixels. The output of the superpixel pooling layer is a matrix. The pooling function is defined as: (3) $f:S_i\to a_i\in {\mathbb{R}}^C,i=1,2,\dots M$(3) where $a_{i,j}$ is the arithmetic mean of the mean and median values of the j-th channel of $n_i$ pixels in superpixel $S_i$ (Gao et al. 2022). In Eq. (3(3) $f:S_i\to a_i\in {\mathbb{R}}^C,i=1,2,\dots M$(3) ), the median vector is used to reduce the effect of singular values on this function. Introducing the superpixel pooling layer into the CNN can effectively fuze key spectral features and spatial structure information obtained in the upsampling phase. This process also retains important spatial information from the original HSI, which is of great significance for identifying complex background information. This layer reduces the requirement for samples, essentially increasing the proportion of samples used. In other words, the adoption of superpixel pooling weakens the dependence of our proposed method on massive training sample sets, allowing this method to perform optimally with limited samples.

Table 1 lists the parameters setting in the CNN path, including the size and the number of convolutional kernels, stride, and output size of the feature map.

Table 1. Parameters setting in convolutional neural network path of SSC-SFN.

Layer	Kernel Size	Number of Kernels	Padding	Stride	Output Size
Conv_1	1 × 1 × 50	50	0	1	512 × 217 × 50
Attention_1					512 × 217 × 50
Conv_2	3 × 3 × 50	30	0	1	510 × 215 × 30
Conv_3	3 × 3 × 30	30	0	2	254 × 107 × 30
Max Pooling_1	2 × 2	–	–	2	127 × 53 × 30
Attention_2					127 × 53 × 30
Conv_4	3 × 3 × 30	40	1	1	127 × 53 × 40
Conv_5	3 × 3 × 40	40	1	2	64 × 27 × 40
Max Pooling_2	2 × 2	–	–	2	32 × 13 × 40
Attention_3					32 × 13 × 40
Conv_6	3 × 3 × 40	50	1	1	32 × 13 × 50
Conv_7	3 × 3 × 50	50	1	2	16 × 7 × 50
Max Pooling_3	2 × 2	–	–	2	8 × 3 × 50

2.2. Spectral attention mechanism

The attention mechanism can focus on key information while ignoring minor information. When classifying images, the attention mechanism is fully compatible with the convolution process, and the highest-priority features are extracted in the form of an encoder-decoder. The attention mechanism can effectively select information that has already been obtained, thereby simplifying the complex mixture of background features. In this study, the SP-CNN is modified through the addition of a spectral attention module to enhance the interdependency between the feature map and spatial background information. Through the information extraction process of the attention module, the convolutional kernel can update the weight. Therefore, spectral-spatial features in feature maps are emphasized, which improves the accuracy of the classification framework. The spectral attention module maintains the dimensionality of the data, and the size of the output remains the same as the input vector. Given the input tensor $X\in {\mathbb{R}}^{W\times H\times C}$, where W, H, and C represent the width, height, and channel of the HSI, respectively, the global context embedding vector Q is denoted as $Q\in {\mathbb{R}}^{1\times 1\times C}$, where. (4) $Q(k,:,:)={\displaystyle \frac{1}{H\cdot W}\sum _{i=1}^H\sum _{j=1}^{W}X(k,i,j)}$(4) and k is the channel index of the HSI. The attention mechanism is used to establish the nonlinear relationship between network layers and improve the correlation for feature mapping. Then, the coefficient matrix T is calculated as follows: (5) $T=\mathrm{ReLU}(W_2\mathrm{ReLU}(W_{1}Q))$(5) where $W_1\in {\mathbb{R}}^{{\displaystyle \frac{C}{\omega }\times C}}$ and $W_2\in {\mathbb{R}}^{C\times {\displaystyle \frac{C}{\omega }}}$. The reduction ratio $\omega$ is introduced to balance the ability of the spectral attention model and computational expense of the framework. Experiments demonstrated that $\omega$ = 16 works well in practice. Finally, the output $Y\in {\mathbb{R}}^{W\times H\times C}$ of the spectral attention model can be recorded as: (6) $Y(k,i,j)=T(k,:,:)\cdot X(k,i,j)$(6)

2.3. The GCN

The GCN extracts high-level features from constructed undirected graphs based on the correlations between vertices. Unlike a traditional CNN that only performs well with regular grid data, a GCN can handle data with an arbitrary non-Euclidean structure. However, in a traditional GCN (Kipf and Welling 2017), using pixels as vertices often leads to a large graph. This not only increases the computational burden, but also tends to yield unsatisfactory classification results that do not consider the spatial information used for constructing the graph. To address this issue, in this study, we used superpixels rather than pixels as vertices in the constructed graph. Both the spatial structure information provided by the superpixels and the spectral information of the pixels contained therein were considered during graph construction.

For a given high-dimensional HSI, we used PCA (Wold, Esbensen, and Geladi 1987) to reduce its dimensionality, and the first principal component to generate a base image. This setup has been used for many experiments (Li, Wu, and Zhang 2017; Makantasis et al. 2015; Mei et al. 2019b; Zhong et al. 2018b). Subsequently, the base image was segmented into superpixels using the ERS and SAM. Considering each superpixel as a vertex, we linked two vertices using a radial basis function: (7) $a_{i,j}=\{\begin{array}{c}exp\left(-{\displaystyle \frac{{({\overline{x}}_i-{\overline{x}}_j)}^2}{{\sigma }^2}}\right),if{S}_i\in N(S_j)or{S}_j\in N(S_i)\\ 0,otherwise\end{array}$(7) where ${\overline{\mathrm{x}}}_{\mathrm{i}}$ is the average of ${\mathrm{n}}_{\mathrm{i}}$ pixels in superpixels${\mathrm{S}}_{\mathrm{i}}$, $\mathrm{N}({\mathrm{S}}_{\mathrm{i}})$denotes the set of spatial neighbors of ${\mathrm{S}}_{\mathrm{i}}$. The kernel parameter σ is set to 0.2 in our experiments.

Let G = (V, E) be the constructed undirected graph, where V and E are the sets of vertices (superpixels) and edges, respectively. Assuming that A is the adjacency matrix of G, and D is the degree matrix of A, the Laplacian matrix L can be expressed as: (8) $L=D-A$(8) To make the matrix symmetric and improve the generalization of the graph structure, a symmetric normalized Laplacian matrix (L_sym) was created as follows: (9) $L_{sym}=D^{-{\displaystyle \frac{1}{2}}}L{D}^{-{\displaystyle \frac{1}{2}}}=I-D^{-{\displaystyle \frac{1}{2}}}A{D}^{-{\displaystyle \frac{1}{2}}}$(9) where I is the identity matrix with the correct size. On this basis, for the graph convolution to act on the spectral domain, it needs to be Fourier-transformed. Then we have: (10) $\mathrm{L}=\mathrm{U}\mathrm{\Lambda }{\mathrm{U}}^{-1}$(10) where U is the set of eigenvectors of L, which is the basis of the Fourier transform, and Λ represents the diagonal matrix composed of the eigenvalues of L. To realize spectral filtering on the graph, a signal processing transformation was conducted to extract the spectral features. This was done to map information to the spectral domain for processing. The convolution process of the two given functions, f and g, can be written as: (11) $\mathrm{F}[\mathrm{f}(\mathrm{t})\ast \mathrm{g}(\mathrm{t})]=\mathrm{F}[\mathrm{f}(\mathrm{t})]\cdot \mathrm{F}[\mathrm{g}(\mathrm{t})]$(11) where $\ast$ and · denote the convolution operator and node-wise multiplication, respectively. Because U is a symmetric matrix, i.e. UU^T  = E, we can obtain: (12) $\mathrm{L}=\mathrm{U}\mathrm{\Lambda }{\mathrm{U}}^{-1}=\mathrm{U}\mathrm{\Lambda }{\mathrm{U}}^{\mathrm{T}}$(12) According to Eq. (12(12) $\mathrm{L}=\mathrm{U}\mathrm{\Lambda }{\mathrm{U}}^{-1}=\mathrm{U}\mathrm{\Lambda }{\mathrm{U}}^{\mathrm{T}}$(12) ), the Fourier transform F of the function f on the graph G can be written as: (13) $\mathrm{GF}[\mathrm{f}]={\mathrm{U}}^{\mathrm{T}}\mathrm{f}$(13) (14) $\mathrm{f}=\mathrm{GF}[\mathrm{f}]\mathrm{U}$(14) Therefore, the convolution between f and g on a graph can be expressed as: (15) $\mathrm{G}[\mathrm{f}\ast \mathrm{g}]=\mathrm{U}\{[{\mathrm{U}}^{\mathrm{T}}\mathrm{f}]\cdot [{\mathrm{U}}^{\mathrm{T}}\mathrm{g}]\}$(15) To reduce the computational cost of feature decomposition, Hammond et al. (Hammond, Vandergheynst, and Gribonval 2011) approximately fitted g_θ(Λ) by introducing the K-th order truncated expansion of Chebyshev polynomials. This can be formulated as: (16) $\mathrm{G}[\mathrm{f}\ast {\mathrm{g}}_{\theta }]\approx \sum _{\mathrm{k}=0}^{\mathrm{K}}{\theta }_{\mathrm{k}}{\mathrm{T}}_{\mathrm{k}}(\tilde{\mathrm{L}})\mathrm{f}$(16) where θ_k and T_k are the Chebyshev coefficients and variable Λ of the Chebyshev polynomials, respectively. The normalized L can be expressed as$\tilde{\mathrm{L}}={\displaystyle \frac{2}{{\lambda }_{\max }}{\mathrm{L}}_{\mathrm{sym}}-\mathrm{I}}$. When K = 1 and the largest eigenvalue is 2, the convolution operation of GCN can be expressed as: (17) ${\mathrm{H}}^{(\mathrm{l}+1)}=\rho \left({\tilde{\mathrm{D}}}^{-{\displaystyle \frac{1}{2}}}\tilde{\mathrm{A}}{\tilde{\mathrm{D}}}^{-{\displaystyle \frac{1}{2}}}{\mathrm{H}}^{(\mathrm{l})}{\mathrm{W}}^{(\mathrm{l})}+{\mathrm{b}}^{(\mathrm{l})}\right)$(17) where H^(l + 1) denotes the output of the l-th layer and ρ (·) is the activation function (e.g. a rectified linear activation function (ReLU) or Softplus function, which are applied in our proposed method). $\tilde{\mathrm{A}}=\mathrm{A}+\mathrm{I}$ and ${\tilde{\mathrm{D}}}_{\mathrm{i},\mathrm{i}}=\sum _{\mathrm{j}}{\tilde{\mathrm{A}}}_{\mathrm{i},\mathrm{j}}$are represent the renormalization of the adjacency matrix A and degree matrix D, respectively, while W and b are the weights and biases obtained by training each model layer.

2.4. Non-local superpixel smoothing

The commonly used GCN-based HSI classification method feeds pixel-level samples into the network, establishes a relationship between the adjacency matrix and samples, and converts the non-Euclidean features extracted by the network layer into a one-dimensional vector. The fully connected neural network was used to classify the resulting vectors at the pixel level. Compared with CNNs, GCNs can effectively share information globally, and are favorable for acquiring and describing long-distance spatial relationships. However, the quality of the collected information is greatly degraded by the noisy data in the HSI. Moreover, the computational cost of classification is not only related to the sample size but also depends on the network depth and input dimensions. The GCN needed to input a large feature matrix and adjacency matrix through a complex graph structure. Therefore, images containing a large amount of data will undoubtedly increase the computational complexity of the algorithm. In addition, because the GCN only allows input of the full batch of feature matrices, all pixel-level samples must be trained simultaneously. If the input image representation is inaccurate, the final classification performance will be degraded. To lessen interference from noise, reduce the computational cost, improve classification performance, and fully learn from the correlations of nearest neighbor elements, we designed a non-local superpixel smoothing strategy, as shown in Figure 2.

Figure 2. Illustration of the non-local smoothing process used in our model.

We segmented the HSI using the SAM combined with a GCN, because this can quickly retain edge pixels. Compared with the simple linear iterative clustering (SLIC) algorithm, the SAM algorithm is more suitable for processing image attributes, and the corresponding image size is more recognizable; this improves the quality of graph convolution. To obtain accurate category edges and refine the segmentation, we first performed a PCA to reduce the dimensionality of high-dimensional data. The most basic preprocessed data were used for the first principal component, which contained the most information. Superpixel technology is based on the first law of geography, i.e. neighboring pixels have a higher probability of belonging to the same category, which is known as the principle of homogeneity of superpixels. Therefore, it is rational to consider the pixels inside a superpixel as a whole during the classification procedure.

Suppose that x is the input feature value, i is the positional information of the input center node, and j is the position of all the other nodes in the same superpixel block. The non-local function is then defined as: (18) ${\mathrm{y}}_{\mathrm{i}}={\displaystyle \frac{1}{\mathrm{N}(\mathrm{x})}\sum _{\mathrm{\forall }\mathrm{j}}\mathrm{f}({\mathrm{x}}_{\mathrm{i}}{\mathrm{x}}_{\mathrm{j}})\mathrm{h}({\mathrm{x}}_{\mathrm{j}})}$(18) where f is a function for calculating the relationship between x_i and x_j, h is a unary function representing the input feature at position j, y is the output eigenvalue of the same size as x, and N(·) is a normalization function. The non-local operation used all of the pixel information in equation (18(18) ${\mathrm{y}}_{\mathrm{i}}={\displaystyle \frac{1}{\mathrm{N}(\mathrm{x})}\sum _{\mathrm{\forall }\mathrm{j}}\mathrm{f}({\mathrm{x}}_{\mathrm{i}}{\mathrm{x}}_{\mathrm{j}})\mathrm{h}({\mathrm{x}}_{\mathrm{j}})}$(18) ), with the corresponding weights assigned according to the neighborhood distances, followed by cyclical calculations.

Unlike the superpixel smoothing methods that are currently used, the non-local operation considers all the pixels inside the superpixel. Specifically, the traditional superpixel smoothing technique selects the nearest neighbor or several neighborhood layers for the iterative update process, i.e. multi-scale superpixel smoothing, which undoubtedly adds a large number of parameters and hinders the identification of the optimal solution. However, the proposed non-local superpixel method treats all nodes within a given superpixel as relational nodes of the target pixel, thereby reducing the number of parameters and minimizing the process of parameter training. In addition, a GCN needs a ‘global node relationship matrix’ that can be retained during non-local operations, thus enabling the classification network to better learn weights. It should be noted that, with this method, the edge weights of the graph are automatically learned.

2.5. Federated classification strategy

Several approaches have been developed to resolve issues caused by multi-source data interaction. However, matching the data to achieve interoperability is worthy of further consideration. When dealing with diverse structural information, multiple models need to be combined to improve classification performance. For example, a CNN can extract the spectral-spatial characteristics of an HSI, whereas a GCN focuses on the topological relationship between pixels. In general, stacking multiple models is more effective than using a single frame. Federated learning (FL) allows data fusion at target nodes, which connects key features and improves network channels (Gaba et al. 2022; Nguyen and Zettsu 2021). However, reliable mechanisms must be established for sharing information updates. Several studies have demonstrated that FL enables joint training of multiple DL frameworks to integrate latent information from multiple networks.

Combining a CNN and GCN is a highly feasible approach. A non-local GCN was used at the superpixel level to improve the simplified SP-CNN. FL of the sample features of the two DL frameworks was realized. Three federated classification strategies were used for the SSC-SFN: SSC-SFN superposition (SSC-SFN-P), SSC-SFN multiplication (SSC-SFN-M), and SSC-SFN splicing (SSC-SFN-S). These strategies were implemented at the superpixel level, the l-th layer features of the CNN and GCN were extracted, and the fully connected layer was then used for classification. This can be expressed as follows: (19) ${\mathrm{H}}_{\mathrm{P}}^{\mathrm{l}}=\mathrm{P}({\mathrm{H}}_{\mathrm{CNN}}^{(\mathrm{l}-1)},{\mathrm{H}}_{\mathrm{GCN}}^{(\mathrm{l}-1)})$(19) (20) ${\mathrm{H}}_{\mathrm{M}}^{\mathrm{l}}=\mathrm{M}({\mathrm{H}}_{\mathrm{CNN}}^{(\mathrm{l}-1)},{\mathrm{H}}_{\mathrm{GCN}}^{(\mathrm{l}-1)})$(20) (21) ${\mathrm{H}}_{\mathrm{S}}^{\mathrm{l}}=\mathrm{S}({\mathrm{H}}_{\mathrm{CNN}}^{(\mathrm{l}-1)},{\mathrm{H}}_{\mathrm{GCN}}^{(\mathrm{l}-1)})$(21) where P (·), M (·), and S (·) denote the superposition, multiplication, and splicing functions, respectively. ${\mathrm{H}}_{\mathrm{CNN}}^{(\mathrm{l}-1)}$ and ${\mathrm{H}}_{\mathrm{GCN}}^{(\mathrm{l}-1)}$ represent the feature matrix extracted by the l-1 layer CNN and GCN, respectively. ${\mathrm{H}}_{\mathrm{P}}^{\mathrm{l}}$, ${\mathrm{H}}_{\mathrm{M}}^{\mathrm{l}}$, and ${\mathrm{H}}_{\mathrm{S}}^{\mathrm{l}}$are the results obtained using the three federated strategies. The function P (·) represents the sum of corresponding elements in the matrix. The function M (·) refers to matrix multiplication; we calculated the Hadamard product of the matrix in this step. Function M (·) can be expressed as follows: (22) $\mathrm{A}\ast \mathrm{B}=[a_{i,j}]\ast [b_{i,j}]$(22) where the size of matrix A = [a_i,j] and matrix B = [b_i,j] must be consistent; its elements are defined as the products of the elements corresponding to the two matrices, i.e. $(\mathrm{A}\ast \mathrm{B}{)}_{i,j}=a_{i,j}{b}_{i,j}$. For function S (·), we utilized the splicing function in Numerical Python, but changed the matrix size. The proposed solution maps the assembled matrix compression into the distance space and approximates a unique solution step-by-step, yielding a feature matrix of the same size as the input matrix M. Function S (·) can be expressed as follows: (23) $M=[a_{n\times n},b_{n\times n}{]}_{n\times 2n},\{M_{\mathrm{n}}{\}}_{\mathrm{n}=0}^{+\mathrm{\infty }}$(23)

3. Results

To confirm the effectiveness of the proposed SSC-SFN method, a large number of experiments were conducted on four generic hyperspectral datasets: Salinas, WHU-Hi-HanChuan, and WHU-Hi-HongHu. These four datasets were used as benchmarks to assess the classification performance of our proposed algorithms, both quantitatively and qualitatively.

3.1. Datasets and evaluation indicators

The Salinas dataset was collected by the airborne visible/infrared imaging spectrometer (AVIRIS) sensor from Salinas Valley (California, USA). As is shown in Figure 3, it is composed of 16 categories of 512 × 217-pixel images, with 204 bands and a spatial resolution of 3.7 m. In the image analyzed herein, there are two spatially adjacent classes, ‘Grapes_U’ and ‘Vineyard_U’ with very similar spectra.

Figure 3. Salinas dataset. (a) False color image. (b) Ground-truth.

The WHU-Hi-HanChuan dataset (Zhong et al. 2018a; Zhong et al. 2020) was collected from Hanchuan, Hubei Province, China, by a Headwall Nano-Hyperspec imaging sensor (8-mm focal length) equipped on the Aibot X6 UAV V1 platform (Leica, Wetzlar, Germany). As is shown in Figure 4, the study area contained seven crop species: strawberry, cowpea, soybean, sorghum, water spinach, watermelon, and greens. This dataset comprised 1,217 × 303-pixel images and 274 bands (400-1,000 nm), with a spatial resolution of about 0.109 m. Because the WHU-Hi-HanChuan dataset was collected in the afternoon when the sun was low, many areas were covered by shadow, which affected the classification.

Figure 4. WHU-Hi-HanChuan dataset. (a) False color image. (b) Ground-truth.

The third dataset used was the WHU-Hi-HongHu dataset (Zhong et al. 2018a; Zhong et al. 2020). As is shown in Figure 5, this dataset was acquired with a 17-mm focal length Headwall Nano-Hyperspec imaging sensor in Honghu City, Hubei Province, China. The research area was typical of a region affected by land fragmentation, and contained 17 crop types including cotton, rape, and cabbage. This dataset comprised 940 × 475-pixel images with a spatial resolution of about 0.043 m and 270 bands (400-1,000 nm). Different varieties of the same crop type are grown in this district, for example, Chinese cabbage/cabbage and Brassica oleracea/small brassica.

Figure 5. WHU-Hi-HongHu dataset. (a) False color image. (b) Ground-truth.

The WHU-Hi-LongKou dataset was collected from Longkou Town, Hubei Province, China (Zhong et al. 2018a; Zhong et al. 2020). The dataset is based on an 8-mm focal length Headwall Nano-Hyperspec imaging sensor equipped on a DJI Matrice 600 Pro UAV platform. As is shown in Figure 6, the study area includes six crops: corn, cotton, sesame, broad-leaf soybean, narrow-leaf soybean, and rice. The size of the dataset is 550 × 400 pixels, there are 270 bands from 400 to 1,000 nm, and the spatial resolution of the hyperspectral images carried by the drone is about 0.463 m.

Figure 6. WHU-Hi-LongKou dataset. (a) False color image. (b) Ground-truth.

In all experiments conducted in this study, the classification results were evaluated by adopting three commonly used indices, i.e. overall accuracy (OA), average accuracy (AA), and the kappa coefficient (κ). To overcome the examination bias caused by random marking, the mean and standard deviation of 10 independent runs were calculated as the final classification results.

3.2. Classification results and analysis

Several state-of-the-art DL classification methods were compared, i.e. MiniGCN (Hong et al. 2021), FuNet-C (Hong et al. 2021), MDGCN (Wan et al. 2020), SSOGCN (Zhang et al. 2021), CNNCRF (Zhong et al. 2020), FPGA (Zheng et al. 2020), and the SSC-SFN-M. All of these methods are based on deep networks and use fully connected networks as the classifier. For all baseline methods, the hyperparameters recommended in the papers that originally described them were adopted. Tables 2–5 compare the results obtained by applying these algorithms to the four datasets.

Table 2. Classification accuracy (%) achieved by applying different methods to the Salinas dataset.

Class	Train/Test	MiniGCN	FuNet-C	MDGCN	SSOGCN	CNNCRF	CEGCN	FPGA	SSC-SFN-M
Weeds_1	30/1979	88.85 ± 0.12	94.85 ± 0.35	94.43 ± 0.87	95.62 ± 1.75	100.0	97.92 ± 0.02	100.0	100.0
Weeds_2	30/3696	82.09 ± 1.24	97.68 ± 0.06	99.87 ± 0.01	97.78 ± 0.91	97.89 ± 0.02	99.89 ± 0.02	99.94 ± 0.02	99.78 ± 0.05
Fallow	30/1946	96.84 ± 0.75	96.79 ± 0.03	89.62 ± 3.75	95.92 ± 1.22	100.0	93.88 ± 0.02	100.0	100.0
Fallow_P	30/1364	90.42 ± 0.42	89.75 ± 3.59	98.27 ± 0.62	98.59 ± 0.43	99.89 ± 0.01	99.89 ± 0.01	100.0	100.0
Fallow_S	30/2648	93.58 ± 0.02	96.45 ± 1.32	93.51 ± 0.53	97.31 ± 1.52	99.45 ± 0.22	98.45 ± 0.02	99.11 ± 0.04	99.41 ± 0.06
Stubble	30/3929	92.11 ± 0.88	99.55 ± 0.11	99.43 ± 0.16	97.56 ± 0.21	89.56 ± 0.02	99.56 ± 0.02	99.61 ± 0.20	100.0
Celery	30/3549	94.35 ± 0.19	99.55 ± 0.10	95.15 ± 0.37	99.23 ± 0.04	99.26 ± 0.02	99.26 ± 0.02	99.34 ± 0.09	100.0
Grapes_U	30/11241	65.48 ± 3.21	75.36 ± 9.22	57.30 ± 2.45	74.85 ± 6.23	98.63 ± 0.14	85.63 ± 0.14	99.04 ± 0.07	99.57 ± 0.06
Soil	30/6173	81.31 ± 0.66	95.85 ± 0.98	94.97 ± 1.83	98.38 ± 0.08	99.53 ± 0.02	96.53 ± 0.02	100.0	99.69 ± 0.22
Corn	30/3248	95.52 ± 0.40	90.21 ± 1.75	86.04 ± 2.94	89.66 ± 3.11	99.64 ± 0.04	97.64 ± 0.04	97.96 ± 0.08	99.76 ± 0.25
Lettuce_4wk	30/1038	79.20 ± 0.13	97.74 ± 6.46	91.81 ± 1.74	93.76 ± 1.48	99.22 ± 0.06	99.22 ± 0.06	100.0	100.0
Lettuce_5wk	30/1897	83.94 ± 0.03	99.94 ± 0.06	92.21 ± 1.04	97.06 ± 1.14	89.35 ± 2.36	96.37 ± 0.37	99.94 ± 0.03	100.0
Lettuce_6wk	30/886	88.25 ± 0.55	98.31 ± 0.06	98.05 ± 0.44	96.69 ± 0.16	94.22 ± 0.06	95.84 ± 0.25	99.85 ± 0.01	99.19 ± 0.41
Lettuce_7wk	30/1040	82.23 ± 1.62	93.65 ± 1.22	90.33 ± 0.82	91.39 ± 1.43	99.22 ± 0.06	94.19 ± 0.15	99.59 ± 0.22	99.92 ± 0.06
Vineyard_U	30/7238	75.93 ± 2.46	69.50 ± 0.06	66.68 ± 2.85	66.75 ± 9.49	90.36 ± 2.10	87.36 ± 2.10	99.12 ± 0.16	99.39 ± 0.08
Vineyard_T	30/1777	86.68 ± 4.35	97.75 ± 9.49	93.71 ± 0.04	95.34 ± 3.77	99.49 ± 0.02	99.49 ± 0.02	100.0	99.83 ± 0.04
OA		85.42 ± 1.74	87.36 ± 2.17	89.94 ± 1.49	92.17 ± 1.27	96.59 ± 0.61	97.62 ± 0.16	99.11 ± 0.04	99.48 ± 0.23
AA		86.36 ± 1.25	88.04 ± 1.56	91.05 ± 0.62	94.05 ± 0.87	96.97 ± 1.15	96.17 ± 1.15	99.68 ± 0.19	99.79 ± 0.09
Kappa		84.57 ± 1.50	86.49 ± 0.99	88.24 ± 1.74	91.46 ± 1.72	95.06 ± 0.22	96.06 ± 0.22	98.69 ± 0.13	99.04 ± 0.30

Table 3. Classification accuracy (%) achieved by applying different methods to the WHU-HI-HanChuan dataset.

Class	Train/Test	MiniGCN	FuNet-C	MDGCN	SSOGCN	CNNCRF	CEGCN	FPGA	SSC-SFN-M
Strawberry	30/44705	72.95 ± 3.42	83.51 ± 3.93	78.57 ± 8.62	74.55 ± 3.47	88.57 ± 0.62	84.78 ± 1.25	88.19 ± 1.45	93.51 ± 0.93
Cowpea	30/22723	71.16 ± 6.62	68.01 ± 8.25	68.73 ± 2.71	67.86 ± 0.61	68.73 ± 2.31	56.98 ± 0.95	85.38 ± 2.89	88.01 ± 0.40
Soybean	30/10257	68.77 ± 0.86	91.80 ± 2.57	90.52 ± 1.82	85.75 ± 0.89	91.50 ± 0.82	81.77 ± 0.83	91.94 ± 0.41	91.80 ± 0.07
Sorghum	30/5323	85.90 ± 0.17	92.02 ± 1.45	93.46 ± 0.48	86.83 ± 0.12	97.46 ± 0.38	95.49 ± 0.13	96.80 ± 0.63	96.52 ± 0.45
Water spinach	30/1170	92.56 ± 1.41	83.33 ± 4.52	93.48 ± 0.87	59.91 ± 2.45	99.48 ± 0.17	97.86 ± 0.32	92.73 ± 0.05	93.33 ± 0.52
Watermelon	30/4503	69.97 ± 5.56	86.76 ± 1.73	77.35 ± 4.71	57.35 ± 8.25	76.35 ± 0.91	63.60 ± 5.02	82.14 ± 0.71	85.46 ± 0.73
Greens	30/5873	90.85 ± 0.68	70.54 ± 3.49	73.12 ± 6.11	78.05 ± 2.21	93.12 ± 0.11	83.85 ± 0.32	88.63 ± 0.10	70.54 ± 3.49
Trees	30/17948	49.25 ± 8.83	52.62 ± 1.27	58.21 ± 3.99	67.79 ± 4.54	71.91 ± 0.89	67.27 ± 0.88	76.40 ± 0.92	77.62 ± 1.14
Grass	30/9439	65.19 ± 2.34	67.96 ± 8.61	72.91 ± 5.13	74.56 ± 5.80	67.31 ± 5.83	67.19 ± 2.33	93.22 ± 0.82	87.96 ± 4.61
Red roof	30/10486	87.64 ± 4.86	64.37 ± 12.41	94.44 ± 1.75	98.49 ± 0.32	78.44 ± 0.75	92.16 ± 0.16	75.60 ± 2.48	86.87 ± 4.01
Gray roof	30/16881	95.58 ± 1.68	95.03 ± 1.42	92.68 ± 0.89	95.69 ± 1.96	96.68 ± 0.27	85.58 ± 0.14	94.85 ± 0.27	98.03 ± 0.82
Plastic	30/3649	94.88 ± 2.64	83.92 ± 1.27	86.92 ± 0.59	85.11 ± 6.83	86.92 ± 0.79	87.88 ± 0.71	83.63 ± 0.92	94.90 ± 0.27
Bare soil	30/9086	69.31 ± 0.78	72.60 ± 4.23	54.87 ± 1.91	66.05 ± 1.39	74.27 ± 1.91	59.31 ± 0.32	63.46 ± 0.39	67.50 ± 0.83
Road	30/18630	86.21 ± 1.88	74.47 ± 6.29	85.73 ± 2.53	71.75 ± 11.93	55.73 ± 3.66	71.21 ± 0.42	86.20 ± 5.62	94.47 ± 6.29
Bright object	30/1106	78.18 ± 1.24	63.60 ± 9.53	64.10 ± 2.09	74.82 ± 7.82	64.10 ± 2.49	65.18 ± 1.29	68.30 ± 7.20	86.30 ± 9.03
Water	30/75371	97.89 ± 0.47	96.06 ± 1.67	90.44 ± 0.35	87.14 ± 1.48	94.44 ± 0.35	89.89 ± 0.40	97.58 ± 0.13	91.06 ± 0.35
OA		74.19 ± 1.43	79.22 ± 2.19	81.32 ± 1.68	83.80 ± 1.29	88.17 ± 1.46	87.59 ± 1.63	90.75 ± 0.39	91.82 ± 0.78
AA		75.37 ± 1.84	81.63 ± 0.53	72.56 ± 0.84	84.88 ± 0.61	89.63 ± 1.73	88.37 ± 0.76	91.45 ± 0.53	92.63 ± 2.40
Kappa		72.90 ± 0.67	78.92 ± 0.08	80.39 ± 0.14	82.39 ± 0.40	88.02 ± 0.34	86.90 ± 1.39	90.22 ± 1.33	90.32 ± 0.58

Table 4. Classification accuracy (%) achieved by applying different methods to the WHU-HI-HongHu dataset.

Class	Train/Test	MiniGCN	FuNet-C	MDGCN	SSOGCN	CNNCRF	CEGCN	FPGA	SSC-SFN-M
Red roof	30/14011	76.10 ± 0.29	87.74 ± 0.21	89.96 ± 1.27	75.38 ± 3.13	80.81 ± 1.51	88.48 ± 2.37	83.46 ± 1.82	99.96 ± 0.01
Road	30/3482	87.68 ± 1.37	89.31 ± 0.25	87.73 ± 0.32	88.90 ± 15.82	76.40 ± 2.79	78.47 ± 6.23	80.44 ± 0.48	86.78 ± 0.92
Bare soil	30/21791	83.04 ± 2.83	81.41 ± 1.11	92.03 ± 0.57	89.67 ± 0.49	77.73 ± 2.85	75.73 ± 2.58	89.35 ± 0.74	97.53 ± 0.57
Cotton	30/163255	95.53 ± 1.05	93.50 ± 0.36	92.94 ± 2.72	93.34 ± 6.97	94.46 ± 0.27	94.43 ± 0.23	97.01 ± 0.35	95.64 ± 1.12
Cotton firewood	30/6188	89.59 ± 0.55	94.77 ± 0.95	92.92 ± 0.68	97.85 ± 0.50	97.06 ± 0.08	92.32 ± 0.47	96.88 ± 0.72	95.34 ± 0.58
Rape	30/44527	93.00 ± 1.51	91.49 ± 0.40	93.60 ± 1.87	88.68 ± 0.61	89.93 ± 0.42	83.57 ± 0.52	93.40 ± 0.33	93.94 ± 0.80
Chinese cabbage	30/24073	43.28 ± 7.23	76.22 ± 3.04	78.75 ± 4.46	67.02 ± 5.49	62.51 ± 4.82	64.61 ± 2.25	80.80 ± 4.90	89.58 ± 2.96
Pakchoi	30/4024	46.60 ± 2.24	58.18 ± 0.86	53.37 ± 2.58	68.18 ± 1.99	73.62 ± 0.68	63.02 ± 0.68	76.58 ± 0.53	73.87 ± 0.18
Cabbage	30/10789	92.46 ± 0.45	94.61 ± 0.42	90.43 ± 0.51	95.34 ± 0.50	89.76 ± 0.73	77.80 ± 4.54	91.97 ± 0.70	92.49 ± 0.24
Tuber mustard	30/12364	74.56 ± 8.84	77.19 ± 1.84	80.46 ± 3.93	67.86 ± 6.34	75.40 ± 0.43	85.38 ± 1.43	83.91 ± 0.96	82.93 ± 1.43
Brassica parachinensis	30/10985	62.05 ± 4.36	88.81 ± 0.92	86.42 ± 0.32	84.13 ± 9.05	94.99 ± 0.32	89.89 ± 1.33	84.74 ± 0.25	87.24 ± 0.38
Brasssica chinensis	30/8924	65.94 ± 0.83	63.26 ± 0.15	76.64 ± 2.73	78.67 ± 4.25	70.36 ± 0.31	75.76 ± 1.31	77.80 ± 3.64	70.04 ± 2.71
Small Brassica chinensis	30/22477	69.15 ± 7.92	64.83 ± 0.19	61.78 ± 0.72	72.38 ± 2.79	81.64 ± 0.92	71.45 ± 2.72	82.53 ± 0.29	84.08 ± 0.42
Lactuca sativa	30/7326	88.67 ± 2.68	86.15 ± 2.03	84.64 ± 0.36	87.76 ± 2.48	88.34 ± 5.82	84.54 ± 7.52	85.09 ± 0.29	95.64 ± 0.84
Celtuce	30/972	87.60 ± 0.08	98.63 ± 0.04	92.38 ± 0.78	86.06 ± 0.85	92.79 ± 0.84	82.76 ± 2.34	95.48 ± 0.38	100.0
Film covered lettuce	30/7232	98.50 ± 0.41	97.62 ± 0.09	93.06 ± 0.49	96.48 ± 0.37	96.33 ± 0.57	92.58 ± 0.92	98.22 ± 0.06	95.60 ± 0.29
Romaine lettuce	30/2980	94.03 ± 0.24	96.99 ± 0.32	95.74 ± 0.56	89.13 ± 1.33	91.81 ± 0.21	91.48 ± 0.48	97.22 ± 0.22	94.49 ± 0.66
Carrot	30/3187	76.35 ± 0.83	92.61 ± 0.21	94.91 ± 0.63	94.60 ± 0.56	91.96 ± 0.74	92.84 ± 0.29	95.80 ± 0.30	100.0
White radish	30/8682	62.49 ± 0.74	71.03 ± 4.29	74.87 ± 0.73	79.36 ± 1.74	77.55 ± 1.04	72.45 ± 1.84	76.96 ± 1.58	78.84 ± 2.36
Garlic sprout	30/3456	97.19 ± 0.05	97.55 ± 0.53	96.67 ± 0.05	81.45 ± 8.36	98.17 ± 0.17	93.06 ± 0.29	98.39 ± 0.95	98.87 ± 0.15
Broad bean	30/1298	84.49 ± 1.26	99.68 ± 0.07	95.84 ± 0.59	95.32 ± 0.41	99.61 ± 0.04	95.61 ± 1.09	97.37 ± 0.65	100.0
Tree	30/4010	92.45 ± 0.41	97.60 ± 0.94	99.47 ± 0.23	96.45 ± 0.46	99.15 ± 0.21	94.65 ± 0.01	91.87 ± 0.97	99.47 ± 0.03
OA		75.66 ± 2.93	80.47 ± 1.38	83.63 ± 1.45	85.22 ± 0.54	86.04 ± 1.35	88.93 ± 1.23	91.39 ± 0.28	92.94 ± 0.71
AA		77.22 ± 3.66	82.95 ± 2.73	85.86 ± 2.25	86.20 ± 1.61	87.15 ± 0.23	89.22 ± 0.96	92.15 ± 0.23	93.60 ± 0.62
Kappa		74.26 ± 1.48	79.37 ± 1.24	82.29 ± 1.14	84.63 ± 0.54	84.77 ± 0.64	87.61 ± 0.48	90.37 ± 0.34	91.45 ± 0.55

Table 5. Classification accuracy (%) achieved by applying different methods to the WHU-HI-LongKou dataset.

Class	Train/Test	MiniGCN	FuNet-C	MDGCN	SSOGCN	CNNCRF	CEGCN	FPGA	SSC-SFN-M
Corn	30/34481	97.92 ± 0.22	94.85 ± 0.35	93.85 ± 0.12	94.43 ± 0.87	94.85 ± 0.35	97.62 ± 7.75	99.92 ± 0.02	95.43 ± 0.84
Cotton	30/8344	98.89 ± 0.12	97.68 ± 0.36	92.09 ± 1.24	99.87 ± 0.01	99.68 ± 0.06	97.78 ± 0.91	99.79 ± 0.01	99.88 ± 0.02
Sesame	30/3001	93.88 ± 0.32	96.79 ± 0.43	91.88 ± 0.75	89.62 ± 3.75	99.79 ± 0.03	95.92 ± 1.22	93.62 ± 1.32	95.89 ± 0.32
Broad-leaf soybean	30/63182	89.89 ± 0.47	76.75 ± 3.59	79.42 ± 2.42	81.27 ± 3.62	83.75 ± 3.59	84.59 ± 2.43	95.89 ± 0.52	96.38 ± 0.24
Narrow-leaf soybean	30/4121	93.45 ± 1.46	96.45 ± 1.32	98.58 ± 0.02	97.51 ± 0.53	96.45 ± 1.32	97.31 ± 1.52	98.45 ± 0.02	98.98 ± 0.04
Rice	30/11824	92.56 ± 1.02	94.55 ± 0.81	98.11 ± 0.88	99.43 ± 0.16	99.55 ± 0.11	97.56 ± 0.21	99.56 ± 0.02	99.61 ± 0.20
Water	30/67026	97.26 ± 0.12	91.55 ± 0.70	94.35 ± 0.19	95.15 ± 0.37	99.55 ± 0.10	99.23 ± 0.04	99.26 ± 0.02	99.55 ± 0.09
Roads and houses	30/7094	65.63 ± 4.74	75.26 ± 9.22	56.48 ± 7.21	67.3 ± 4.45	68.36 ± 9.22	74.85 ± 6.23	78.63 ± 4.24	88.48 ± 1.77
Mixed weed	30/5199	96.53 ± 0.32	95.85 ± 0.98	97.31 ± 0.66	94.97 ± 1.83	95.85 ± 0.98	98.38 ± 0.08	96.53 ± 0.02	99.31 ± 0.06
OA		84.19 ± 1.88	85.16 ± 1.68	86.24 ± 1.32	87.67 ± 0.95	89.78 ± 0.25	92.09 ± 0.84	94.51 ± 0.42	98.52 ± 0.19
AA		85.25 ± 2.86	85.54 ± 2.92	86.43 ± 2.28	88.05 ± 2.67	90.04 ± 1.56	91.25 ± 0.32	94.47 ± 1.15	98.48 ± 0.49
Kappa		83.79 ± 1.52	84.91 ± 1.61	85.57 ± 0.70	86.24 ± 1.78	88.49 ± 0.93	91.84 ± 1.72	94.06 ± 0.22	98.09 ± 0.14

Table 2 shows the results obtained by applying several classification algorithms to the Salinas dataset. According to the information in Table 2, MiniGCN, FuNet-C, MDGCN, SSOGCN, CNNCRF, and CEGCN could not accurately identify the ‘Vineyard_U’ class, while SSC-SFN-M had the advantage of identifying the ‘Grapes_U’ class. Because the spectral curves of ‘Vineyard_U’ and ‘Grapes_U’ were very similar, and they were close spatial neighbors, it was not appropriate to rely on spectral values to classify and capture their boundary information, as shown in Figure 7. Therefore, to ensure excellent classification accuracy for the Salinas dataset, it was necessary to extract spatial information. Our method was capable of this, and the multiplication strategies resulted in classification accuracy of >99.4%.

Figure 7. Classification maps of the eight algorithms on Salinas dataset.

Table 3 compares the results obtained by applying different methods to the WHU-Hi-HanChuan dataset, the best OA, AA, and κ values are in bold. The classification accuracy of the proposed method was 91.82%, which was about 10% higher than that of MDGCN (81.32%). We concluded that the classification accuracy of our proposed method was superior to its competitors. It should be noted that the dataset was affected by light and shade coverage when it was collected, and the spectral values of some categories fluctuate, this makes it difficult to distinguish the ‘soybean’ and ‘water’ categories in particular. However, we could capture changes in the distribution of the shaded regions using the non-local superpixel network, which rendered the classification performance of our proposed method for these categories superior to that of the other algorithms. From Figure 8, it can be seen that there were large errors in the display of partially overlapping objects and isolated areas, except when SSC-SFN-M was used, this may be attributable to the segmentation ability of superpixel.

Figure 8. Classification maps were obtained using eight algorithms with the WHU-Hi-HanChuan dataset.

The classification statistics for several methods applied to the WHU-Hi-HongHu dataset are listed in Table 4. The OA of our method was about 10% higher than that of FuNet-C, which represents an important performance breakthrough and confirms the utility of our framework. The OA of MDGCN, which only used graph convolution to extract spatial-spectral information, was 83.63%. This finding shows the importance of expressing spatial neighborhood relationships. In contrast, CEGCN and FPGA required a large number of labeled samples to learn the network parameters. Serious misclassifications occurred in multiple classes, such as ‘Chinese cabbage’ and ‘tuber mustard’, when the labeled samples were limited. This dataset had 22 categories, and the uneven distribution of samples was one of the major barriers to classification. Our method could not accurately identify and locate the ‘road’ and ‘Brassica chinensis’ categories. This may have been because after the dataset was segmented, the fragmented distribution of the classes led to numerous very small patches, which weakened the influence of the global information regarding the topological structure on the classification. Figure 9 shows that it was difficult to identify criticality, especially in terms of the shapes of the ‘trees’ and ‘Chinese cabbage’ classes in the upper left, and the ‘Brassica chinensis’ class in the lower right. MiniGCN, FuNet-C, SSOGCN, CNNCRF, CEGCN, and SSC-SFN-M did not achieve satisfactory classification results. In summary, the frequency spectrum of these categories was complex and the airspace overlapped. The framework must be improved to enhance the recognition accuracy for these categories.

Figure 9. Classification maps were obtained using eight algorithms with the WHU-Hi-HongHu dataset.

The classification results of the WHU-Hi-LongKou dataset calculated by SSC-SFN and comparison methods are shown in Figure 10, and the specific accuracy is recorded in Table 5. Compared with the other three datasets, this dataset is relatively easy to classify, and the inter-class characteristics are obvious. The OA of MiniGCN reached 84.19%, which may be due to the lack of effective integrated context information, resulting in feature fragmentation. For example, cotton and broad-leaf, corn, and mixed weeds are distinguished with less precision (Figure 10a). The MDGCN classification results identified a small number of broad-leaves as sesame, which resulted in a serious misclassification (Figure 10c). The proposed SSC-SFN-M model has generally obtained high accuracy, with OA reaching 98.52%. SSC-SFN-M gives a good answer to the mixed misclassification of broad-leaf and sesame, showing that the non-local segmentation features and the global context information are accurately fuzed.

Figure 10. Classification maps were obtained using eight algorithms with the WHU-Hi-LongKou dataset.

4. Discussion

In this section, we discuss the influence of the number of training samples on the classification results, the roles of federated network strategies and non-local superpixels, compare the operating times of several algorithms, and consider the impact of key parameters on the efficiency of the proposed method. All experiments were conducted using an Intel i3 9100F CPU, 16 GB memory, and an NVIDIA GEFORCE RTX 2080Ti GPU.

4.1. Impact of various labeled samples on the classification results

We first compared the proposed framework with other algorithms using four datasets with different numbers of training samples per class, the more training samples, the higher the classification accuracy for each method. To be specific, we varied the number of labeled examples per class from 5 to 30, and reported the OA values obtained by all methods for all four datasets& nbsp;(Rao et al. 2019). Figure 11 shows that our method had the best performance when applied to all four datasets, which demonstrated the effectiveness of SSC-SFN-M for obtaining spectral-spatial information, as well as its suitability for large areas with complex background information under the constraint of few samples. In addition, the federated strategy of multiplication obtained stable results, which also confirmed the importance of continuing research on superpixel multiplication. In particular, there was a discrepancy between FPGA and our new method, due to the advantage of combining the GCN and CNN mentioned previously. As expected, the SSC-SFN-M classification of the Salinas dataset was very accurate, as the accuracy of other methods is > 85% when labeling 30 samples per class. Our method had an obvious advantage in terms of its ability to operate with a limited number of labeled samples. Some comparative methods performed poorly for the three WHU-Hi datasets, indicating that these algorithms cannot easily extract complex background information, and that the proposed method has good performance for complex and large datasets. More interestingly, our method achieved relatively high classification accuracy when the sample size was about 30 per class, which also confirmed the suitability of the method for HSI classification tasks.

Figure 11. The OA according to the number of training samples. (a) Salinas (b) WHU-Hi-HanChuan (c) WHU-Hi-HongHu, and (d) WHU-Hi-LongKou.

4.2. Federated network strategy

Stacking multiple models is one way to improve the efficiency of HSI classification algorithms. After extracting deep features separately through CNN and GCN, we implemented three different federated classification strategies to improve the algorithm& nbsp;(Chen et al. 2016). Tables 6–9 list the classification results of the traditional methods, superpixel GCN (S-GCN), SP-CNN, and SSC-SFN using different federated classification strategies with four datasets. From the four tables, it is apparent that CNN was better than GCN for solving the HSI classification problem. The accuracy of SP-CNN was also greatly improved, indicating that the superpixel pooling process is useful. For the application of the three federated classification strategies to the Salinas, WHU-Hi-HongHu datasets, and WHU-Hi-LongKou, the multiplication strategy is more suitable for splicing and superposition. This may be because the input of SSC-SFN-M can better characterize the topological relationship between superpixels after normalizing the depth eigenvalues. For the WHU-Hi-HanChuan dataset, superpixel splicing showed superior accuracy, likely because the best approximation value of the solution was useful for analyzing the shaded areas. Interestingly, the traditional CNN performed better than S-GCN when applied to the Salinas dataset, demonstrating that it was more suitable for fixed window feature extraction and recognition. Our experiments were all implemented in a unified environment using TensorFlow (GPU version).

Table 6. Results obtained by applying several Classification schemes to the Salinas dataset.

	CNN	GCN	S-GCN	SP-CNN	SSC-SFN-P	SSC-SFN-M	SSC-SFN-S
OA	89.66 ± 0.98	85.13 ± 0.85	88.64 ± 0.88	97.49 ± 0.47	99.15 ± 0.34	99.48 ± 0.23	99.19 ± 0.68
AA	91.94 ± 1.62	85.99 ± 1.62	89.05 ± 0.87	97.87 ± 0.76	99.68 ± 0.36	99.79 ± 0.09	99.73 ± 0.10
Kappa	89.19 ± 0.61	84.73 ± 0.75	87.79 ± 0.52	96.46 ± 0.55	98.78 ± 0.59	99.04 ± 0.30	98.83 ± 0.39

Table 7. Results obtained by applying several Classification schemes to the WHU-HI-HanChuan dataset.

	CNN	GCN	S-GCN	SP-CNN	SSC-SFN-P	SSC-SFN-M	SSC-SFN-S
OA	68.69 ± 1.89	66.89 ± 0.73	74.69 ± 1.29	85.97 ± 0.98	90.35 ± 0.83	91.82 ± 0.78	91.95 ± 0.64
AA	68.88 ± 3.85	68.24 ± 3.42	75.08 ± 2.61	86.07 ± 2.76	91.56 ± 0.84	92.63 ± 2.40	92.72 ± 0.51
Kappa	66.72 ± 1.40	65.33 ± 0.74	73.39 ± 0.48	84.90 ± 0.39	89.39 ± 0.51	90.32 ± 0.58	91.15 ± 0.21

Table 8. Results obtained by applying several Classification schemes to the WHU-HI-HongHu dataset.

	CNN	GCN	S-GCN	SP-CNN	SSC-SFN-P	SSC-SFN-M	SSC-SFN-S
OA	70.85 ± 2.38	67.02 ± 1.94	70.69 ± 1.47	83.16 ± 1.43	91.51 ± 0.83	92.94 ± 0.71	92.01 ± 0.68
AA	73.37 ± 2.61	71.21 ± 2.78	71.70 ± 2.61	82.72 ± 1.96	93.06 ± 1.35	93.60 ± 0.62	92.15 ± 1.32
Kappa	67.96 ± 3.48	66.02 ± 1.52	66.76 ± 1.27	79.51 ± 1.08	89.99 ± 0.84	91.45 ± 0.55	90.35 ± 0.62

Table 9. Results obtained by applying several Classification schemes to the WHU-HI-LongKou dataset.

	CNN	GCN	S-GCN	SP-CNN	SSC-SFN-P	SSC-SFN-M	SSC-SFN-S
OA	82.16 ± 1.68	78.64 ± 2.83	82.19 ± 1.88	90.99 ± 0.37	98.05 ± 0.14	98.52 ± 0.19	98.29 ± 0.61
AA	83.54 ± 2.92	78.99 ± 4.62	83.25 ± 2.86	90.97 ± 1.56	97.98 ± 0.96	98.78 ± 0.49	98.43 ± 0.39
Kappa	82.9 ± 1.61	78.73 ± 1.75	81.79 ± 1.52	90.26 ± 0.45	97.08 ± 0.09	97.29 ± 0.10	97.03 ± 0.69

4.3. Influence of the number of superpixels

The number of superpixels also affects the classification results of our method. Therefore, we used four datasets to determine the optimal parameters of SSC-SFN-M. Notably, the maximum interclass variance method (i.e. OTSU) (Otsu 1979) automatically obtains the optimal threshold of superpixel size and thus does not rely on other methods such as transfer learning or data augmentation. Figure 12 shows how the classification results changed with the number of superpixels for the four datasets. The OA of SSC-SFN-M was excellent under all parameter conditions. In addition, the performance of SSC-SFN-M was robust because non-local superpixels can integrate spatial features. The optimal numbers of superpixels for the Salinas, WHU-Hi-HanChuan, WHU-Hi-HongHu, and WHU-Hi-LongKou datasets were 1,100, 700, 1,000, and 1,100, respectively. This was because the optimal hyperparameters were affected by many factors, including the spectral resolution, spatial resolution, and distribution characteristics of the ground features of HSI. Generally, the higher the spatial resolution, the clearer the distribution of ground features, and the more appropriate it is to increase the number of superpixels. However, imbalance of the dataset leads to ‘refinement’ of the superpixels, and in turn to a higher misclassification rate; this process occurred for the WHU-Hi-HanChuan dataset. The number of samples in each category varies greatly among the tested datasets, which impedes mining of more detailed information about spatial structures. If detailed features are to be divided during image segmentation, the number of superpixels must be sufficient, as too few superpixels will reduce the effect of superpixel segmentation. In addition, as the number of superpixels increases, time consumption and memory requirements should also be considered.

Figure 12. The OA of SSC-SFN-M according to the numbers of superpixels.

4.4. Influence of network depth

Network depth is a hyperparameter that must be considered in DL frameworks, as it can have a large impact on network performance. Therefore, 10 independent experiments were conducted and OA was used as the performance criterion to determine the impact of network depth on the results of the proposed method. As shown in Figure 13, for the Salinas dataset, the performance of the SSC-SFN-M increased with network depth. The reason for this may be superpixel smoothing and the irregular edge extraction of the GCN path. Deeper networks could suppress more noise or outliers, resulting in better accuracy. For the WHU-Hi-HanChuan dataset, a network depth of 2–3 was the most reasonable. However, the method results were greatly affected by shadow occlusion and mixed ground objects, indicating that networks that were too deep could not effectively eliminate such interference. For the WHU-Hi-HongHu dataset, SSC-SFN-M achieved the best performance with three network layers. Because the WHU-Hi-HongHu dataset had ‘tighter’ large-scale land cover types at the spatial level, deepening the network will cause the gradient to vanish and there will be no performance improvement. Thus, for real datasets with complex land cover and large amounts of data, if the SSC-SFN-M is too deep performance degradation may occur. The network depth of the SSC-SFN-M was fixed to three in all of the other experiments.

Figure 13. The OA of SSC-SFN-M according to network depth for the four datasets. (a) Salinas (b) WHU-Hi-HanChuan, (c) WHU-Hi-HongHu and (d) WHU-Hi-LongKou.

4.5. Non-local superpixel

The non-local superpixel is an important facet of the proposed method, as it effectively alleviates the problem of having an entire feature matrix of GCN inputs. Non-local smoothing reduces the computational cost and preserves important edge information through iterative smoothing. This is important for the subsequent GCN process, to fully learn from the relationships of nearest neighbor pixels& nbsp;(Gimeno et al. 2021). Therefore, for quantitative analysis of the role of the non-local superpixel in this section, all OA values presented are average results for 10 independent experiments. To demonstrate the smoothing effect of the non-local superpixel method, we selected several existing superpixel segmentation algorithms for comparison, including Watershed (Vincent and Soille 1991), Meanshift (Fukunaga and Hostetler 1975), Quickshift, SLIC, ERS (Liu et al. 2011), SLIC0 and SAM. As shown in Table 10, non-local segment strategy showed satisfactory performance for the four experimental datasets, improving OA by approximately 10%. The reason for this improvement is that internal smoothing of superpixels can extract more critical boundary features and appropriate spatial information, indicating that this attempt at local pattern search smoothing was successful. Moreover, the non-local superpixel has lower sample requirements, which represents a breakthrough in solving the labeled sample problem. For the WHU-Hi-HanChuan dataset, the OA of non-local segment strategy was improved by > 8% compared with other methods, demonstrating that non-local segment strategy is excellent for considering shadow areas on farmland. SLIC0 did not perform as well as SLIC, likely because SLIC0 does not handle irregular areas of texture as well as SLIC. Interestingly, for the relatively regular Salinas dataset and the WHU-Hi-HongHu dataset, SLIC0 was advantageous, as it adds a constant compression factor to improve the shape of superpixel blocks. In addition, SAM demonstrated the advantages of the graph structure when applied to Salinas and WHU-Hi-HanChuan datasets, providing better classification results than SLIC and SLIC0; this is one of the reasons that SAM was selected as the basic segmentation algorithm for this study. The classification results for Watershed, Meanshift, and Quickshift were not ideal, indicating that the background complexity of the four datasets is high, the recognition of objects is difficult, and the possibility of missegmentation of complex scenes is markedly increased.

Table 10. Results of several superpixel segmentation algorithms for four datasets.

	Watershed	Meanshift	Quickshift	SLIC	ERS	SLIC0	SAM	ERS + SAM (Non-local)
Salinas	90.84	90.47	91.03	93.82	94.22	94.15	92.34	99.48
HanChuan	79.25	79.35	79.02	84.32	85.54	83.16	84.64	91.82
HongHu	80.88	81.36	81.92	86.05	86.36	87.72	85.38	92.94
LongKou	88.47	89.25	89.92	90.64	92.74	92.58	91.47	98.52

4.6. Running time

Running time is one of the most important metrics for evaluating the effectiveness of an HSI classification algorithm. An efficient classification algorithm is most suitable for practical engineering applications. Table 11 reports the training and testing times of several HSI classification algorithms using four images. FuNet-C is an improved algorithm based on MiniGCN. The advantages of the mini-patch gradually disappear after the amelioration process. In addition, the slow speed of CNNCRF may be due to the CNN not considering the spatial node relationship of the graph during image interpretation, such that CNNCRF takes longer to achieve higher accuracy, making this model more time-consuming than others. The efficiency of FPGA is relatively high, as FPGA alleviates the complexity issue in highly overlapping areas and can train FreeNet via global learning with the free-patch step. Almost all of the fast algorithms employ superpixel techniques, including MDGCN, CEGCN, and the proposed method. However, the dynamic multi-scale processing adopted by MDGCN may limit its efficiency. Moreover, CEGCN adds pixel-level features for learning, which undoubtedly increases the computational cost. Compared with several other methods, our method saves time because the combination of the superpixel segmentation technique and federated network strategies makes the input classifier data smaller.

Table 11. Running time (seconds) of eight classification schemes for four datasets with 30 labeled pixels per class.

		MiniGCN	MDGCN	FuNet-C	SSOGCN	CEGCN	FPGA	CNNCRF	C2FN-M
Salinas	Training	759.242	268.934	2741.835	704.805	137.796	247.365	590.892	116.372
	Testing	63.832	11.495	92.375	25.824	6.589	5.047	18.291	5.923
WHU-Hi-HanChuan	Training	948.752	341.858	3038.151	908.16	178.925	309.377	739.026	165.802
	Testing	189.332	29.769	244.651	87.356	19.463	15.512	79.293	10.872
WHU-Hi-HongHu	Training	1660.314	575.093	4519.283	1545.082	313.215	541.809	1295.837	331.802
	Testing	84.715	27.328	287.773	70.482	15.873	27.435	203.882	12.873
WHU-Hi-HongHu	Training	703.532	364.92	2859.931	930.503	81.924	148.936	204.921	102.841
	Testing	96.362	29.544	194.386	34.553	10.927	5.923	12.501	5.792

4.7. Performance and generalization ability analysis of SSC-SFN

In the process of model training, the problem of overfitting often occurs, that is, the model can match the training data well, but cannot predict the data other than the training set. If the test data is used to adjust the model parameters at this time, it is equivalent to the information of some of the test data known at the time of training, which will affect the accuracy of the final evaluation results. Generally, a part of the training data is randomly selected as the validation data to evaluate the training effect of the model. We divided the experimental data into training and validation sets. The test set is training-independent data, not involved in training at all, for the evaluation of the final model. The validation data is taken from the training data, but it does not participate in the training, so that the model matches the data outside the training set relatively objectively. The validation method selects K-cross validation, which divides the original data into K groups, and makes each subset of data separately. The K is defined as 100 in the experiment. In Figure 14, when K = 20, the model achieves stable evaluation accuracy and shows that the results are as close as possible to the performance of the model on the test set, and the parameters can be used as the best parameters for model optimization.

Figure 14. The results of K-cross validation show that when K = 20, the model performs best.

When training the model, there may encounter insufficient training data, the training data cannot estimate the distribution of the whole data, or overtraining the model often leads to model overfitting. To prevent model overfitting, we added a Dropout layer to the model. Specifically, each neuron in the input tensor is set to 0 with a certain probability. The keep_prob was set to 0.2 (indicating that 80% of neurons were placed to 0 and the remaining 20% of neurons were divided by 0.2). The main idea of Dropout layer is distributed expression to the features (Zhou et al. 2023). During the training, some nodes are randomly discarded, so that these nodes do not participate in the update training of the parameters (generally set to 0.5, which closes the output of 50% of the hidden layer neurons). When the data is updated again, the election is randomly conducted. The Dropout layer allows to reduce of the dependence between neurons, because random selection of nodes makes neurons not always acting simultaneously, which effectively prevents model overfitting.

5. Conclusions

In this study, a spectral-spatial DL model for HSI classification based on a non-local segment federated network is described. This method exploits the homogeneity of superpixels to perform non-local smoothing within each superpixel to optimize each superpixel block. Thus, spatial information is obtained for small batches, which facilitates the classification of HSIs with complex backgrounds. The proposed SSC-SFN combines the CNN and GCN techniques to exploit the available spectral and spatial structure information. More importantly, the federated network strategies based on CNN and GCN used in this framework improve the robustness of feature recognition for HSI classification tasks. In summary, SSC-SFN solved the problem of classifying large-scale complex background research areas with limited labeled samples. Many experiments were conducted on four widely applied datasets. Compared with state-of-the-art DL algorithms, SSC-SFN-M achieved a higher OA (> 91.8%). Specifically, SSC-SFN-M improved accuracy by > 12% relative to FuNet-C. When the sample size per class was reduced from 30 to 5, the proposed method retained outstanding accuracy and had an extremely fast analysis speed, demonstrating the effectiveness and superiority of the proposed SSC-SFN method. The main contribution of SSC-SFN is integrating two paths between superpixels, to automatically obtain the optimal threshold for superpixel size (thus reducing the hyper-parameter optimization problem). It also achieves accurate and fast classification of large hyperspectral datasets with a small number of training samples.

Acknowledgements

We acknowledge all the reviewers and editors of the journal for their valuable comments, suggestions, and revisions on this paper.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The data that support the findings of this study are available from the corresponding author, [Taixia Wu], upon reasonable request.

Additional information

Funding

This work was supported by a grant from Key Laboratory of Emergency Satellite Engineering and Application, Ministry of Emergency Management; Science for a Better Development of Inner Mongolia Program: [Grant Number 2022EEDSKJXM003].