Hyperspectral image classification using improved multi-scale block local binary pattern and bi-exponential edge-preserving smoother

ABSTRACT

In this paper, a multi-strategy fusion (MSF) framework, based on improved MBLBP and bi-exponential edge-preserving smoother (BEEPS), is proposed for hyperspectral image (HSI) classification. First, MBLBP operator is adopted to characterize the overall structural information of HSI, where the averaging strategy allocates same weights for the pixels in a local sub-region, so that the edges tend to be blurred due to being isotropic. To solve this question, the steering kernel is first introduced into MBLBP for learning the local structure prior of HSI. Then, a support vector machine classifier is used to calculate the soft classified probabilities of pixels. Furthermore, BEEPS is adopted to smooth the soft classified probabilities maps in the post-processing stage, and the purpose is to further improve classification accuracy of HSI by considering context-aware information for each class label. Experiments are performed on three real hyperspectral datasets, namely, Indian Pines, KSC, and Houston 2013, only 1%, 6, and 5 labeled samples are randomly selected for training, the overall accuracy(kappa) obtained by MSF is 99.47%(99.40), 99.52%(99.47), and 94.25%(93.78), respectively, which is far better than the contrast methods.

KEYWORDS:

• Classification
• hyperspectral image
• multiple strategy fusion
• multi-scale block local binary pattern
• edge-preserving filtering

Introduction

Hyperspectral image (HSI), collected by sensors technology, can record rich spectral and spatial information from hundreds of continuous spectral channels, which support the fine recognition of various land cover objects (Fu et al., 2016). Therefore, HSI has attracted more and more attention in different application fields such as precision agriculture (Zhang et al., 2021), target detection (Y. Li et al., 2019; Ou et al., 2020), food safety (Aviara et al., 2022), and military application (Hupel & Stütz, 2022). Particularly, HSI classification is an important and challenging task in remote sensing, and the goal is to recognize the pixels into a certain class. In the past few decades, people often focused on spectral information, utilizing only spectral information to finish the classification task, and proposed a large number of pixel-wise classification methods, such as random forests (RF) (Ham et al., 2005), sparse representation (Y. Chen et al., 2011), neural networks (Ratle et al., 2009), and support vector machine (SVM) (Melgani & Bruzzone, 2004). Among these algorithms, the SVM classifier provides competitive performance in terms of classification accuracy, since it is insensitive to the “curse of dimensionality” problem, i.e., the high computational burden and the lack of training samples can lead to the high computational complexity along with may even affect the performance of classifiers (H. Liu et al., 2019; Peng et al., 2015).

Although pixel-wise classification algorithms have the advantages of conceptual simplicity and computational effectiveness by fully utilizing spectral information, they often happen with undesired salt and pepper appearance because spatial information and the variability of spectral features have not been appropriately addressed (Fauvel et al., 2013). In recent years, various spatial-spectral analysis technologies have been developed that exploit both spectral and spatial contextual information to complement pixel-wise classification. For example, Markov models were adopted for HSI classification by integrating spatial and contextual information into the classifier (Cao et al., 2020; Jiang et al., 2020). However, this model has two disadvantages of heavy computation and insufficient samples to characterize the object of interest. Another family of spatial-spectral classification is based on the kernel combination or fusion, e.g. morphological, composite, and graphic kernels (Anand et al., 2021; Ergul & Bilgin, 2020). The kernel-based algorithms, particularly, in the joint with SVM have been demonstrated to provide good classification performance in HSI (B. Kumar & Dikshit, 2017; S. Yang et al., 2019).

Inspired by the sparse coding mechanism of human vision systems (Olshausen & Field, 1996), sparse representation has been proved to be an effective tool in HSI classification based on the hypothesis that an unknown test pixel can be approximately represented by a few atoms among all training samples (Y. Chen et al., 2011). Recently, the joint representation models have emerged, drawing on the promotion of sparse representation and collaborative representation, such as kernel-based joint sparse model, structured joint sparse model, and multiscale adaptive sparse representation (Fang et al., 2014; Yan et al., 2019; W. Yang et al., 2020).

Another important model of spatial-spectral classification methods is image segmentation. First, based on the homogeneity of either texture or intensity, the HSI dataset is segmented into different regions where pixels within the same region can be deemed as a spatial neighborhood (Sun et al., 2021); Then, the pixel-wise classification results with the segmentation map obtained by image segmentation are integrated by the majority voting rule. At present, different segmentation technologies such as hierarchical segmentation, watershed, and minimum spanning forest have been proposed for HSI classification (Akbari, 2019; Lv et al., 2020). In general, the segmentation techniques have a great influence on the performance of segmentation-based algorithms. To obtain better segmentation results, multiple spatial-spectral classification algorithms can be combined, but this strategy may be time-consuming (Majdar & Ghassemian, 2020).

Recently, edge-preserving filtering (EPF) plays an important role in the field of remote sensing and has been widely applied in many fields, such as image segmentation, image denoising, classification, detail enhancement, and image fusion (Kang et al., 2014; S. Li et al., 2013; Xing et al., 2021). Up to now, as a preprocessing step, a large number of edge-preserving filtering techniques have been applied before performing HSI classification task, and the representative methods include bilateral filter (BF), bi-exponential edge-preserving smoother (BEEPS), curvature filter, recursive filter, and guided image filter (GIF) (Hao et al., 2022; J. Liao et al., 2019; Wan & Zhao, 2019). Among them, BF, BEEPS, and GIF have been broadly utilized for their simplicity and effectiveness. In particular, compared with the BF and GIF, BEEPS provides competitive performance as it presents no ripples in the frequency domain and the computational complexity of each pixel is constant. Inspired by the spatial variability due to the noise can be smoothed out with the help of edge-preserving filtering technologies, researches have proved that post-processing classification with EPF is also an important strategy in enhancing the performance of classifiers (Wang et al., 2017). A complementary spectral-spatial method for HSI classification was developed, which integrates texture smoothing pre-processing feature extraction method and post-processing feature extraction method based on edge preserving filtering (Shi et al., 2022).

Besides, deep learning (DL) models have been extensively applied in the field of HSI classification due to their ability of exploiting intrinsic and differentiable high-level features from original images. HSI classification method based on stacked autoencoder was exploited to generate higher classification accuracy by using the spatial-spectral features (Y. Chen et al., 2014). Later, the DL model based on deep belief network (DBN) was presented (Tong et al., 2015). Since then, various deep learning methods, like recurrent neural network (RNN), generative adversarial network (GAN), random patches network (RPNet), convolutional neural network (CNN), and others, have been developed for HSI classification, respectively. Among these networks, CNN has attracted extensive attention in HSI classification because of its excellent classification ability. For instance, A 3-DCNN-based feature extraction model with combined regularization is built to extract the spectral-and-spatial-based deep features for HSI classification (Y. Chen et al., 2016). Furthermore, a novel supervised classification algorithm for HSI classification was developed, which combines both spectral and spatial information in a unified Bayesian framework (Cao et al., 2018). Very recently, a deep CNN and Markov random field (MRF)-based two-stage classification framework is developed for HSI classification (Singh et al., 2021). To obtain higher classification accuracy under the condition of limited training samples, a multidimensional CNN combined with an attention mechanism model was proposed (J. Liu et al., 2022).

Generally speaking, the deep models could exploit the robust and high-level features, which provide competitive results to state-of-the-art methods in terms of classification accuracy. However, the structure of a deep network consists of a large number of parameters and layers, and these disadvantages make DL very time-consuming in the training process (G. Zhao et al., 2020). To alleviate this situation, a new broad learning system (BLS) (C. Chen & Liu, 2018) was proposed. Compared with DL networks, less network training parameters and fewer labeled training samples are needed in BLS, leading to the training time is significantly lower. At present, some spatial-spectral classification methods based on BLS have been developed for HSI classification (G. Zhao et al., 2020; L. Zhao et al., 2022).

Well known, hyperspectral images consist of hundreds of contiguous spectral channels, nevertheless many of which include redundant information. General speaking, band selection (e.g. linear prediction error (LPE)) and linear projections (e.g. principal component analysis (PCA)) are two main ways to reduce the dimensionality. Recently, researchers are increasingly focusing their attention on utilizing nature-inspired optimization algorithms for the selection of features for HSI classification. A three-dimensional discrete wavelet transform-based spatial feature extraction was proposed by using improved whale optimization-based band selection for HSI classification (Manoharan & Boggavarapu, 2021). A whale optimization-based band selection for the hyperspectral image is presented by resembling the hunting behaviour of humpback whales (B. L. N. P. Kumar & Manoharan, 2021). A novel unsupervised multi-objective multi-verse optimizer-based band selection approach was developed for HSI classification (Sawant et al., 2022).

Apart from the global structure obtained from dimensionality reduction approaches, e.g. principal component analysis (PCA) and linear discriminant analysis (LDA), the local structure is also significant in practical applications (Uddin et al., 2021). Local binary pattern (LBP) is a simple and effective texture operator with low computational complexity, which has effectively encapsulated integrated spatial structure of local texture information for texture classification in pattern recognition and computer vision fields (Ye et al., 2020). In general, traditional LBP is demonstrated as a useful and powerful local descriptor for micro-structures of images, but it also has some shortcomings for recognizing and classifying scene categories (Y. Li et al., 2022): noisy image generates low-quality results for basic LBP operator because it is built in a small spatial support area, and features extracted from a local neighborhood window incapable characterize larger-scale structure information that may be stapled textural features of the scene. To solve the abovequestions, the multiscale block LBP (MBLBP) is proposed to extract the overall structural information by averaging the values of subregions, which can prevent the loss of detailed features to a certain extent (Y. Li et al., 2022; S. Liao et al., 2007). Currently, the MBLBP operator has provided very effective and robust performance in some research fields, such as target recognition, surface defect detection, image retrieval, and object tracking (Halidou et al., 2014; W. Li et al., 2015).

Previous researches have demonstrated that the MBLBP descriptor characterizes a relatively complete image representation by encoding the microstructures and macrostructures. Furthermore, post-processing classification is a critical step in enhancing the quality of classifiers. Inspired by these ideas, to characterize both the overall structural and edge-preserving filtering information to improve the accuracy of HSI classification, a spatial-spectral HSI classification model based on the improved MBLBP and BEEPS algorithms is proposed in this paper. For the sake of convenience, this proposed HSI classification method is called MSF with a multi-strategy fusion scheme.

The main contributions of our proposed framework are as follows:

An improved MBLBP descriptor by introducing the steering kernel is applied to HSI in the pre-processing stage, which could capture the spatial correlation among neighboring pixels, and could mitigate this phenomenon of “same class has different spectral information” and “different classes have same spectral information” to a certain extent.

BEEPS is introduced into the initial probability maps in the post-processing stage, where the extracted context-aware information for each class label can significantly improve HSI classification accuracy. Namely, it is to make up for the information loss of feature extraction by adding smoothing to the probability maps.

Comparative experiments are performed on three real hyperspectral datasets, and the experimental results demonstrate that the proposed MSF method is superior to other state-of-the-art algorithms in terms of classification accuracy and visual classification maps, especially suitable for small-sample problems in HSI classification.

The rest of this paper is arranged as follows. Section 2 gives the flowchart of MSF framework and presents the description of IMBLBP methodology. Section 3 provides the experimental results and discussions on three real hyperspectral images. Finally, the paper is summarized and future research directions are given in Section 4.

Methodology

In view of the above description, the application that we are focused on in this paper is a multi-strategy supervised classification framework. For illustrative intents, the diagram of the proposed classification framework is depicted in Figure 1, which is constituted of three main strategies: (1) Extract overall structural information of principal components (PCs); (2) Acquire the original soft-classified probabilities maps; (3) Smooth the soft-classified probability maps by the BEEPS. Concretely, principal component analysis (PCA) is employed to transform the high-dimensional HSI into a lower dimensional subspace. After dimensionality reduction, the proposed IMBLBP algorithm is applied to each principal component, to characterize the overall structural information. In addition, the obtained IMBLBP features as input are fed into the SVM classifier to generate initial probability maps. Finally, a BEEPS based post-processing is applied to the initialed probability maps, where the maximum probability criteria are adopted to determine the final classification result.

Figure 1. Illustration of the proposed classification framework.

In this section, the feature extraction based on IMBLBP operator is introduced. For the basic LBP operator, a small spatial support region, so that the bit-wise comparison between the central pixel and its neighborhood pixels is easily affected by noise. Furthermore, features extracted in a 3 × 3 neighborhood window cannot characterize large scale structures that may lose the detailed features. To address the limitations of LBP, an improved multiscale block LBP with steering kernel (IMBLBP) is proposed, which is applied to extract the structural features of HSI.

In traditional MBLBP, the comparison operator between the average intensity values of rectangular sub-regions is adopted to replace with the comparison between single pixels in LBP. An MBLBP descriptor contains nine sub-regions, and each sub-region is a square block with 2n + 1 pixels (or just one pixel, is considered as the basic LBP), as shown in Figure 2. Formally, the resulting MBLBP can be calculated as:

Figure 2. The 9×9 MB-LBP operator.

(1) $MBLB{P}_{{\bar{g}}_c}=\sum _{p=0}^{7}s({\bar{g}}_p-{\bar{g}}_c)$(1)

where ${\bar{g}}_p=\frac{1}{s^2}\sum _{i=1}^{s^2}{g}_p(i)$ denotes the average intensities of neighborhood block pixels, and ${\bar{g}}_c=\frac{1}{s^2}\sum _{i=1}^{s^2}{g}_c(i)$ represents the average intensity of the center block. The function $s(\cdot )$ is interpreted as follows:

(2) $s({\bar{g}}_p-{\bar{g}}_c)=\left\{\begin{array}{c}1\hspace{0.17em}{\bar{g}}_p\ge {\bar{g}}_c\\ 0\hspace{0.17em}{\bar{g}}_p<{\bar{g}}_c\end{array}\right.$(2)

Besides the local structures, MBLBP can characterize the global structures of an image by averaging pixels in the overlapping sub-regions. Note that, averaging strategy allocates the same weights for the pixels in a local sub-region and ignores the directions of edges in the image so that the edges are inclined to be blurred in the averaging process.

In view of the edge structure plays a critical role in executing MBLBP algorithm and it is expected to improve the distinguish ability of texture information by taking advantage of the edge direction. In this paper, instead of the mean filtering strategy in MBLPB, the steering kernel is introduced into the MBLBP descriptor to learn local structure prior of HSI. Due to the steering kernel concerning the edge direction and the radiometric similarities of pixels in a square block, the strategy enables the proposed IMBLBP to be more reliable than the basic MBLBP. Mathematically, the steering kernel is defined as:

(3) $w_{ik}=\frac{\sqrt{det(C_i)}}{2\pi h^2}exp\left\{-\frac{{(x_i-x_k)}^T{C}_i(x_i-x_k)}{2h^2}\right\}$(3)

where $h$ represents the smoothing parameter to manage the supporting range of the kernel, $x_i$ and $x_k$ denote the pixel coordinates. Here, $C_i$ represents a symmetric gradient covariance matrix calculated from a local square block $w_i$. Specially, to acquire a steady estimate of covariance matrix, a parametric approach is utilized in (Takeda et al., 2007). That is, the matrix $C_i$ is supposed a composition of three elements as follows:

(4) $C_i={\gamma }_i{U}_{{\theta }_i}{\mathrm{\Lambda }}_i{U}_{{\theta }_i}^T$(4)

(5) $U_{{\theta }_i}=\left[\begin{array}{c}cos{\theta }_i\mathrm{s}\mathrm{i}\mathrm{n}{\theta }_i\\ -sin{\theta }_i\hspace{0.17em}cos{\theta }_i\end{array}\right]$(5)

(6) ${\mathrm{\Lambda }}_i=⌊\begin{array}{c}{\sigma }_{i}0\\ 0\hspace{0.17em}{\sigma }_i^{-1}\end{array}⌋$(6)

where ${\gamma }_i$ represents the scaling parameter, $U_{{\theta }_i}$and ${\mathrm{\Lambda }}_i$ denote the rotation and elongation matrix, respectively. Here, the singular value decomposition of the local gradient matrix $G_i$ is used to determine the three parameters ${\gamma }_i$,${\theta }_i$, and ${\sigma }_i$.

As mentioned before, instead of the averaging strategy in MBLBP, the proposed IMBLBP adopts the weighted averaging method to characterize the edge direction information by introducing the steering kernel. In this way, the normalized local steering kernel is used as the weight function, and then ${\bar{g}}_p$ and ${\bar{g}}_c$ in Equation (1)(1) $MBLB{P}_{{\bar{g}}_c}=\sum _{p=0}^{7}s({\bar{g}}_p-{\bar{g}}_c)$(1) are recalculated as:

(7) ${\bar{g}}_p={\sum }_{k\in {\omega }_i}{w}_{ik}{g}_p$(7)

(8) ${\bar{g}}_c={\sum }_{k\in {\omega }_i}{w}_{ik}{g}_c$(8)

After calculating the weighted average intensities of neighborhood block pixels and the weighted average intensity of the center block, i.e. ${\bar{g}}_p$ and ${\bar{g}}_c$, respectively. The function $s(\cdot )$ is still interpreted by Equation (2)(2) $s({\bar{g}}_p-{\bar{g}}_c)=\left\{\begin{array}{c}1\hspace{0.17em}{\bar{g}}_p\ge {\bar{g}}_c\\ 0\hspace{0.17em}{\bar{g}}_p<{\bar{g}}_c\end{array}\right.$(2) .

Experimental results and discussions

Experimental setup

Dataset description

Three publicly available hyperspectral datasets are conducted to verify the superiority of the proposed method, i.e. the Indian Pines image, the Kennedy Space Center (KSC) image, and the Houston 2013 image.

The Indian Pines dataset was gathered by the Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) sensor in Northwestern Indiana, with a spectral coverage changing from 0.4 to 2.5 μm. The image scene comprises of 16 types of crops, with a size of 145 × 145 pixels, and the spatial resolution is 20 m per pixel. After eliminating 20 water immersion and noisy channels, the remaining 200 spectral bands are used for experimental analysis. The category information and the number of samples used for training and testing in the Indian Pines dataset are shown in Table 1.

Table 1. Class labels and train-test distribution of samples for the Indian Pines dataset.

Class	Name	Train	Test
1	Alfalfa	1	53
2	Corn-notill	15	1419
3	Corn-mintill	9	825
4	Corn	3	231
5	Grass-pasture	5	492
6	Grass-trees	8	739
7	Grass-pasture-mowed	1	25
8	Hay-windrowed	5	484
9	Oats	1	19
10	Soybean-notill	10	958
11	Soybean-mintill	25	2443
12	Soybean-clean	7	607
13	Wheat	3	209
14	Woods	13	1281
15	Buildings-Grass-Trees	4	376
16	Stone-Steel-Towers	1	94
Total		111	10255

The second real-world HSI dataset was captured by the AVIRIS instrument over the KSC of Florida in 1996, with a geometric resolution of 18 m per pixel. The size of the image is 512 × 614 and a total of 224 spectral channels, where the wavelength ranging from 0.4 to 2.5 μm. The number of spectral channels is 224 for KSC, and the remaining 176 bands are utilized for subsequent analysis after abandoning the low signal-to-noise ratio and water absorption channels. The category information and the number of samples used for training and testing in the KSC dataset are shown in Table 2.

Table 2. Class labels and train-test distribution of samples for the KSC dataset.

Class	Name	Train	Test
1	Scrub	6	755
2	Willowswamp	6	237
3	CPhammock	6	250
4	CP/oak	6	246
5	Slashpine	6	155
6	Oak/broadleaf	6	223
7	Hardwoodswamp	6	99
8	Graminoidmarsh	6	425
9	Spartinamarsh	6	514
10	Cattailmarsh	6	398
11	Saltmarsh	6	413
12	Mudflats	6	497
13	Water	6	921
Total		78	5133

The third dataset was captured by the compact airborne spectrographic imager (CASI) sensor over the Houston University campus and the neighboring urban area. This image consists of 144 bands with each band of size 349 × 1905, where the spatial resolution is 2.5 m/pixel, and the spectral coverage ranges from 0.38 to 1.05 µm. There are a total of 15,029 labeled samples available that are divided into 15 different land cover types. The category information and the number of samples used for training and testing in the Houston 2013 dataset are shown in Table 3.

Table 3. Class labels and train-test distribution of samples for the Houston 2013 dataset.

Class	Name	Train	Test
1	Healthy grass	5	1246
2	Stressed grass	5	1249
3	Synthetic grass	5	692
4	Trees	5	1239
5	Soil	5	1237
6	Water	5	320
7	Residential	5	1263
8	Commercial	5	1239
9	Road	5	1247
10	Highway	5	1222
11	Railway	5	1230
12	Parking lot1	5	1228
13	Parking lot2	5	464
14	Tennis court	5	423
15	Running track	5	655
Total		75	14954

Parameter setting

In our experiments, three standard performance metrics: the overall accuracy (OA), the average accuracy (AA), and the kappa coefficient (κ) are adopted for quantifying the quality of different classification algorithms. In this section, we evaluate the proposed IMBLBP and MSF algorithms with some state-of-the-art classification techniques, such as the raw spectral features with the SVM classifier (SVM) (Melgani & Bruzzone, 2004), the guided image filtering features using the SVM classifier (GF-SVM), the bi-exponential edge-preserving smoothing features with the SVM classifier (BEEPS-SVM) (Wan & Zhao, 2019), decision-level fusion with the LBP features along with initial spectral features and global Gabor features (DF-SVM) (W. Li et al., 2015), multiscale adaptive sparse representation (MASR) (Fang et al., 2014), spectral-spatial shared kernel ridge regression (SSSKRR) (C. Zhao et al., 2019), a spatial-spectral joint classification frame based on the edge-preserving filtering (EPF-B-g) (Kang et al., 2014) and local binary pattern features using the SVM classifier (LBP-SVM) (W. Li et al., 2015), 3D convolutional neural network with logistic regression (3D-CNN-LR) (Y. Chen et al., 2016) and a spatial-spectral classification using locality preserving projection, local binary pattern, and broad learning system (LPP-LBP-BLS)) (G. Zhao et al., 2020). All contrastive algorithms utilize the same parameter setting provided by the original papers.

Before performing the proposed IMBLBP and MSF methods, it is needed to conduct parameter setting. To maximize the overall classification accuracy, 10fold cross validation is used to empirically regulate the related parameters, i.e. the patch size $W$, the number of principle components $K$, the spatial parameter $\lambda$, and the range parameter $\sigma$. For IMBLBP, the number of principle components $K$ is chosen from the set $\left\{3,5,7,9,11,13,15\right\}$, and the patch size $W$ is chosen from the set $\left\{5,9,13,17,21,25,29,33\right\}$. Here, $K$ and $W$ are set as $K=7$, $W=25$ for the Indian Pines and the KSC datasets, and $K=11$, $W=21$ for the Houston 2013 dataset, respectively. For MSF, the spatial parameter $\lambda$ is chosen from the interval $\left\{0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9\right\}$, and the range parameter $\sigma$ is chosen from the interval $\left\{1,2,3,4,5,6,7,8,9\right\}$. For the Indian Pines dataset, $\lambda$ and $\sigma$ are roughly set as $\lambda =0.5$, $\sigma =7$; for the KSC dataset, $\lambda$ and $\sigma$ are roughly set as $\lambda =0.8$, $\sigma =5$; for the Houston 2013 dataset, and $\lambda$ and $\sigma$ are roughly set as $\lambda =0.7$, $\sigma =4$. Specially, the training sets are consistent with Tables 1–3 in the process of selecting the appropriate parameters mentioned previously, and the selection process is further explained in the next section.

Experimental results

Results on Indian Pines dataset

The first experiment is conducted on the Indian Pines data set, and it is commonly adopted and relatively difficult to classify. To analyze the small sample size problem, 1% labeled samples are randomly chosen from each class to compose the training set, and the rest samples are then employed as the testing set. The OA, AA, and κ metrics of different methods are present in Table 4. Note that, to make the comparison impartial, each experiment is executed 10 times, and the average classification accuracies are recorded in Table 4.

Table 4. Accuracy for every class (%), OA (%), AA (%), and κ of various classification methodologies for the Indian Pines image.

Class	SVM	GF- SVM	BEEPS- SVM	DF- SVM	MASR	SSSKRR	EPF- B-g	3D- CNN- LR	LPP- LBP- BLS	LBP- SVM	IMBLBP	MSF
1	35.10	20.38	3.57	94.03	71.32	45.28	96.30	36.42	92.08	86.41	88.68	100.0
2	60.52	63.63	75.41	77.30	79.41	81.96	89.33	57.06	82.18	78.38	8.22	99.80
3	42.01	58.52	68.03	82.83	78.79	72.61	53.60	37.20	72.53	77.92	78.19	99.78
4	36.54	64.68	46.49	83.19	66.75	69.27	17.81	29.48	9.04	76.11	8.76	99.91
5	61.06	60.57	69.92	67.89	85.85	80.12	91.12	71.40	68.74	76.55	76.50	99.52
6	88.69	60.54	91.31	89.27	94.07	89.76	99.33	75.40	84.33	79.49	8.98	99.97
7	53.60	71.20	8.80	95.33	99.20	91.33	98.07	30.12	1.0	83.20	89.00	100.0
8	91.65	93.10	91.03	91.70	99.71	90.88	94.38	86.60	9.79	98.84	98.88	100.0
9	54.74	48.42	82.11	89.47	77.89	82.46	9.00	31.82	66.32	97.89	94.48	100.0
10	52.65	60.35	77.14	78.03	78.23	75.02	63.34	58.50	79.69	79.41	84.03	99.90
11	66.81	80.60	74.16	91.60	92.16	83.89	99.64	61.98	84.11	87.88	9.89	100.0
12	52.13	61.09	59.24	76.52	72.42	72.54	8.51	34.40	72.52	83.39	77.01	97.53
13	97.51	79.14	86.41	95.85	92.63	95.30	1.0	75.32	77.99	75.31	79.81	91.42
14	91.43	85.71	89.95	97.88	97.97	93.00	99.61	87.64	95.16	93.21	88.96	100.0
15	28.14	87.82	56.38	80.32	54.31	75.80	41.14	55.66	81.81	83.88	84.90	96.63
16	36.38	79.36	73.40	64.19	83.62	64.72	95.61	76.20	5.42	61.06	5.21	100.0
OA (%)	65.33	71.72	75.81	85.49	85.62	82.17	85.24	60.81	82.45	83.83	84.60	99.47
AA (%)	59.31	67.19	72.02	84.71	83.52	78.99	81.86	56.57	8.54	82.43	82.72	99.03
κ (%)	60.38	67.67	72.54	83.41	82.77	79.68	82.81	55.12	8.09	81.63	82.46	99.40

Compared with the spectral-based classifier SVM, the accuracy metrics generated by other various spatial-spectral analysis methodologies are significantly better. For instance, GF-SVM offers over 6.39% (0.0729) greater OA(κ) than SVM, and BEEPS-SVM generates over 10.48% (0.1216) greater OA(κ) than SVM. The reason for the high OAs of GF-SVM and BEEPS-SVM is that edge-preserving filtering technology can extract spatial-spectral information of HSI while effectively reducing noise interference. In particular, DF-SVM acquires over 20.16% (0.2303) greater OA(κ) than SVM, mainly because decision-level fusion-based SVM that merges the advantages from a classifier ensemble of spectral features, LBP features, and Gabor features. As can be observed, the gains (in OA, AA, and Kappa) of the MASR are more than 20%, as compared to the SVM. This is due to the fact that the MASR effectively extracts the complementary yet correlated information at multiple scales. From the results, the SSSKRR provides over 16.84% (0.1930) greater OA(κ) than SVM, based on the reason that SSSKRR considers the non-linear separability between various objects by adopting the kernel ridge regression and nonlinear sharing subspace method.

Moreover, EPF-B-g yields superior performance than most of the compared methodologies. This is because post-processing filtering technology which combines both spatial and spectral features is extraordinarily useful for the HSI classification. Note that the classification accuracy of 3D-CNN is lower compared to other classification methods, and the main reason is that the large number of parameters in deep learning model leads to overfitting under the condition of deficient training samples, resulting in the poor classification performance.

In particular, the OA(κ) values of LBP-SVM for the Indian Pines scene, which are 18.50% (0.2125), successively higher than that of SVM. The main reason is that LBP operator encodes the information of image texture configuration while supplying local structure patterns. We also provide the experimental results of LPP-LBP-BLS as comparison. It is worth noting that the IMBLBP is superior to LBP-SVM and LPP-LBP-BLS. Mainly because the spatial features from IMBLBP are more effective for classifier than that of the LBP features, in addition, the feature extraction of HSI using IMBLBP can effectively alleviate this phenomenon of “same class has different spectral information” and “different classes have same spectral information”. By contrast, the proposed MSF framework provides the best classification metrics compared to other state-of-the-art algorithms, i.e. OA = 99.47%, AA = 99.03%, and κ = 99.40%. Among all the 16 land cover classes, MSF carries out the best in 15 classes and surpasses 99% accuracies in 13. The result can be explained that the classification accuracy of classifier can be obviously improved by integrating the spatial correlation among neighboring pixels in the pre-processing and adding smoothing term in the post-processing.

Figure 3 presents the classification maps acquired by different methodologies for the Indian Pines scene. From this figure, we can clearly observe that the classification maps resulting from SVM and 3D-CNN-LR are very bad since lots of noisy evaluations are visible [Figures 3(c)-(j)], where the classification maps are unanimous with the classification results as given in Table 4. By contrast, MASR and SSSKRR methods [Figure 3(g),(h)] produce cleaner classification maps than that from employing spectral features only [Figure 3(c)]. It is also important to notice that the edge-preserving filtering-based methods [Figure 3(d), Figure 3(e)-(i)] can smooth classification maps by characterizing spatial information in contrast to SVM; moreover, the classification map of EPF-B-g contains more homogeneous regions when compared with the GF-SVM and BEEPS-SVM methods. This means that, compared to the GF and BEEPS features, edge-preserving filtering technology as a post-processing approach, is more helpful for the classification task. In addition, the LBP-based methods [Figure 3(f), Figure 3(k), Figure 3(l)-(m)] can provide competitive classification maps due to the property of LBP features, especially in some small local regions, such as the class of alfalfa, the class of grass-pasture-mowed and the class of oats. It can be explained by the fact that LBP is capable of well capturing local structures of HSI. As expected, the classification map from our MSF framework presents the least noise, which is almost close to the ground-truth map [Figure 3(n)-(b)]. As a consequence, we deem that the classification performance of the classifier can be markedly improved by integrating multiscale block LBP features and context-aware information for class labels.

Figure 3. Classification maps on the Indian Pines scene. (a) three-band color composite image; (b) ground truth map, and the classification results generated by the (c) SVM, (d) GF-SVM, (e) BEEPS-SVM, (f) DF-SVM, (g) MASR, (h) SSSKRR, (i) EPF-B-g, (j) 3D-CNN-LR, (k) LPP-LBP-BLS, (l) LBP-SVM, (m) IMBLBP, and (n) MSF.

Results on the KSC dataset

Table 5 provides the statistical metrics of different methods on the KSC dataset. In this section, 6 labeled samples are arbitrarily selected from every class for constructing the training dataset, and the rest of samples are then utilized for the experimental verification. From this Table 5, it can be seen that the accuracy metrics gained by the spatial-spectral classification techniques are markedly better than the pixel-wise classifier SVM (i.e. the GF-SVM, BEEPS-SVM, DF-SVM, MASR, EPF-B-g, and LPP-LBP-BLS methods), and these compared methods achieve over 9% advantage in OA, AA, and κ than that of SVM. In particular, the classification accuracy of 3D-CNN-LR is worst than that of other classification algorithms, mainly because the large number of parameters in deep learning model leads to overfitting under the condition of deficient training samples.

Table 5. Accuracy for every class (%), OA (%), AA (%), and κ of various classification methodologies for the KSC image.

Class	SVM	GF- SVM	BEEPS- SVM	DF- SVM	MASR	SSSKRR	EPF- B-g	3D- CNN- LR	LPP- LBP- BLS	LBP- SVM	IMBLBP	MSF
1	73.51	99.27	99.37	93.62	93.09	100.0	99.92	93.10	89.71	93.74	95.28	100.0
2	61.18	83.89	92.80	86.46	84.81	39.38	1.0	77.88	72.38	87.08	84.58	98.35
3	68.40	72.93	95.50	85.85	96.56	95.53	79.38	64.00	88.48	81.28	84.48	99.94
4	80.89	85.74	81.35	73.38	77.64	13.49	94.84	55.28	85.23	86.19	84.77	98.48
5	30.97	93.26	8.25	81.55	94.45	44.19	98.14	37.38	88.77	78.37	98.67	100.0
6	42.15	83.01	96.03	89.27	92.29	52.69	93.89	58.78	99.37	91.69	93.52	100.0
7	75.76	96.21	1.0	88.72	100.0	98.49	1.0	69.14	1.0	92.41	9.15	100.0
8	74.59	94.70	92.58	92.11	76.61	95.14	99.72	52.00	78.85	84.27	76.18	96.25
9	86.96	92.77	98.85	96.15	97.08	95.43	1.0	71.22	96.73	92.75	91.93	100.0
10	79.15	99.34	96.06	91.93	91.46	81.66	1.0	81.10	92.92	93.81	96.57	100.0
11	89.35	96.79	99.23	96.79	94.72	96.53	1.0	98.14	98.45	95.54	97.39	99.84
12	82.49	95.10	97.21	96.88	80.64	42.82	91.57	75.66	91.19	91.47	93.77	100.0
13	99.89	99.75	1.0	98.20	100.0	98.52	1.0	99.58	83.06	98.52	97.06	100.0
OA (%)	79.19	92.69	95.92	92.02	91.35	80.71	97.56	77.31	88.92	9.77	91.21	99.52
AA (%)	72.71	91.75	94.56	90.07	90.37	73.37	96.73	71.78	89.63	89.78	91.10	99.45
κ (%)	76.87	91.88	95.46	91.12	90.72	78.43	97.29	74.78	87.70	89.73	9.21	99.47

As expected, compared with the basic LBP, the OA(κ) of IMBLBP is increased from 90.77% (0.8973) to 91.21% (0.9021). This means that, by introducing multiscale block and steering kernel, the extracted overall structural information is more useful for classification tasks as compared to the LBP features. Obviously, the proposed MSF outperforms other compared classification algorithms in terms of OA, AA and κ. It signifies that the context-aware information for each class label to the IMBLBP can further enhance classification performance.

Figure 4 provides the classification maps of different methods on the KSC dataset. From the result, we can summarize at least two clear conclusions. Firstly, classification maps produced from the classifier using spatial features [e.g. Figure 4(f), Figure 4(i)(k)] are smoother than that from using spectral features [Figure 4(c)]. Secondly, MSF obtains the best classification map in all comparison methodologies [Figure 4(n)].

Figure 4. Classification maps on the KSC scene. (a) three-band color composite image; (b) ground truth map, and the classification results obtained by the (c) SVM, (d) GF-SVM, (e) BEEPS-SVM, (f) DF-SVM, (g) MASR, (h) SSSKRR, (i) EPF-B-g, (j) 3D-CNN-LR, (k) LPP-LBP-BLS, (l) LBP-SVM, (m) IMBLBP, and (n) MSF.

Results on Houston 2013 dataset

The classification results of all algorithms on the Houston 2013 dataset are presented in Table 6. In this section, five labeled samples are randomly chosen from every class for constructing the training samples, and the remaining samples are adopted to test the model. It can be observed that the performance obtained by the edge-preserving filtering techniques is much better than that with the original spectral features only. For example, GF-SVM offers over 3.5% higher accuracy than SVM, and BEEPS-SVM yields 3.9% higher accuracy than SVM. Moreover, the EPF-B-g achieves much higher classification accuracy than the GF-SVM and BEEP-SVM methods. This means that performing edge-preserving filtering in the post-processing stage is very useful for promoting the classifier performance. From the results, the classification result of the advanced 3D-CNN-LR methodology is not satisfied compared to other algorithms. The reason for the low OA of 3D-CNN-LR is that the spatial features may lead to over fitting phenomenon or even misclassify the image under the condition of limited training samples. Despite LBP being highly discriminative spatial descriptor, the LBP-based algorithms (i.e. the LPP-LBP-BLS, LBP-SVM, and IMBLBP-SVM methods) still yield unsatisfactory performance. Mainly because the adjacent interval of different land-cover classes is relative small and the distribution of the same land-cover class is too scattered for the Houston 2013 image, and this may leads to the presence of highly mixed pixels in boundary region that complicates the classification problem. It is worth noting that the proposed MSF algorithm not only produces the highest OA(94.25%), AA(95.26%), and κ(93.78%), but also most of the classes surpass other methodologies in terms of classification accuracy (e.g. Healthy grass, Stressed grass, Highway, Water, Railway, Parking lot1, and Parking lot2). This means that integrating context-aware information for each class label in the post-processing step is very critical to enhance the classification accuracy of classifier.

Table 6. Accuracy for every class (%), OA (%), AA (%), and κ of various classification methodologies for the Houston 2013 image.

Class	SVM	GF- SVM	BEEPS- SVM	DF- SVM	MASR	SSSKRR	EPF- B-g	3D- CNN- LR	LPP- LBP- BLS	LBP- SVM	IMBLBP	MSF
1	87.21	75.04	87.28	82.43	92.10	80.39	95.38	90.50	39.22	43.84	32.53	95.94
2	86.72	78.42	94.59	87.05	89.86	74.04	97.53	93.46	38.55	48.08	34.05	97.93
3	99.63	98.31	97.16	96.00	98.47	98.32	1.0	86.60	88.66	73.28	89.53	99.89
4	96.35	83.52	95.50	85.08	88.52	79.28	99.98	91.50	45.69	47.44	61.41	95.85
5	86.36	91.38	92.89	88.71	99.09	93.18	1.0	88.86	59.11	74.31	72.38	99.32
6	96.14	93.87	97.54	97.10	78.75	62.88	97.35	83.36	85.91	65.44	89.85	100.0
7	67.79	74.79	78.72	68.99	36.20	72.24	86.17	71.54	69.29	59.53	31.23	83.03
8	83.42	82.53	65.74	78.29	54.59	26.17	37.27	69.30	38.49	58.88	5.96	79.61
9	56.23	69.76	77.10	56.44	67.78	64.41	99.25	57.82	3.27	47.34	19.41	82.62
10	54.14	67.28	53.17	67.03	80.39	54.40	79.33	51.64	88.51	67.19	95.52	99.85
11	51.46	61.64	47.33	56.44	65.22	70.42	96.08	54.84	68.50	76.35	6.81	99.66
12	51.46	64.14	63.06	48.25	43.75	64.20	8.18	48.64	37.78	67.09	49.39	99.40
13	33.65	57.02	61.17	39.35	42.41	78.45	92.07	54.78	86.57	51.17	94.03	99.87
14	86.59	94.46	96.83	88.72	99.62	97.73	1.0	81.38	1.0	99.76	1.0	100.0
15	98.82	99.46	98.33	97.55	100.0	90.54	1.0	96.72	62.12	81.74	73.79	95.97
OA(%)	71.85	75.50	75.77	72.19	74.23	71.37	89.04	70.19	56.90	6.34	57.05	94.25
AA(%)	75.73	79.44	8.43	75.83	72.14	73.78	9.71	74.73	62.58	64.10	63.65	95.26
κ (%)	69.58	73.53	73.82	69.96	75.78	69.12	88.17	67.82	53.49	57.20	53.76	93.78

Figure 5 shows the false-color image, the ground truth, and the classification visualization results of all methodologies on the Houston 2013 dataset. In general, the classification map generated by the SVM presents many noisy scatter points [Figure 5(c)], while the edge preserving filtering features with the SVM classifier mitigate this problem through the elimination of noisy interference phenomenon [e.g. Figure 5(d), Figure 5(e)-(i)]. It is obvious that 3D-CNN-LR, LPP-LBP-BLS, LBP-SVM, and IMBLBP-SVM algorithms produce poor classification maps, which present noises in some categories and show more misclassifications [Figure 5(j), Figure 5(k)-(m)], roughly speaking, the classification maps are unanimous with the classification results as given in Table 6. Specifically, compared to other classical methodologies, there is least noise and best homogeneity in the classification result map obtained by MSF, which is closer to the ground truth distribution map [Figure 5(n)]. In general, the visual comparison results in Figure 5 and quantitative evaluation results reported in Table 6 lead to a similar conclusion, which fully prove the superiority of this proposed method under the condition of small training samples.

Figure 5. Classification maps on the Houston 2013 scene. (a) three-band color composite image; (b) ground truth map, and the classification results obtained by the (c) SVM, (d) GF-SVM, (e) BEEPS-SVM, (f) DF-SVM, (g) MASR, (h) SSSKRR, (i) EPF-B-g, (j) 3D-CNN-LR, (k) LPP-LBP-BLS, (l) LBP-SVM, (m) IMBLBP, and (n) MSF.

Evaluation of the influence of parameters

Effect of the filtering parameters

Before performing the proposed MSF method, the proper parameters of IMBLPB need to be evaluated first. To be fair, the remaining model parameters are fixed apart from $K$ and $W$, moreover, the average OA is reported after repeating 10 times in each experiment. Figure 6 illustrates the average OA versus different numbers of selected bands. It can be observed that the OA of IMBLBP dramatically ascends when the number of bands increases. As the number of bands further increases, the OA of IMBLBP tardily rises or becomes relatively stable. Note that more selected bands will increase the computational complexity and running time. Therefore, the number of selected bands $K$ is roughly set as $K=7$ for the Indian Pines scene and the KSC datasets. For the Houston 2013 dataset, the OA of IMBLBP dramatically ascend when the number of bands increases and then tardily descends with the number of bands further increases. Maybe because more redundant information is generated with the number of bands increases. In this case, $K$ is roughly set as $K=11$ for the Houston 2013 dataset, where the setting provides a good compromise between the calculation time and the classification accuracy.

Figure 6. Analysis of the influence of the number of selected bands on overall accuracy using the IMBLBP method for (a) Indian Pines, (b) KSC, and (c) Houston 2013 datasets.

Figure 7 shows the average OA versus different patch sizes. Similarly, with the increase in patch size $W$, the OA of IMBLBP intensely increases and then become gradually stabilizes. Here, IMBLBP reaches the maximum OA with patch size $W=25$ for the Indian Pines and the KSC datasets. For the Houston 2013 dataset, with the increase in patch size $W$, the OA of IMBLBP intensely increases and then gradually descends. Here, IMBLBP reaches the maximum OA with patch size $W=21$ for the Houston 2013 dataset.

Figure 7. Analysis of the influence of different patch sizes on overall accuracy using the IMBLBP method for (a) Indian Pines, (b) KSC, and (c) Houston 2013 datasets.

Effect of the filtering parameters

Figure 8 shows the OA of the proposed MSF algorithm versus the smoothing parameter $\lambda$ and the range parameter $\sigma$. To avoid any bias, the rest of the model parameters are fixed besides $\lambda$ and $\sigma$, in addition, the classification results are repeated over 10 trials. From Figure 8, we observe that the OA of MSF is sensitive to the parameters $\lambda$ and $\sigma$, and the OA value is not strictly increasing or decreasing, when $\lambda$and $\sigma$gradually increase. Commonly, with the parameters $\lambda$and $\sigma$grow, the OA of MSF dramatically rises and then become declines. In this case, the parameters $\lambda$ and $\sigma$ are roughly selected in this section. For the Indian Pines dataset, we can observe that the MSF algorithm produces the highest OA value when the smoothing parameter is set to $\lambda =0.5$ and the range parameter is set to $\sigma =7$. For the KSC dataset, the MSF algorithm obtains the highest OA value when the smoothing parameter is set to $\lambda =0.8$ and the range parameter is set to $\sigma =5$. For the Houston 2013 dataset, the MSF algorithm obtains the highest OA when the smoothing parameter is set to $\lambda =0.7$ and the range parameter is set to $\sigma =4$.

Figure 8. Analysis of the influence of the range parameter and the spatial parameter on overall accuracy using the MSF method for (a) Indian Pines, (b) KSC, and (c) Houston 2013 datasets.

Investigation on the effect of the training sample size

Figure 9 analyzes the effect of different training sizes on all methods. The training size is randomly chosen from the ranges of [1%, 3%, 5%, 7%, 9%], [3, 6, 9, 12, 15], and [5, 10, 15, 20, 25] for the Indian Pines, KSC, and Houston 2013 datasets, respectively. As shown in Figure 9, as the training sizes increase, the performance of all the compared methods significantly increases. It is obvious that the proposed MSF framework always yields the highest classification accuracy compared with others classification methods. By contrast, the advantage of MSF mainly reflects in the case of insufficient training samples, for instance, 1% ratio of training samples is randomly chosen for the Indian Pines dataset and six training samples per class for the KSC dataset, MSF even acquires over 99.4% and 99.5% overall accuracies, respectively. For the Houston 2013 dataset, the OAs of MSF with different training samples are higher than other classification methods. When the number of training samples per class is only 5, the OA of MSF reaches 94.25%. When the number of training samples per class is 25, the OA of MSF is 99.85%. This verifies that the proposed method can not only generate good classification in small samples but also keep the good classification performance in large samples, which further demonstrates the availability of the proposed method.

Figure 9. Overall accuracy with various numbers of training samples per class for the (a) Indian Pines, (b) KSC, and (c) Houston 2013 datasets.

Conclusion

In this paper, a simple yet efficacious multi-strategy fusion framework has been developed for spatial-spectral HSI classification. The proposed framework aims at extracting multiscale block overall structural information in pre-processing stage, where steering kernel is firstly introduced into MBLBP to enhance the encoding performance. Then, BEEPS is employed to smooth the soft-classified probabilities maps in post-processing stage, and this step aims at optimizing the classification accuracy by concerning context-aware information for each class label. The experimental results on three real hyperspectral datasets demonstrate that the proposed MSF is superior to the comparison methods. Additionally, the major advantage is MSF obtains excellent classification results with different training sizes. Even with a very small training sample size (e.g. 1% ratio of training samples per class for the Indian Pines dataset), the proposed MSF can obtain approximately 13.85% overall accuracy enhancement over the advanced MASR, 14.23% enhancement over EPF-B-g, and 17.02% enhancement over the recently proposed LPP-LBP-BLS. Although the proposed MSF method generates best classification results for datasets in different situations, but there are still some shortcomings.

1. The proposed algorithm only uses the principal component analysis method to reduce the dimensionality of HSI, while ignoring the impact of different transform based or band selection-based dimensionality reduction methods on classification performance.

2. The IMBLBP operator is capable of capturing the spatial correlation among neighboring pixels; however, it cannot fully characterize the joint spectral-spatial structure without considering the cubic structure of HSI.

Therefore, based on the shortcomings listed previously, the focus of our future research is as follows:

1. The ways of reducing the number of spectral channels and improving classification effect through optimizing band selection.

2. How to fully analyzing the cubic structure of HSI and how to fully characterize the joint spectral-spatial structure by utilizing the 3-D LBP model.

Acknowledgments

This research was supported by the Hengyang Normal University Fund Project (2022QD07).

Disclosure statement

No potential conflict of interest was reported by the author(s).