A classification framework with multi-level fusion of object-based analysis and convolutional neural network: a case study for land use classification in mining areas

ABSTRACT

To enhance the accuracy of land use classification in mining areas, the Object-based Convolutional Neural Network (OCNN) method has been widely used. However, existing researches tend to neglect the importance of decision-level fusion, focusing only on feature-level fusion. This study proposes a new classification framework with Multi-level Fusion of Object-based analysis and CNN (MFOCNN) to achieves high-accuracy land use classification in mining areas. First, Simple Linear Iterative Cluster (SLIC) is employed to generate image objects, which serve as the basic unit for classification. Second, an improved DenseNet is proposed to extract deep features from image patches, which represent image objects, and provide the classification result. Third, handcrafted features including spectral, textural and geometric of the image objects are extracted and fused with the deep features to obtain the classification result with random forest classifier. Finally, the Dempster-Shafer (DS) evidence theory is applied to fuse the two previously described classification results at the decision-level to obtain the final result. Experiments conducted in the mining areas of Erdos using Gaofen-6 images demonstrate that the proposed MFOCNN achieves the best visual performance and accuracy among all tested methods. The MFOCNN, with its feature-level fusion and decision-level fusion, significantly improves the accuracy of land use classification in mining areas. The results suggest that the proposed MFOCNN is a promising method for achieving high-accuracy land use classification in mining areas.

KEYWORDS:

• Mining areas
• land use classification
• object-based analysis
• DenseNet

1. Introduction

Mineral resources constitute a vital component of natural resources, and the exploration and utilization of these resources are pivotal for social and economic progress (Xie et al. 2021). Open-pit mining offers favorable production conditions and high mining efficiency (Li, Xing, et al. 2023). However, while open-pit mining promotes regional economic growth, it also disrupts the normal land use landscape, and can potentially trigger geological disasters like landslides and collapses (Li, Xing, et al. 2023; Li, Xu, et al. 2023). As resource extraction continues, the tension between open-pit mining and the preservation of delicate ecosystems becomes increasingly pronounced. China is accelerating the development of green mining and intensifying efforts in ecological preservation (Chen et al. 2020). The implementation of this policy relies on mine environmental monitoring, especially open-pit mines. Land use classification is an important part of mine environmental monitoring. The classification of land use in open-pit mining areas is paramount for gaining a comprehensive understanding of the impact of mining activities on the regional geological environment (Chen et al. 2018b). It not only contributes to a thorough comprehension of the ecological damage process caused by open-pit mining but also aids in the scientific formulation of ecological restoration plans to mitigate such ecological harm (Chen et al. 2022). Moreover, accurate land use data can also facilitate a comprehensive understanding of mining activities and the identification of illegal mining practices (Li, Xing, et al. 2023). In summary, achieving high-accuracy and refined land-use classification in mining areas is of utmost importance (Du, Li, et al. 2022; Du, Xing, et al. 2022).

With the development of earth observation system, the spatial resolution, temporal resolution and spectral resolution of remote sensing images have been continuously improved, making it possible to implement land use classification (Li et al., ”Land Cover Classification from Remote Sensing Images,” 2022; Yu et al. 2022). In recent years, remote sensing images combined with traditional machine learning algorithms including Support Vector Machine (SVM) and Random Forest (RF) have been applied to the study of land use classification in mining areas. For instance, the algorithms of SVM and RF and images of ZiYuan-3 and WorldView-2 were used by Chen et al. (2018a) and Chen, Li, and Wang (2020) to conduct land use classification in several mining areas in China. However, the land use categories in mining areas are complex, with large variations in shape and scale (Figure 1), and traditional machine learning algorithms have difficulties in extracting robust features to represent complex and heterogeneous scenes, resulting in achieving high-accuracy land use classification in mining areas remains a challenge (Du, Xing, et al. 2022; Li, Xing, et al. 2023).

Figure 1. A mining scene. (a) Complex scenes and heterogeneous composition; (b) large variation in shape, scale and spectrum.

With the continuous breakthrough of deep learning technology and the continuous improvement of data processing performance, more and more CNNs are proposed for image classification (e.g. VGGNet and ResNet etc.) (He et al. 2016; Simonyan and Zisserman 2014), semantic segmentation (e.g. U-Net and DeepLabv3+ etc.) (Chen et al. 2018; Ronneberger, Fischer, and Brox 2015; Xu et al. 2022) and object detection (e.g. Faster R-CNN and Yolo etc.) (Chen et al. 2023; Wang, Du, et al. 2022; Wang, Tan, et al. 2022). The application of CNNs has significantly enhanced the accuracy of land use classification across diverse and complex scenes. For instance, in intricate and heterogeneous wetland environments, Jamali et al. (2022) developed a 3D Generative Adversarial Network (3D GAN), achieving high accuracy in large-scale wetland mapping. In urban settings, Cai and Chen (2022) proposed the Multiscale Residual Convolutional Autoencoder (MRCAE), a novel approach for learning potential representations of different urban morphology categories. Li et al., ”Land Cover Classification from Remote Sensing Images,” (2022) addressed issues related to single-scale convolutional kernel limitations by constructing a Multi-Scale Fully Convolutional Network (MSFCN). Similarly, Chen et al. (2021) introduced the Dual Context Prior and Spatial Propagation Network (DSPNet), a simple yet effective network incorporating dual context prior and spatial propagation. Both methods were validated on publicly available datasets, demonstrating their effectiveness. Beyond land use classification, CNNs prove invaluable in tackling special target recognition challenges. The Earthquake-Flood-Fire-Cyclone Network (EFFCNet), for example, excels in high-precision classification and recognition of disasters such as earthquakes, floods, fires, and hurricanes in complex environments (Yuan et al. 2023).

Some published researches have applied CNNs in mining areas, for instance, Du, Xing, et al. (2022) constructed OM-DeepLab by fusing multi-scale low-level and high-level features, which was used for the extraction of open-pit mines. RATT-UNet which combines residual connection, attention mechanism, and U-Net was proposed to extract mine road from remote sensing images (Xiao, Yin, and Fu 2022). Xie et al. (2021) and Zhao et al. (2022) proposed DUSegNet and VoVNet-v2 for the whole mining area extraction, respectively. All of the above studies have achieved good performance. However, Du, Xing, et al. (2022) and Xiao, Yin, and Fu (2022) only conducted extraction experiments for a certain mining area land use category. Whereas Xie et al. (2021) and Zhao et al. (2022) extracted the whole mine area as the target, all the above studies didn’t carry out fine-grained land use (such as open-pit mine, reclamation area, and waste dump) classification experiments within the mining area. Furthermore, some researchers have employed CNNs to achieve land use classification in mining areas. For instance, Chen et al. (2022) introduced a hybrid open-pit mining mapping framework utilizing Gaofen-2 images and an improved U-Net, which successfully achieved fine-grained classification of land use categories in mining areas. Similarly, Zhou et al. (2023) proposed the Edge Feature Enhancement and Graph Convolutional UNet (EG-UNet), incorporating an edge feature enhancement module and long-range information module, which demonstrated outstanding performance on both a constructed mining land use classification dataset and the publicly available Gaofen Image Dataset. Both of the aforementioned studies employed image pixels as the basic unit for land use classification. However, several researches have highlighted the advantages of using image objects as the basic classification unit in comparison to pixels (Liu et al. 2020; Zhang et al. 2018; Zhao, Du, and Emery 2017). These advantages include: 1) The boundaries of image objects fit the real boundaries of ground objects, making land use results based on image objects better match human perception. 2) Merging image pixels into objects according to formulated rules can reduce internal heterogeneity and mitigate possible pepper noise in the classification results. 3) Using objects as classification units during the classification process facilitates the extraction of handcrafted features, such as geometric features, which have proved to be useful in land use classification.

Therefore, the combination of object-based analysis and CNN is of great practical importance. The Object-based CNN (OCNN) has been proposed and applied in this context, which improves the classification accuracy and efficiency of remote sensing image (Zhang et al. 2018). The published researches about OCNN mainly adopts the following three modes for land use classification. The first one is to directly extract deep features of image objects and classify using CNNs, for instance, Zhang et al. (2018) used the CNN and remote sensing images for urban land use classification. The second one involves calculating handcrafted features such as spectral, texture, and geometric features from segmented remote sensing images. These calculated features are then directly input into the constructed CNNs to conduct land use classification. For instance, Wang, Chen, and Plaza (2023) incorporated these handcrafted features into the Mining Feature-Enhanced Resnet (MFE-ResNet), effectively enhancing the accuracy of land use classification in open-pit mining areas. The last one is to first extract the deep features of the image objects using CNNs, and then fused with the handcrafted features, finally feeds the fused features to the traditional machine learning classifier, such as RF for land use classification. For instance, Zhao, Du, and Emery (2017) and Liu et al. (2020) first fused deep features and handcrafted features, and then feed the fused features into machine learning classifiers to achieve high-accuracy urban land use classification. All modes have good performance in land use classification tasks, however, existing studies still suffer from the following problems.

1. Multi-scale segmentation algorithms are widely used to segment images to generate image objects, however CNN requires a fixed-size image patch as input, the image patch captured from segmented images are difficult to represent the image object effectively.

2. CNN have shown excellent performance in feature extraction and classification. However, in mining scenes with highly heterogeneous, the generalization of CNN needs to be improved due to the large variations in shape and scale of land use categories.

3. Existing OCNN methods usually focus only on feature-level fusion, ignoring decision-level fusion, which can fuse different levels of complementary information to improve the accuracy of land use classification.

Given these issues and challenges, a classification framework with Multi-level Fusion of Object-based analysis and CNN (MFOCNN) is proposed and applied to fine land use classification in mining areas. Experimental results show that this framework outperforms benchmark image classification networks and semantic segmentation networks, with the best quantitative assessment result and visual effect.

The contributions of the study can be summarized as follows:

1. The Simple Linear Iterative Cluster (SLIC) is employed to segment images and generate image objects. Because, the boundaries of the image objects generated by SLIC match the real boundaries of the ground objects and the captured image patches better represent the image objects.

2. An improved DenseNet is proposed for extracting robust deep features to conquer the problem of heterogeneity in mining areas.

3. MFOCNN is constructed to fuse classification information in multiple levels, thus improving classification accuracy. The framework not only provides the technical support for land use classification in mining areas, but also has a wide range of generalizability.

2. Methods

The proposed framework consists of three parts, which is illustrated in Figure 2.

Figure 2. Flowchart of the proposed MFOCNN.

1. Image segmentation: SLIC is employed to segment images into image objects considering that SLIC has good results in ground object contour preservation and that the image objects generated by SLIC are more suitable as input to CNN than multi-scale segmentation algorithm.

2. Feature extraction and fusion: First, the image object is represented by a fixed-size image patch, and then the improved DenseNet is used to extract deep features and obtain classification result (Classification result based on improved DenseNet, result₁). Second, the handcrafted features of the image object, including spectral, textural and geometric features, are extracted and fused with deep features. Finally, the fused features are fed into the RF classifier to obtain classification result (Classification result based on fused feature-random forest, result₂).

3. Decision-level fusion of classification results: DS evidence theory is applied to fuse result₁ with result₂ at the decision-level to obtain the high-accuracy classification result (result₃), as it has better application in fusing the results of multi-classifiers.

2.1. SLIC for image segmentation

The multi-scale segmentation algorithm is widely employed to segment images into image objects which match the boundaries of the ground objects (Figure 3(a)) (Liu et al. 2020). However, CNN requires a fixed-size image patch as input (He et al. 2015), the image patch captured from segmented image is difficult to represent the image object effectively. For instance, the image object is too small, so that the image patch covers a large area of non-study image objects (Figure 3(c)) or the image object is too large for the image patch to cover the full image object (Figure 3(d)). Given this problem, SLIC is introduced to replace the multi-scale segmentation algorithm to generate image objects that both match the boundaries of the ground objects and are uniform in shape and scale (Figure 3(b)), aiming to solve the problem that fixed-size image patches fail to effectively represent the image objects (Figures 3(e,f) (Jing et al. 2020).

Figure 3. Image segmentation result. (a) Result of multi-scale segmentation, (b) result of SLIC, (c) and (d) image patches are too small or too large to effectively represent the image objects in existing OCNN method, (e) and (f) image patches can effectively represent the image objects.

SLIC is a K-Means based super pixel segmentation method proposed by Achanta (Achanta et al. 2012). The principle of SLIC is to first select the seed points for clustering based on the number of objects expected to be segmented. Second, each pixel in the domain of the seed point is determined to belong to which cluster center, finally clustering and iterating until the segmentation result does not change, thus clustering the image into a number of super pixel objects with similar texture, color features, etc (Li, Li, et al. 2022). For details about SLIC, please refer to the study (Achanta et al. 2012).

2.2. Classification based on improved Densenet

The DenseNet was proposed by Huang et al. in 2017 (2017). DenseNet has been used in land use classification and has achieved good performance in practical applications (Cai et al. 2021; Su et al. 2022). The basic module of DenseNet is dense block. The output features of any two layers in the Dense Block are connected via skip connection, which facilitates the propagation and reuse of features and also helps to alleviate the problem of gradient vanishing (Huang et al. 2017).

Although DenseNet performs well in many application scenes, it still falls short in land use classification of mining areas where the scene is complex and heterogeneous, as demonstrated by: (1) The size and shape of the standard convolutional kernel used in DenseNet are fixed, which limits the ability of the deep features to represent ground objects with large variation in size and shape (Zuo et al. 2022); (2) Each pixel in the image patch has a different effect on whether the image object can be correctly classified. Therefore, if the deep features are extracted indiscriminately at each pixel location, the invalid information acquired will interfere with the accuracy of the classification result (Li et al., “Semantic Segmentation of Remote Sensing Images,” 2022). To address the above-mentioned problems, deformable convolution and attention mechanism are introduced to improve DenseNet (Dai et al. 2017; Woo et al. 2018). The structure of the improved DenseNet is shown in Figure 4.

Figure 4. The structure of the improved DenseNet.

As is shown in Figure 5, the deformable convolution adds a direction vector to the weight of each convolution kernel on the basis of the standard convolution operation, so that the shape of the convolution kernel can change with the actual shape of the ground object (Dai et al. 2017).

Figure 5. The structure of deformable convolution.

The eigenvalue calculation method of a certain position of the output feature of the deformable convolution can be defined by Equation (1)(1) $\mathit{f}{}_{\mathit{out}}\left(p_0\right)=\sum _{p_n\in \mathit{R}}\mathit{w}\left(p_n\right)\ast \mathit{f}{}_{\mathit{in}}\left(p_n+p_0+\mathrm{\Delta }{p}_n\right)$(1) .

(1) $\mathit{f}{}_{\mathit{out}}\left(p_0\right)=\sum _{p_n\in \mathit{R}}\mathit{w}\left(p_n\right)\ast \mathit{f}{}_{\mathit{in}}\left(p_n+p_0+\mathrm{\Delta }{p}_n\right)$(1)

where, $\mathit{R}$ is the spatial position of each weight in the convolution kernel relative to the central position weight, and $\mathit{R}\in \left\{\left(-1,-1\right),\left(-1,0\right),\dots \left(0,1\right),\left(1,1\right)\right\}$; $\mathit{w}$ is the convolution kernel; $f_{\mathit{in}}$ is the input feature; $f_{\mathit{out}}$ is the output feature; $p_0$ is the position to be calculated in the input feature; $\mathrm{\Delta }{p}_n$ is the offset of each weight in the convolution kernel.

The attention mechanism helps guide DenseNet to enhance concentration on important pixels and important channels of the features, therefore the Convolutional Block Attention Module (CBAM) is introduced in this study (Woo et al. 2018). As is shown in Figure 6, the CBAM consists of channel attention module and spatial attention module.

Figure 6. The structure of CBAM.

CBAM is defined in Equations (2)(2) $w_c=\mathit{\sigma }\left(\mathit{MLP}\left(\mathit{AvgPool}\left(\mathit{f}{}_{\mathit{in}}\right)\right)+\mathit{MLP}\left(\mathit{maxPool}\left(\mathit{f}{}_{\mathit{in}}\right)\right)\right)$(2) –(5(5) $\mathit{f}{}_{\mathit{s}\_\mathit{out}}=\mathit{f}{}_{\mathit{c}\_\mathit{out}}\times w_s$(5) ):

(2) $w_c=\mathit{\sigma }\left(\mathit{MLP}\left(\mathit{AvgPool}\left(\mathit{f}{}_{\mathit{in}}\right)\right)+\mathit{MLP}\left(\mathit{maxPool}\left(\mathit{f}{}_{\mathit{in}}\right)\right)\right)$(2)

(3) $\mathit{f}{}_{\mathit{c}\_\mathit{out}}=\mathit{f}{}_{\mathit{in}}\times w_c$(3)

(4) $w_s=\mathit{\sigma }\left(\mathit{Conv}\left(\mathit{Cat}\left(\mathit{AvgPool}\left(f_{\mathit{c}\_\mathit{out}}\right),\mathit{maxPool}\left(f_{\mathit{c}\_\mathit{out}}\right)\right)\right)\right)$(4)

(5) $\mathit{f}{}_{\mathit{s}\_\mathit{out}}=\mathit{f}{}_{\mathit{c}\_\mathit{out}}\times w_s$(5)

where, $\mathit{\sigma }\left(\cdot \right)$ denotes sigmoid function; $\mathit{MLP}\left(\cdot \right)$ denotes Multi-layer Perceptron (MLP) function; $\mathit{AvgPool}\left(\cdot \right)$ denotes global average pooling function; $\max \mathit{Pool}\left(\cdot \right)$ denotes global max average pooling function; $\mathit{Cat}\left(\cdot \right)$ denotes feature concatenated in the channel dimension; $\mathit{Conv}\left(\cdot \right)$ denotes convolution operation; $w_c$ is the channel attention weight; $w_s$ is the spatial attention weight; $f_{\mathit{in}}$ represent input feature, respectively. $f_{\mathit{c}\_\mathit{out}}$ and $f_{\mathit{s}\_\mathit{out}}$ represent the output feature of the channel attention module and spatial attention module, respectively.

As is shown in Figure 4. The image patch captured from segmented images is fed into the improved DenseNet, and then the deep features adapted to the heterogeneous scene of the mining area are extracted through operations including Dense Block, deformable convolution, and CBAM. In addition, the fully connected layer at the end of the improved DenseNet is employed to get the result₁.

2.3. Classification based on feature-level fusion

Several published researches have demonstrated that fusing handcrafted features and deep features can improve the accuracy of land use/land cover classification (Ge, Xie, and Meng 2022; Guo et al. 2021; Qu et al. 2022). Therefore, in this study, handcrafted features including spectral, textural and geometric of the image objects are designed and extracted for land use classification in the mining area (Zhao et al. 2017). Among them, the extraction of texture features and geometric features also contributes to the reduction of classification errors caused by the presence of shadows in the images (Ahamad et al. 2022). The feature extraction process involves several steps: First, it relies on Python, leveraging libraries such as Scikit-Image and the Geospatial Data Abstraction Library (GDAL) to perform SLIC on the image and obtain boundary vectors. Second, both the images and boundary vectors are imported into eCognition software. Third, multi-scale segmentation is executed within eCognition to obtain image objects, using the boundary vectors as constraints. It is important to note that when employing these vectors as constraints, the results of multi-scale segmentation are consistent with the results of SLIC. Finally, spectral, textural, and geometric features of the image objects are calculated within eCognition (Chen et al. 2020). After feature extraction, the most suitable feature combination is determined with the Feature Space Optimization function of eCognition (Wang, Chen, and Plaza 2023). As is shown in Table 1, a total of 70 handcrafted features are selected, including 22 spectral features, such as NDVI and NDWI. And 40 texture features, which are calculated based on the Gray-Level Co-occurrence Matrix (GLCM). The geometry features are 8 in total, such as boundary length and rectangular fit.

Table 1. The handcrafted features.

Category	Name	Number	Name	Number
Spectral features	Mean	4	Ratio	4
	Standard deviations	4	Skewness	4
	Normalized Difference Vegetation Index (NDVI)	1	Saturation	1
	Normalized Difference Water Index (NDWI)	1	Intensity	1
	Brightness	1	Hue	1
Textural features	GLCM(Homogeneity)	5	GLCM (Ang. 2nd moment)	5
	GLCM(Contrast)	5	GLCM (Correlation)	5
	GLCM(Dissimilarity)	5	GLCM (Mean)	5
	GLCM(Entropy)	5	GLCM (Standard deviations)	5
Geometric features	Area	1	Length/width	1
	Boundary length	1	Compactness	1
	Length	1	Shape index	1
	Width	1	Rectangular Fit	1

These handcrafted features are fused with the deep features extracted by the improved DenseNet, and the fused features can be calculated from the equation: $f_f=\mathit{Cat}(f_h,f_d)$, where, $f_h$ represents the handcrafted features and $f_d$ represents the deep features.

RF is an ensemble learning method for classification, which utilizes multiple decision trees to train and predict samples, and then, all trees vote for the output class (Li, Jing, and Wang 2021). Due to the higher generalization ability, RF are widely used in land use classification. In this study, the fused features are fed into the RF classifier to obtain result₂.

2.4. Classification based on decision-level fusion

To fully combine the complementary information of multiple classifiers, DS evidence theory is employed to fuse result₁ with result₂ at the decision-level to obtain high-accuracy land use classification result of mining areas (result₃).

DS evidence theory is usually used to deal with uncertainty problems, which can integrate the classification results of different experts or classifiers at the decision-level (Du et al. 2021; Shafer 1992). The classification probabilities of each classifier for each category of ground objects can be obtained from Section 2.2and Section 2.3 , and then the final result of DS evidence theory can be calculated by Equation (6)(6) $p_{\mathit{r}3}\left(\mathit{i}\right)=\frac{{\mathit{p}}_{\mathit{r}1}\left(\mathit{i}\right)\times p_{\mathit{r}2}\left(\mathit{i}\right)}{{\sum }_j^n\left(p_{\mathit{r}1}\left(\mathit{j}\right)\times p_{\mathit{r}2}\left(\mathit{j}\right)\right)}$(6) .

(6) $p_{\mathit{r}3}\left(\mathit{i}\right)=\frac{{\mathit{p}}_{\mathit{r}1}\left(\mathit{i}\right)\times p_{\mathit{r}2}\left(\mathit{i}\right)}{{\sum }_j^n\left(p_{\mathit{r}1}\left(\mathit{j}\right)\times p_{\mathit{r}2}\left(\mathit{j}\right)\right)}$(6)

where, $p_{\mathit{r}1}\left(\mathit{i}\right)$, $p_{\mathit{r}2}\left(\mathit{i}\right)$, $p_{\mathit{r}3}\left(\mathit{i}\right)$ denote the classification probabilities of the i-th category in result₁, result₂ and result₃, respectively; $\mathit{n}$ is the number of land use categories.

3. Experiments and results

3.1. Study area and dataset

The study area is located in the coal mining areas of Erdos, Inner Mongolia Autonomous Region, China, covering an area of 101 km². The Gaofen-6 satellite was launched on 2 June 2018, and was the sixth satellite of the China High-resolution Earth Observation System (CHEOS). As an optical remote sensing satellite, Gaofen-6 is carrying the Panchromatic & Multispectral Scanners (PMS) and the Wide Field of View Multispectral Cameras (WFV). The used images in this study were acquired by PMS on 10 December 2019, and collected from the Land Observation Satellite Service (https://data.cresda.cn/). The spatial resolution of the Panchromatic (PAN) image of the Gaofen-6 is 2 m, and the spatial resolution of the Multispectral (MS) images is 8 m. The preprocessing steps of Gaofen-6 images include radiometric calibration, atmospheric correction, geometric correction, orthorectification, and pansharpening fusion of PAN and MS images, using the PIE-Basic 6.3 (Zhang et al. 2023). The fused image includes 4 bands of R, G, B, and NIR, with a spatial resolution of 2 m. The location and the images coverage of the study area are shown in Figure 7.

Figure 7. The study area.

After field investigation, six land use categories are determined, including Open-pit Mine (OM), Waste Dump (WD), Reclamation Area (RA), Building (BD), Natural Vegetation (NV) and Bare Soil (BS). Samples are produced through manual visual interpretation, followed by dividing the samples into training set, validation set and testing set. The number of samples is shown in Table 2.

Table 2. Number of samples.

Category	Training Set	Validation Set	Testing Set	Sum
OM	331	149	116	596
WD	218	100	108	426
RA	240	122	107	469
BD	200	99	120	419
NV	344	160	175	679
BS	201	101	102	404
Sum	1534	731	728	2993

3.2. Implementation details

The experiments are implemented on a desktop computer with Intel Xeon Gold 5118, 16.0 GB memory, and NVIDIA GeForce RTX2080Ti. All algorithms are performed using Python language (v3.6.5) on the PyCharm platform (Community Edition 2020.2.1 × 64). The deep learning framework is pytorch (v1.7.0 + cu110). To obtain the optimal classification result, the method of manual parameter tuning is employed to search for the optimal hyperparameters (Du, Li, et al. 2022). The hyperparameters of the MFOCNN are shown in Table 3.

Table 3. The hyperparameters of MFOCNN.

Method	Parameters
SLIC	n_segments: 120,000; compactness: 1; sigma: 10，
Improved DenseNet	Patch size: 64 × 64; Epoch: 300; Loss function: Cross-Entropy loss; Optimizer: SGD; Learning rate: 0.0001; Batch size: 16
RF	n_estimators:40; Maximum depth: 100

To evaluate the performance of the classification result quantitatively, F1– score, Overall Accuracy(OA) and Kappa are introduced in this study. F1– score is used to evaluate the classification effectiveness of individual categories, Overall Accuracy and Kappa are used to evaluate the overall classification effectiveness (Wang et al. 2023; Yang et al. 2022). The three indicators are defined in Equations (7)(7) $\mathit{F}1-\mathit{score}=\frac{2\times \mathit{UA}\times \mathit{PA}}{\mathit{UA}+\mathit{PA}}\times 100\mathrm{\%}$(7)  ~ (9(9) $\mathit{Kappa}=\left(\mathit{OA}-p_e\right)/\left(1-p_e\right)$(9) ).

(7) $\mathit{F}1-\mathit{score}=\frac{2\times \mathit{UA}\times \mathit{PA}}{\mathit{UA}+\mathit{PA}}\times 100\mathrm{\%}$(7)

where $\mathit{UA}$ means User Accuracy and can be calculated by $\sum _{\mathit{i}=1}^n{x}_{\mathit{ii}}/\mathit{R}$; $\mathit{PA}$ means Producer Accuracy and can be calculated by $x_{\mathit{ii}}/\sum _{\mathit{j}=1}^n{x}_{\mathit{ij}}$; $x_{\mathit{ii}}$ denotes the number of samples correctly classified as category $\mathit{i}$. $\sum _{\mathit{j}=1}^n{x}_{\mathit{ji}}$ denotes the number of samples classified as category $\mathit{i}$.

(8) $\mathit{OA}=\left(\sum _{\mathit{i}=1}^n{x}_{\mathit{ii}}/\mathit{R}\right)\times 100\mathrm{\%}$(8)

where $\mathit{n}$ is the number of land use categories; $\mathit{R}$ is the number of samples used for accuracy evaluation;

(9) $\mathit{Kappa}=\left(\mathit{OA}-p_e\right)/\left(1-p_e\right)$(9)

where $p_e=\sum _{\mathit{i}=1}^n\left(\sum _{\mathit{j}=1}^n{x}_{\mathit{ji}}\times \sum _{\mathit{j}=1}^n{x}_{\mathit{ij}}\right)/R^2$.

3.3. Experimental results

The proposed MFOCNN has three classification results, i.e. result₁, result₂ and result₃. To evaluate the effectiveness of the proposed method, the three results are compared and analyzed. In addition, the classification result of DenseNet (result_DN) are also added to verify the performance of the improved DenseNet.

Figure 8 shows the four results. In general, the four results differ significantly. For region1, the NV located underneath the BD is misclassified as OM in result_DN and result₁, whereas this problem is mitigated in result₂ and result₃. The problem of misclassified NV as OM also exists in region2 for all results. However, the area of misclassified regions is significantly smaller in result₃. The problem of misclassified NV as RA exists in result_DN, result₁, and result₂, yet it is alleviated in result₃ (region3). There are also problems in result₃, for example, some holes in the classification result of OM (region4).

Figure 8. Classification visualization results of different comparison methods.

The quantitative evaluation results are shown in Table 4. The $\mathrm{F}1-\mathrm{score}$ of each category in result₃ is the highest, with 95.69%, 97.67%, 82.86%, 98.33%, 93.53%, and 96.59%, respectively. Result₁ and result₂ each have three land use categories with higher $\mathrm{F}1-\mathrm{score}$ than the other. Only the $\mathrm{F}1-\mathrm{score}$ of the RA in result₁ is lower than that in result_DN. The $\mathrm{OA}$ and $\mathrm{Kappa}$ of the four results showed a trend of increasing, with result₃ achieving the highest $\mathrm{OA}$ and $\mathrm{Kappa}$ of 95.6% and 0.95 respectively.

Table 4. Quantitative evaluation results of different comparison methods.

	$\mathit{F}1-\mathit{score}\left(\mathrm{\%}\right)$						$\mathit{OA}\left(\mathrm{\%}\right)$	$\mathit{Kappa}$
	OM	WD	RA	BD	NV	BS
resultDN	94.51	96.3	86.67	96.58	89.89	92.61	92.58	0.91
result1	95.2	94.34	88.89	97.52	93.22	94.58	93.95	0.94
result2	94.12	95.81	91.16	98.33	92.84	93.6	94.23	0.94
result3	95.69	97.67	92.86	98.33	93.53	96.59	95.60	0.95

Note: The best results are highlighted in bold.

In summary, the improved DenseNet is more suitable for land use classification in mining areas than DenseNet, and the generalization ability of the model has improved considerably by incorporating the CBAM and deformable convolution. The result₂, which fused handcrafted features, has a further improvement compared to result₁. The DS evidence theory incorporates the advantages of both classification results, resulting in the highest accuracy in result₃.

4. Discussion

4.1. Comparison of different segmentation algorithms

To evaluate the effectiveness of multi-scale segmentation algorithm and SLIC for land use classification in mining areas, comparative experiments based on the two segmentation algorithms are conducted using the DenseNet. The method of manual parameter tuning is employed to search for the suitable segmentation parameters. The parameters of multi-scale segmentation are set as follows: scale parameter: 20, shape index: 0.5 and compactness: 0.5, the parameters of SLIC are listed in Table 3. The results obtained by multi-scale segmentation and SLIC are defined as result_MSS and result_DN (the same as result_DN in Section 3.3).

The result_MSS and result_DN are shown in Figure 9. Both the two results misclassified NV as OM (region1 and region2) and also misclassified NV as RA (region3 and region5). However, the area of misclassified regions in result_DN is smaller in comparison.

Figure 9. Classification visualization results based on different segmentation algorithms.

The quantitative evaluation results are shown in Table 5. The $\mathrm{F}1-\mathrm{score}$ for each category in result_DN is higher than that in result_MSS, except for RA and BS. The $\mathrm{OA}$ and $\mathrm{Kappa}$ of result_MSS is 1.36% points and 0.02 higher than that of result_DN. In summary, the SLIC is more suitable for land use classification in mining areas.

Table 5. Quantitative evaluation results based on different segmentation algorithms.

	$\mathit{F}1-\mathit{score}\left(\mathrm{\%}\right)$						$\mathit{OA}\left(\mathrm{\%}\right)$	$\mathit{Kappa}$
	OM	WD	RA	BD	NV	BS
resultMSS	92.45	93.15	86.96	96.45	87.89	92.68	91.22	0.89
resultDN	94.51	96.30	86.67	96.58	89.89	92.61	92.58	0.91

Note: The best results are highlighted in bold.

4.2. Comparison of benchmark classification networks

To evaluate the effectiveness of the improved DenseNet, experiments are conducted to compare the proposed improved DenseNet with benchmark classification networks including ResNet50 (He et al. 2016), VGGNet (Simonyan and Zisserman 2014), and GoogLeNet (Szegedy et al. 2015).

The results of the four methods i.e. result_RSN, result_VGG, result_GGN, and result₁ (the same as result₁ in Section 3.3) are shown in Figure 10. result_RSN, result_VGG and result_GGN all have the problem of misclassified BD as OM, but not in result₁ (region6). The results of the OM in result₁ are neatly distributed with fewer holes than the other results (region7 and region8), and have fewer regions of NV misclassified as RA (region3).

Figure 10. Classification visualization results based on different classification networks.

The quantitative evaluation results are shown in Table 6. The $\mathrm{F}1-\mathrm{score}$ for each category in result₁ is higher than the other results, except for RA and BD. The $\mathrm{OA}$ and $\mathrm{Kappa}$ of result₁ is significantly higher than the other results. Among the remaining three results, result_GGN has the highest $\mathrm{OA}$ and $\mathrm{Kappa}$. In summary, the improved DenseNet outperformed GoogLeNet, ResNet and VGGNet.

Table 6. Quantitative evaluation results based on different classification networks.

	$\mathit{F}1-\mathit{score}\left(\mathrm{\%}\right)$						$\mathit{OA}\left(\mathrm{\%}\right)$	$\mathit{Kappa}$
	OM	WD	RA	BD	NV	BS
resultRSN	82.73	92.75	77.59	88.00	82.18	83.08	84.07	0.81
resultVGG	79.23	90.73	81.39	90.83	79.66	82.35	83.52	0.80
resultGGN	86.42	91.18	91.15	97.87	91.17	92.39	91.62	0.89
result1	95.20	94.34	88.89	97.52	93.22	94.58	93.95	0.94

Note: The best results are highlighted in bold.

4.3. Comparison of benchmark semantic segmentation networks

To further evaluate the effectiveness of the proposed MFOCNN, experiments are conducted with the benchmark semantic segmentation networks, including FCN (Shelhamer, Long, and Darrell 2017), U-Net (Ronneberger, Fischer, and Brox 2015) and DeepLabv3+ (Chen et al. 2018).

The classification results of the four methods are shown in Figure 11. Overall, there are more fragmented spots in result_FCN and result_DLb. In terms of detail, the BD in result_FCN, result_DLb, and result_UNet have problems with misclassification (region1), and the OM have problems with broken boundaries and internal holes (region7 and region8). Compared to the other three results, the RA and NV in result₃ are best classified (region9).

Figure 11. Classification visualization results based on different semantic segmentation networks.

The quantitative evaluation results are shown in Table 7. The OM and RA in result_UNet are 2.81 and 6.69% points higher compared to result₃, respectively, with the remaining categories achieving the highest values for result₃. Result₃ achieved the highest $\mathit{OA}$ and $\mathit{Kappa}$ of 95.60% and 0.95, respectively, which are 8.54% points and 0.11 higher than the sub-optimal FCN. Therefore, the proposed MFOCNN outperformed the benchmark semantic segmentation networks.

Table 7. Quantitative evaluation results based on different semantic segmentation networks.

	$\mathit{F}1-\mathit{score}\left(\mathrm{\%}\right)$						$\mathit{OA}\left(\mathrm{\%}\right)$	$\mathit{Kappa}$
	OM	WD	RA	BD	NV	BS
resultFCN	87.78	90.00	89.92	86.25	83.07	91.88	87.06	0.84
resultUNet	98.50	96.18	99.55	79.56	88.51	90.05	82.81	0.78
resultDLb	80.58	85.34	71.54	65.16	69.33	67.83	73.22	0.66
result3	95.69	97.67	92.86	98.33	93.53	96.59	95.60	0.95

Note: The best results are highlighted in bold.

4.4. Comparison of different segmentation parameters

Segmentation parameters have a direct effect on the size and shape of the image objects, which in turn affects the accuracy of classification result. To explore the effect of segmentation parameters on classification result, the most important segmentation parameter of SLIC, i.e. n_segments, is discussed and analyzed.

It can be seen from Figure 12 that $\mathrm{OA}$ and $\mathrm{Kappa}$ show a trend of increasing and then decreasing as the number of n_segments increase, with $\mathrm{OA}$ of all the results above 91% and $\mathrm{Kappa}$ of all results above 0.9, indicating the robustness of the proposed MFOCNN. The $\mathrm{OA}$ and $\mathrm{Kappa}$ achieved the highest value of 95.6% and 0.95, respectively when n_segments is set to 120,000. In summary, n_segments being set to 120,000 is a good choice for the experiment data in this study. Additionally, it’s important to take into consideration that the optimal n_segments for a given study may vary due to differences in image resolution and study area across different studies.

Figure 12. The $\mathrm{OA}$ and $\mathrm{Kappa}$ of different n_segments.

4.5. Advantages and possible improvements

Compared to existing OCNN modes, the proposed MFOCNN has the following improvements. First, the SLIC is employed to generate image objects, which are more suitable for land use classification than the multi-scale segmentation algorithm used in previous researches. Second, the improved DenseNet utilized in the MFOCNN can extract deep features with greater robustness. Third, the handcrafted features and deep features are fused at the feature-level, and the results of the two OCNN modes are fused at the decision-level, which enables multi-level fusion and better utilization of complementary classification information.

Mining scenes present complex, heterogeneous, and diverse environments (see Figure 1), which make land use classification challenging. However, the experiments show that the proposed MFOCNN outperforms existing methods in mining area land use classification. In theory, the method has the potential for application in other natural scenes, such as cities (Du, Xing, et al. 2022).

There is still potential to further improve the classification accuracy. First, handcrafted features and deep features represent different levels of information, and the method used in this study, direct fusion through channel concatenation, may not be optimal for fully utilizing these features. More effective feature fusion methods, such as Feature Pyramid Networks, will be explored in our research. Second, there are still instances of misclassification or holes in the classification results, indicating that further optimization is needed (Li, Xing, et al. 2023). In future research, the classification results will be improved by utilizing spatial topological relationships between land use categories and morphological algorithms. Last but not least, due to the rugged terrain and complex land use categories in the mining area, shadows are inevitable in remote sensing images. While compared to result_DN, the proposed MFOCNN partially mitigates this error through the incorporation of handcrafted features and multi-level fusion module, the challenge persists (Figure 13). In future research, the fusion of high accuracy Digital Elevation Model (DEM) and contemporaneous multisource images to compensate for the spectral values of shadows will be considered.

Figure 13. Classification results in shadow areas in result_DN and result₃, the yellow circled area in region 10 is shadow.

5. Conclusions

In this study, the proposed MFOCNN is applied to land use classification in mining areas, and the experiments show the best quantitative assessment result and visual effect. The conclusions of the study can be summarized as follows:

1. The SLIC can segment images into image objects with uniform shape and size which can be conveniently represented by fixed-size image patches, thus these image objects are suitable for input into CNN network that requires fixed-size image patches.

2. The generalization ability of the DenseNet has improved considerably by incorporating the CBAM and deformable convolution. The fusion of deep features with handcrafted features can further improve the robustness of the features. Finally, decision-level fusion is performed using DS evidence theory can fully combine the complementary information of multiple classifiers to further improve the accuracy of land use classification in mining areas.

3. It is demonstrated that the proposed MFOCNN exhibits satisfactory robustness and can achieve better classification results compared to benchmark classification networks and semantic segmentation networks.

In the future, efforts should focus on enhancing land use classification accuracy in mining areas, including the innovation of feature fusion methods for more efficient feature utilization, optimization of classification results in post-processing, and mitigation of classification errors caused by the presence of shadows.

Acknowledgements

The authors thank China Centre for Resources Satellite Data and Application (Land Observation Satellite Service, https://data.cresda.cn/) for providing us with high-resolution Gaofen-6 images. The authors also thank Bo Liu from Peking University for assisting with data analysis.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The remote sensing image used in this study can be acquired from the following link: https://figshare.com/s/0cd166e71ac14e73a0bc. The codes using python language can be founded from the following link: https://figshare.com/s/b1d5079b9859321dcf88.

Additional information

Funding

This work was supported by the National Natural Science Foundation of China [grant number 42271480], the Fundamental Research Funds for the Central Universities [grant numbers 2022YQDC08, 2022XJDC02, and BBJ2023026], the Open Foundation of the Key Laboratory of Coupling Process and Effect of Natural Resources Elements [grant number 2022KFKTC001].

Notes on contributors

Jianghe Xing

Jianghe Xing is pursuing his PhD degree in surveying and mapping science and technology at China University of Mining and Technology-Beijing. His research interest is land use/land cover classification in mining areas.

Zhenwei Li

Zhenwei Li is pursuing his Master’s degree in Geodesy and Surveying Engineering at China University of Mining and Technology-Beijing. His research interest is land use/land cover classification in mining areas.

Jun Li

Jun Li received his PhD from Peking University, China. Currently he is a professor and the associate dean of the College of Geoscience and Surveying Engineering at China University of Mining and Technology-Beijing. His research interests are land spatial planning and intelligent computing and natural resource monitoring and evaluation.

Shouhang Du

Shouhang Du received his PhD from Peking University, China. Currently he is a lecturer at College of Geoscience and Surveying Engineering, China University of Mining and Technology-Beijing. His research interest is intelligent interpretation of remote sensing images based on deep learning.

Wei Li

Wei Li is pursuing his Master’s degree in Geodesy and Surveying Engineering at China University of Mining and Technology-Beijing. His research interests are ground objects change detection and extraction.

Chengye Zhang

Chengye Zhang received his PhD from Peking University, China. Currently he is an associate professor at College of Geoscience and Surveying Engineering, China University of Mining and Technology-Beijing. His research interests are geographic artificial intelligence technology and application.