A classification framework with multi-level fusion of object-based analysis and convolutional neural network: a case study for land use classification in mining areas

Figures & data

Figure 1. A mining scene. (a) Complex scenes and heterogeneous composition; (b) large variation in shape, scale and spectrum.

Figure 2. Flowchart of the proposed MFOCNN.

Figure 3. Image segmentation result. (a) Result of multi-scale segmentation, (b) result of SLIC, (c) and (d) image patches are too small or too large to effectively represent the image objects in existing OCNN method, (e) and (f) image patches can effectively represent the image objects.

Figure 4. The structure of the improved DenseNet.

Figure 5. The structure of deformable convolution.

Figure 6. The structure of CBAM.

Table 1. The handcrafted features.

Category	Name	Number	Name	Number
Spectral features	Mean	4	Ratio	4
	Standard deviations	4	Skewness	4
	Normalized Difference Vegetation Index (NDVI)	1	Saturation	1
	Normalized Difference Water Index (NDWI)	1	Intensity	1
	Brightness	1	Hue	1
Textural features	GLCM(Homogeneity)	5	GLCM (Ang. 2nd moment)	5
	GLCM(Contrast)	5	GLCM (Correlation)	5
	GLCM(Dissimilarity)	5	GLCM (Mean)	5
	GLCM(Entropy)	5	GLCM (Standard deviations)	5
Geometric features	Area	1	Length/width	1
	Boundary length	1	Compactness	1
	Length	1	Shape index	1
	Width	1	Rectangular Fit	1

Figure 7. The study area.

Table 2. Number of samples.

Category	Training Set	Validation Set	Testing Set	Sum
OM	331	149	116	596
WD	218	100	108	426
RA	240	122	107	469
BD	200	99	120	419
NV	344	160	175	679
BS	201	101	102	404
Sum	1534	731	728	2993

Table 3. The hyperparameters of MFOCNN.

Method	Parameters
SLIC	n_segments: 120,000; compactness: 1; sigma: 10，
Improved DenseNet	Patch size: 64 × 64; Epoch: 300; Loss function: Cross-Entropy loss; Optimizer: SGD; Learning rate: 0.0001; Batch size: 16
RF	n_estimators:40; Maximum depth: 100

Figure 8. Classification visualization results of different comparison methods.

Table 4. Quantitative evaluation results of different comparison methods.

	$\mathit{F}1-\mathit{score}\left(\mathrm{\%}\right)$						$\mathit{OA}\left(\mathrm{\%}\right)$	$\mathit{Kappa}$
	OM	WD	RA	BD	NV	BS
resultDN	94.51	96.3	86.67	96.58	89.89	92.61	92.58	0.91
result1	95.2	94.34	88.89	97.52	93.22	94.58	93.95	0.94
result2	94.12	95.81	91.16	98.33	92.84	93.6	94.23	0.94
result3	95.69	97.67	92.86	98.33	93.53	96.59	95.60	0.95

Note: The best results are highlighted in bold.

Figure 9. Classification visualization results based on different segmentation algorithms.

Table 5. Quantitative evaluation results based on different segmentation algorithms.

	$\mathit{F}1-\mathit{score}\left(\mathrm{\%}\right)$						$\mathit{OA}\left(\mathrm{\%}\right)$	$\mathit{Kappa}$
	OM	WD	RA	BD	NV	BS
resultMSS	92.45	93.15	86.96	96.45	87.89	92.68	91.22	0.89
resultDN	94.51	96.30	86.67	96.58	89.89	92.61	92.58	0.91

Note: The best results are highlighted in bold.

Figure 10. Classification visualization results based on different classification networks.

Table 6. Quantitative evaluation results based on different classification networks.

	$\mathit{F}1-\mathit{score}\left(\mathrm{\%}\right)$						$\mathit{OA}\left(\mathrm{\%}\right)$	$\mathit{Kappa}$
	OM	WD	RA	BD	NV	BS
resultRSN	82.73	92.75	77.59	88.00	82.18	83.08	84.07	0.81
resultVGG	79.23	90.73	81.39	90.83	79.66	82.35	83.52	0.80
resultGGN	86.42	91.18	91.15	97.87	91.17	92.39	91.62	0.89
result1	95.20	94.34	88.89	97.52	93.22	94.58	93.95	0.94

Note: The best results are highlighted in bold.

Figure 11. Classification visualization results based on different semantic segmentation networks.

Table 7. Quantitative evaluation results based on different semantic segmentation networks.

	$\mathit{F}1-\mathit{score}\left(\mathrm{\%}\right)$						$\mathit{OA}\left(\mathrm{\%}\right)$	$\mathit{Kappa}$
	OM	WD	RA	BD	NV	BS
resultFCN	87.78	90.00	89.92	86.25	83.07	91.88	87.06	0.84
resultUNet	98.50	96.18	99.55	79.56	88.51	90.05	82.81	0.78
resultDLb	80.58	85.34	71.54	65.16	69.33	67.83	73.22	0.66
result3	95.69	97.67	92.86	98.33	93.53	96.59	95.60	0.95

Note: The best results are highlighted in bold.

Figure 12. The $\mathrm{OA}$ and $\mathrm{Kappa}$ of different n_segments.

Figure 13. Classification results in shadow areas in result_DN and result₃, the yellow circled area in region 10 is shadow.

Data availability statement

The remote sensing image used in this study can be acquired from the following link: https://figshare.com/s/0cd166e71ac14e73a0bc. The codes using python language can be founded from the following link: https://figshare.com/s/b1d5079b9859321dcf88.