D-FusionNet: road extraction from remote sensing images using dilated convolutional block

ABSTRACT

Deep learning techniques have been applied to extract road areas from remote sensing images, leveraging their efficient and intelligent advantages. However, the contradiction between the effective receptive field and coverage range, as well as the conflict between the depth of the network and the density of geographic information, hinders further improvement in extraction accuracy. To address these challenges, we propose a novel semantic segmentation network called D-FusionNet. D-FusionNet integrates the Dilated Convolutional Block (DCB) module, which serves as a technique for expanding the receptive field and mitigating feature loss, resembling a residual mechanism during encoding. We evaluate the extraction capability of D-FusionNet using GF-2 (Gaofen-2) satellite datasets and Massachusetts aerial photography datasets. The experimental results demonstrate that D-FusionNet performs well in road extraction tasks. Compared to FCN, UNet, LinkNet, D-LinkNet, and FusionNet, D-FusionNet achieves an average improvement of 5.35% in F1-score, 7.12% in IoU (Intersection over Union), and 5.61% in MCC (Matthews Correlation Coefficient) on the GF-2 dataset. For the Massachusetts dataset, there is an average improvement of 2.48% in F1-score, 3.25% in IoU, and 2.25% in MCC. This study provides valuable support for road extraction from remote sensing images.

KEYWORDS:

• Road extraction
• dilated convolution block
• D-FusionNet
• remote sensing images

1. Introduction

The quick growth of road infrastructure has brought new challenges for extracting and updating the road network information (Dai et al. 2020). The traditional human-computer interaction vectorized road information extraction method has the limitations of long time consumption and a high cost (Jia et al. 2021) is increasingly failing to meet production needs. Therefore, this calls for the immediate development of intelligent and efficient road network extraction algorithms (Liu et al. 2020). Convolutional Neural Networks (CNNs) have become a widely used method in the field of road extraction from remote sensing images with the development of segmentation network models such as FCN (Long, Shelhamer, and Darrell 2015), U-Net (Ronneberger, Fischer, and Brox 2015), LinkNet (Chaurasia and Culurciello 2017), and DeepLab (Chen et al. 2014, 2017, 2018). These models have improved the accuracy and efficiency of road extraction, capturing wide interest.

Recently, the focus of road extraction from remote sensing images has shifted from applying various CNN models to optimizing network structures based on specific requirements of road scenes (Abdollahi et al. 2020): (1) The Generative Adversarial Networks (GAN) can effectively alleviate the overfitting phenomenon caused by the lack of road data (Goodfellow et al. 2014; Mansourifar et al. 2022). For example, based on the traditional GAN, Chen et al. (2022) proposed an improved conditional GAN (NIGAN, neighborhood probability enhancement and improved conditional generative adversarial network) framework for extracting information from sparsely sampled mountainous roads. Alternatively, to reduce the network’s dependence on high-quality datasets during training, Chen et al. (2022) put forward a semi-weak GAN (SW-GAN) approach. (2) The proper utilization of attention mechanisms in networks can provide an additional boost to the network’s focus on road areas (Vaswani et al. 2017). For instance, Li et al. (2022) introduced a cascaded attention-enhanced architecture that utilized spatial attention residual blocks to address the problem of jagged road boundaries, which preserved and refined the boundary information while capturing road morphology more flexibly. In a similar vein, Dai et al. (2023) designed a road augmentation module (RAM) based on deformable convolution and proposed the Road-Augmented Deformable Attention Network (RADANet) to accomplish road extraction tasks. (3) Graph convolution takes into account both the neighborhood and self-information of nodes, expanding the receptive field without sacrificing location information. This approach effectively addresses the challenges posed by the loss of location information and the need for global context information. For instance, Cui et al. (2022) integrated the outputs of graph convolutional networks (GCNs) and deep convolutional neural networks (DCNNs) through parallel computation, effectively tackling the incomplete and discontinuous problems encountered in road extraction. This idea is also reflected in the work of Zhou et al. (2021), where they cleverly designed a novel road extraction framework called a Split Depth-Wise (DW) Separable Graph Convolutional Network (SGCN). The SGCN achieved outstanding results on the Massachusetts roads dataset as well as their own dataset. (4) Additionally, optimizing road extraction results can be achieved by changing the hierarchical connectivity of the network, incorporating residual structures, or utilizing operations such as dilated convolutions. For example, Cheng et al. (2017) proposed a method for automatic road detection and centerline extraction based on a cascade end-to-end convolutional neural network. Zhou et al. (2018) employed pre-training and dilated convolutions to enhance the feature extraction capability of road networks. Zhang et al. (2018) developed a deep residual U-Net model that reduced feature loss through the use of dense connections and residual structures. He et al. (2019) improved road extraction performance by introducing the Atrous Spatial Pyramid Pooling (ASPP) module. Zhang and Wang (2019) adopted the JointNet model for segmenting roads and buildings, which employed dense connections to enhance interconnectivity between different layer feature maps and utilized dilated convolutions to improve operational efficiency. Abdollahi et al. (2022) addressed the challenge of road disconnection by combining boundary learning with a recurrent residual convolutional neural network, taking into account road connectivity. Yang et al. (2022) designed a denser network (SDUNet) that introduced DULR modules to capture richer road features.

However, there are two challenges that require further exploration in road extraction tasks. The first challenge is a contradiction between the effective receptive field and coverage range (Jiang, Zhong, and Zhang 2023). CNNs are mainly designed for multi-segmentation tasks, such as U-Net for biomedical image segmentation, FCN, SegNet, and DeepLab for segmentation tasks on public datasets (Pascal VOC). When utilizing CNNs to extract roads from remote sensing images, the limited receptive fields for feature encoding may limit the learning of road shape features, consequently impacting the accuracy of extraction (Liu et al. 2019). Another challenge arises from the conflict between the depth of the network and the density of geographic information (Lu et al. 2022). Encoder may potentially blur the distinction between the foreground and background in remote sensing images. Therefore, it is necessary to employ feature-preserving methods during the encoding process and select a more compact network to minimize the loss of features.

To address these two challenges, a new semantic segmentation network (D-FusionNet) is proposed to extract the road networks. The main contributions of the study are summarized as follows:

1. The Dilated Convolutional Block (DCB) module was utilized as a feature buffering mechanism. Five DCBs were constructed with dilation rates of {1, 2, 4, 8} to fuse features from different layers and achieve a dense feature representation. By enhancing the network’s ability to detect road regions and reducing feature loss during forward propagation, the DCB module improves the performance of segmentation networks.

2. A novel semantic segmentation network called D-FusionNet was proposed for road extraction. D-FusionNet combines the DCB and FusionNet to expand the receptive field and minimize feature loss, showing the better performance than the other CNN models for road extraction from remote sensing images.

3. To evaluate the extraction performance of our network, we utilized two datasets: GF-2 dataset and Massachusetts dataset. Our experiments encompassed ablation and comparative studies. In the discussion, we examined four potential factors, both qualitatively and quantitatively, that could impact road extraction. Additionally, we conducted an evaluation of D-FusionNet’s spatial transferability.

2. Methods

2.1 Dataset

To evaluate the performance of the proposed D-FusionNet model, both in-house GF-2 datasets and publicly available Massachusetts datasets were collected in this study. The GF-2 satellite, which is part of the China High-resolution Earth Observation System, carries various of remote sensing payloads such as panchromatic and multispectral. We selected three GF-2 images from Luxian County in the south of the Sichuan Basin, China, to extract the road information. The dataset was manually annotated and consisted of 325 train samples and 9 test samples (Figure 1). Detailed parameters of these images are provided in Table 1. The area includes low mountains, large hills, medium hills with narrow valleys, low hills with wide valleys, and valley terraces. The terrain is higher in the northeast and lower in the southwest, with a height difference of 539.7 meters. Luxian County has well-developed transportation facilities. The Massachusetts dataset is an aerial drone dataset widely used in road extraction research from remote sensing images. It includes 1108 train samples, 14 validation samples, and 49 test samples. This dataset is known for its high-quality images and labels (Mnih 2013).

Figure 1. The GF-2 dataset (a to j represent data from the GF-2 dataset. We used white color to depict the annotated road areas by overlaying road labels onto image tiles).

Table 1. The parameters of GF-2 images used in this study.

Time	Centre	Resolution (m)	Orbit Number	Sensor	Cloud Cover (%)
2021.09.20	N29.20, E105.39	3.24	38317	PMS2	0
2020.04.15	N29.18, E105.42	3.24	30575	PMS2	0
2018.03.09	N29.17, E105.35	3.24	19209	PMS2	0

2.2 Dilated convolutional block

Roads in remote sensing images typically appear as thin linear structures with gray or earthy-yellow colors. They exhibit consistent texture features and radiation characteristics due to the use of hard surface materials like asphalt and cement, as well as their relative flatness compared to the surrounding environment. To effectively capture the road features, exploring different convolution approaches is a valuable perspective. With the advancement of CNNs, convolutional kernels have become increasingly diverse. For example, in the GCN (Global Convolutional Network), the conventional n × n convolution kernel is split into two strip kernels with sizes of 1 × n and n × 1 (Peng et al. 2017). MobileNet (v1, v2) (Howard et al. 2017; Sandler et al. 2018) adopts depthwise separable convolution, which preserves features while significantly reducing the number of parameters to 1/8 of those used in ordinary convolutions. Among various kernels, dilated convolution is widely applied in segmentation tasks that require capturing large-scale targets. For instance, it is utilized in networks such as ConDINet++ (Yang et al. 2022) and D-LinkNet (Yu, Koltun, and Funkhouser 2017).

Dilated convolution enhances the receptive field by adjusting the spacing between effective units of the convolutional kernel in comparison to regular convolution. The calculation of equivalent receptive field of n-th dilated convolution is expressed as (Krizhevsky, Sutskever, and Hinton 2012):

(1) $r_n=r_{n-1}+(K-1)\prod _{n=1}^{n-1}{S}_i\leftarrow K=(k-1)(d-1)+k$(1)

where $r_n$ and $r_{n-1}$ are the receptive field of the current and previous layers, respectively; $K\hspace{0.17em}$is the size of the equivalent convolutional kernel of the current layer, which can be calculated by the convolutional kernel size $k$ and dilation rate $d$; $S_i$ is the stride of layer $i$. When considering dilated convolution as a regular convolution with multiple “hole,” it still adheres to the principles of convolution calculation (Simonyan and Zisserman 2014):

(2) $(I\ast K)(i,j)=\sum _m\sum _{n}I(i+m,j+n)K(m,n)$(2)

where $I$ is an input feature map at position $(i,j)$; and $K(m,n)$ is the weight of the convolutional kernel at a corresponding position on a certain dimension.

However, the features obtained solely through a single dilated convolution are discontinuous, because the “holes” in dilated convolution cannot be learned. Therefore, in the study, we employ DCB consisting of four dilated convolutions with different dilation rates. The module is structured in a series-parallel combination (Figure 2). When the feature map enters into DCB’s working range, dilated convolutions with dilation rates of {1, 2, 4, 8} and kernel sizes of {3, 5, 9, 17} will generate receptive fields of {3, 7, 15, 31}, enabling the detection and capture of more extensive features. It is worth noting that the DCB has already been introduced in D-LinkNet and is not a novel module.

Figure 2. Dilated convolutional block (the green square represents the learning region of dilated convolution, the white square represents the “hole” of dilated convolution, which refers to the gaps between the convolutional units. The blue shadows at the bottom of the figure depict the receptive field).

To ensure that the feature maps extracted by each dilated convolution in DCB can be integrated, padding is introduced when performing the dilated convolution (Yu and Koltun 2015):

(3) $O=⌊\frac{I+2p-k-(k-1)(d-1)}{s}⌋+1$(3)

where $I$ denotes the input size; $O$ denotes the output size; $k$ denotes the kernel size; $d$ denotes the dilation rate; $s$ denotes the stride; and $p$ denotes the padding. The output size of the feature maps can be guaranteed to be the same with input size when the size of the padding is consistent with the dilation rate.

2.3 D-FusionNet model

D-FusionNet, a CNN based on FusionNet model, is utilized for road extraction. FusionNet is a type of UNet-based CNNs with denser connections, which prevents feature degradation during the feature extraction process (Quan, Hildebrand, and Jeong 2021). In D-FusionNet, feature maps of different levels are directly into the feature map dimension, without cutting and merging channel dimensions, and the DCB is embedded into the D-FusionNet to integrate feature information, with dilation rates of {1,2,4,8} select to achieve dense semantic information expression.

Figure 3 illustrates the structure of D-FusionNet. The network comprises an encoder and decoder separated by a bridge layer, which facilitates downsampling and upsampling processes. In the encoder, the image undergoes compression into a smaller size through four downsampling layers and four internally embedded DCB modules (He et al. 2016). Each downsampling layer consists of a series of convolutional operations and a residual structure. These layers not only extract features but also compress the input image size. During the processing of feature maps, D-FusionNet integrates the features collected by the current downsampling layer into the DCB for secondary processing and integration. The resulting densely packed features are then transmitted to the next downsampling layer. Bridge layer functions similarly to the downsampling but solely expands the channel dimension without altering the size. In the decoder, image restoration is achieved through a four-time upsampling operation, which includes deconvolution, feature map merging, residual structure, and convolutions (Zeiler et al. 2010; Zeiler, Taylor, and Fergus 2011). During upsampling of feature maps, deconvolution progressively increases their size, restoring them to a similar size with the input image.

Figure 3. Structure of D-FusionNet. (The encoder-decoder in this figure is composed of different modules, represented by the rectangular blocks of various colors. The dashed lines indicate the specific operations performed by each block, while the solid lines represent the connections within the network. The rectangles of different colors correspond to the various standard operations in CNNs, and the group of gray rectangles represents the current feature map).

In theory, the DCB in D-FusionNet can optimize the network in three ways. Firstly, it gradually expands the receptive field, mitigating the grid effect produced by conventional dilated convolutions. Secondly, the “hole” in DCB does not possess learning ability, resulting in higher parameter efficiency compared to methods that utilize large convolutions to expand the receptive field. Lastly, the DCB acts as a feature buffer zone and enhances the connection between feature layers through a special series-parallel mechanism, which may reduce the feature loss in data encoding.

2.4 Evaluation criteria

Three performance indexes, namely F1-Score, MCC, and IoU, were utilized to evaluate the effectiveness of the proposed method. The F1-score represents the harmonic mean of precision and recall, while IoU measures the intersection-over-union ratio between the ground truth and extracted results. MCC is a correlation coefficient that measures the differences between predicted results and labels. These indexes can be expressed as (Martin, Fowlkes, and Malik 2004; Taran et al. 2018):

(4) $precision=\frac{TP}{TP+FP}$(4)

(5) $recall=\frac{TP}{TP+FN}$(5)

(6) $F1=2\frac{precision\times recall}{precision+recall}$(6)

(7) $IoU=\frac{TP}{TP+FP+FN}$(7)

(8) $MCC=\frac{TP\times TN-FP\times FN}{\sqrt{(TP+FP)\times (TP+FN)\times (TN+FP)\text{break}\times (TN+FN)}}$(8)

where TP represents the number of pixels extracted as road and labeled as road; TN represents the number of pixels extracted as background and labeled as background; FP represents the number of pixels extracted as road but labeled as background; and FN represents the number of pixels extracted as background but labeled as road.

2.5 Experiment procedures

The study utilized PyTorch to establish the experimental environment, encompassing network structure, data loading, training, and testing. The experiments were carried out on a GPU with CUDA support to improve computational efficiency. The server utilized was an NVIDIA GeForce RTX 3090 with 24GB of GPU memory. The approximate process of performing road extraction task using D-FusionNet is illustrated in Figure 4.

Figure 4. The flowchart of the experimental process (The sources of a-1 and b-1 are the Massachusetts dataset and the GF-2 dataset, respectively. The extraction results can be seen in a-2 and b-2. c demonstrates the Spatial transferability of D-FusionNet).

The Adam optimization method (Kingma and Ba 2014) and the loss function of cross-entropy were employed to improve the efficiency and accuracy of the processing in this study. The Adam optimization method is a commonly used adaptive algorithm in deep learning that combines momentum and adaptive learning rates. Its core idea is to use first and second moment estimates of gradients to adjust learning rates and momentum parameters for more efficient updates of model parameters. Cross-entropy can measure the difference between the labels of dataset and the predicted value of the network. The loss under a batch can be expressed as (Ho and Samuel 2020):

(9) $Loss=\frac{-1}{M}\sum _{i=1}^M\{y_{i}log{a}_i+(1-y_i)log(1-a_i)\}$(9)

where $M$is the batch size; $y_i$ is the true; and $a_i$ is the prediction.

The ReLU (Rectified Linear Unit) and ELU (Exponential Linear Unit) were selected as the activation functions to avoid overfitting that may occur in the training process. The ReLU activation function enhances the nonlinear relationship between the network layers and realizes the unilateral inhibition of data transmission, and finally achieves the effect of sparse activation, which is calculated as:

(10) $f(x)=max(0,x)$(10)

where x represents the input and corresponds to the value of each unit in the feature map generated by the previous layer. ELU is an activation function proposed on the basis of ReLU and sigmoid. The left side of the function has soft saturation so that the activation mean tends to be balanced. Therefore, the influence of bias can be reduced, and the gradient will be closer to the natural gradient:

(11) $f(x)=\left\{\begin{array}{cc}x& x>0\\ \alpha (e^x-1)& x\le 0\end{array}\right.$(11)

where the parameter α is a hyperparameter of ELU, which determines the saturation point for the negative values.

3. Results

3.1 Results of ablation experiments

Ablation experiments are commonly used in CNNs to analyze the influences of different module components. In this study, two sets of ablation experiments were conducted to assess the effectiveness of proposed D-FusionNet. This was done by modifying the dilation rates and embed positions of the DCB module in the network.

3.1.1 Dilation rate

The varied dilation rate of DCBs in the encoder was tested to analyze its effect on road extraction when using the proposed D-FusionNet. The controlling models were formed by varying the dilation rate based on D-FusionNet. Two sets of comparison models were created by using dilation rates of {2,4,8,16} and {4,8,16,32}, respectively. Considering the hierarchical priority of DCB data flow, we inverted the dilation rates in the above two models as well as those in D-FusionNet, and resulted in three new comparison models. Table 2 presents the accuracy of comparative models after modifying the dilation rates, and specifies the degree of decline compared to D-FusionNet. Additionally, we utilized Grad-CAM (Gradient-weighted Class Activation Mapping) to illustrate the attention of D-FusionNet and other ablation models on road regions in different datasets (Figure 5) (Selvaraju et al. 2017; Zhou et al. 2016).

Figure 5. Grad-CAM analysis of ablation models in dilation rate experiment (ablation I, II, III, IV and V represents the ablation model by replacing the dilation rate in DCB with {2, 4, 8, 16}, {4,8,16,32}{32,16,8,4}{16,8,4,2}{8,4,2,1}, respectively).

Table 2. Ablation experiments by modifying the dilation rates.

DCB		Dataset		F1-Score (%)	IoU (%)	MCC (%)
Dilation Rate	Position	GF-2	MASSACHUSETTS
{1,2,4,8}	{1,2,3,4,5}	√		81.70	69.05	80.77
			√	78.11	64.08	77.88
{2,4,8,16}	{1,2,3,4,5}	√		81.31 ↓0.39	68.51 ↓0.54	80.38 ↓0.39
			√	77.82 ↓0.29	63.70 ↓0.38	77.76 ↓0.12
{4,8,16,32}	{1,2,3,4,5}	√		80.99 ↓0.71	68.05 ↓1.00	80.02 ↓0.75
			√	77.14 ↓0.97	62.78 ↓1.3	77.19 ↓0.69
{32,16,8,4}	{1,2,3,4,5}	√		80.79 ↓0.91	67.78 ↓1.27	79.82 ↓0.95
			√	77.55 ↓0.56	63.33 ↓0.75	77.44 ↓0.44
{16,8,4,2}	{1,2,3,4,5}	√		80.34 ↓1.36	67.14 ↓1.91	79.34 ↓1.43
			√	77.73 ↓0.38	63.58 ↓0.5	77.41 ↓0.47
{8,4,2,1}	{1,2,3,4,5}	√		81.44 ↓0.26	68.69 ↓0.36	80.50 ↓0.27
			√	77.43 ↓0.68	63.17 ↓0.91	77.33 ↓0.55

D-FusionNet's accuracy and the maximum differences observed on different datasets are highlighted in bold

The results reveal that the DCBs with {1,2,4,8} dilation rates demonstrate the best performance, while modifying the dilation rate of DCBs leads to a decrease in F1-score, IoU, and MCC. In the GF-2 dataset, F1-score, IoU, and MCC are evenly decreased by 0.73%, 1.02%, and 0.76%, respectively. The largest decrease occurs when the dilation rate of DCBs is increased by two times and reversed in D-FusionNet, resulting in a decrease of 1.36%, 1.91%, and 1.43% for F1-score, IoU, and MCC, respectively (the fifth row in Table 2). The smallest decrease occurs when the dilation rate of DCBs in D-FusionNet is reversed without changing the dilation rate, resulting in a decrease of 0.26%, 0.36%, and 0.27% for F1-score, IoU, and MCC, respectively (the last row in Table 2). Similar phenomena are also observed in Massachusetts dataset.

3.1.2 Embedding positions

The influence of DCB embedding position on the network’s extraction capability was analyzed in this study. In the experiment, the dilation rate of DCBs was set as {1,2,4,8}. Initially, the DCB in the first downsampling layer of D-FusionNet was removed while preserving the DCBs in other layers. Subsequently, the module from the second, third, fourth, and bridge layers underwent the same removal process to create four corresponding controlling models. The accuracy of comparative models by modifying the embedding positions and the degree of decline compared to D-FusionNet are illustrated in Figure 6 and Table 3.

Figure 6. The Grad-CAM analysis of ablation models in embedding positions experiment. Ablation I, II, III, IV, and V represent the ablation model by removing the downsampling performed after the first, second, third, fourth, and fifth iteration.

Table 3. Ablation experiments by modifying the embedding positions.

DCB		Dataset		F1-Score (%)	IoU (%)	MCC (%)
Dilation Rate	Position	GF-2	MASSACHUSETTS
{1,2,4,8}	{1,2,3,4,5}	√		81.70	69.05	80.77
			√	78.11	64.08	77.88
{1,2,4,8}	{2,3,4,5}	√		81.11 ↓0.59	68.21 ↓0.84	80.19 ↓0.58
			√	77.56 ↓0.55	63.34 ↓0.74	77.79 ↓0.09
{1,2,4,8}	{1,3,4,5}	√		81.28 ↓0.42	68.46 ↓0.59	80.34 ↓0.43
			√	77.21 ↓0.9	62.88 ↓1.20	77.15 ↓0.73
{1,2,4,8}	{1,2,4,5}	√		81.09 ↓0.61	68.19 ↓0.86	80.13 ↓0.64
			√	77.86 ↓0.25	63.74 ↓0.34	77.72 ↓0.16
{1,2,4,8}	{1,2,3,5}	√		81.37 ↓0.33	68.60 ↓0.45	80.43 ↓0.34
			√	77.24 ↓0.87	62.92 ↓1.16	77.49 ↓0.39
{1,2,4,8}	{1,2,3,4}	√		81.54 ↓0.16	68.84 ↓0.21	80.60 ↓0.17
			√	77.55 ↓0.56	63.33 ↓0.75	77.40 ↓0.48

D-FusionNet's accuracy and the maximum differences observed on different datasets are highlighted in bold

The results indicate that the DCB embedding position at {1,2,3,4,5} achieves the best performance. Taking the Massachusetts dataset as an example, F1-score, IoU, and MCC are evenly decreased by 0.63%, 0.84%, and 0.37%, respectively. The largest drop occurs when removing the DCB in the second layer, resulting in a decrease of 0.9%, 1.20%, and 0.73% for F1-score, IoU, and MCC, respectively (the third row in Table 3). The smallest decrease occurs when the DCB in the third layer, resulting in a decrease of 0.25%, 0.34%, and 0.16% for F1-score, IoU, and MCC, respectively (the fourth row in Table 3). Figure 6 provides a visual representation of the attention level of the road area for both D-FusionNet and the control model.

3.2 Comparing D-FusionNet with other CNNs

This study compares the performance of D-FusionNet with several other CNNs, including FCN, UNet, LinkNet, D-LinkNet, and FusionNet. UNet and FusionNet are the complex models based on FCN, while LinkNet and D-LinkNet are the lightweight models. Table 4 shows the comparisons of D-FusionNet with the other five networks. The results indicate that D-FusionNet outperforms other networks in all three evaluation indexes.

Table 4. Comparisons among FCN, UNet, LinkNet, D-LinkNet, FusionNet, and proposed D-FusionNet models.

MODEL	Dataset		F1-Score (%)	IoU (%)	MCC (%)
	GF-2	MASSACHUSETTS
FCN	√		70.15 ↑11.55	54.02 ↑15.03	68.69 ↑12.08
		√	74.87 ↑3.24	59.83 ↑4.25	74.94 ↑2.94
UNet	√		80.65 ↑1.05	67.57 ↑1.48	79.66 ↑1.11
		√	76.52 ↑1.59	61.97 ↑2.11	76.59 ↑1.29
LinkNet	√		73.46 ↑8.24	58.05 ↑11.00	72.13 ↑8.64
		√	73.34 ↑4.77	57.91 ↑6.17	73.18 ↑4.70
D-LinkNet	√		76.45 ↑5.25	61.88 ↑7.17	75.25 ↑5.52
		√	75.80 ↑2.31	61.03 ↑3.05	75.74 ↑2.14
FusionNet	√		81.04 ↑0.66	68.12 ↑0.93	80.09 ↑0.68
		√	77.60 ↑0.51	63.40 ↑0.68	77.72 ↑0.16
Ours	√		81.70	69.05	80.77
		√	78.11	64.08	77.88

D-FusionNet's accuracy and the maximum differences observed on different datasets are highlighted in bold

It can be seen in Table 4 that D-FusionNet has the best performance in road extraction. For the GF-2 dataset, the comparisons with other five modes show that the F1-score, IoU, and MCC have an average improvement of 5.35%, 7.12%, and 5.61%, respectively. For the Massachusetts dataset, the average improvement is 2.48%, 3.25%, and 2.25%, respectively.

Taking GF-2 dataset as an example, the largest and smallest improvements were analyzed. The largest improvement is between D-FusionNet and FCN, where improvements are 11.55%, 15.03%, and 12.08%, respectively. Conversely, the smallest improvement is between D-FusionNet and FusionNet, where improvements are 0.66%, 0.93%, and 0.68%, respectively. The FCN has the sparse network connection, while FusionNet has the dense network connection. It is speculated that dense network connections may enhance the learning ability of the network toward road extraction. Additionally, the main difference between D-FusionNet and FusionNet lies in the utilization of the DCB module. Therefore, the incorporation of DCB improves the extraction results.

4. Discussion

After conducting comparison of D-FusionNet and other CNNs, we summarized and analyzed four distinct experimental phenomena. In the following sections, we will present a detailed analysis and explanation of each phenomenon to improve the overall understanding of the road extraction tasks and D-FusionNet.

4.1 Impact of dense connection mode on road extraction

The comparisons between D-FusionNet and two different connection density networks (FCN and UNet) were discussed in this study, as shown in Figure 7. Both highlighted roads (red rectangle) and evaluation parameters (TP, FN, TN, and FP) indicate that the denser networks have the better performance of road recognition. This phenomenon can be attributed to the fact that a dense connection pattern can effectively minimize feature loss and enhance extraction accuracy. FCN adopts a sparse connection mode and gradually merges features from the last two downsampling layers during the upsampling procedure. Starting from UNet, CNNs gradually adopt a dense connection pattern with multiple skip connections between the encoder and decoder. This dense connection approach, along with the symmetric encoder-decoder structure, has evolved into modern CNN frameworks, including FusionNet. Based on UNet, FusionNet introduces a residual structure, which enables a connection within the encoder and decoder to reduce the feature loss. The proposed D-FusionNet in this study follows the same approach. During the encoding process, D-FusionNet expands the receptive field using DCBs. This integration further enhances the network’s density. As mentioned, in Section 2.2, the feature buffering mechanism of DCB effectively decreases the feature loss during the forward propagation, improving road integrity and feature extraction accuracy in practical applications.

Figure 7. The comparisons of road extraction among FCN, UNet and D-FusionNet models (for each model, the left shows the extracted road and the right shows the evaluation parameters: TP with green, FN with red, TN with black, and FP with blue. The red rectangles highlight the comparisons among FCN, UNet and D-FusionNet models).

4.2 Impact of parameter number on road extraction

By incorporating the DCB in D-FusionNet, the feature extraction accuracy is improved, while it also leads to an increase in network parameters. The number of additional parameters associated with DCBs rises significantly with their depth, as shown the following equation:

(12) $P=k^2{C}_i{C}_o$(12)

where $C_i$ represents the number of input channels; $C_O$ represents the number of output channels; and $k$ represents the kernel size. The DCB module is equivalent to the parameter number brought by four 3 × 3 ordinary convolutions. The parameter number of DCB can be further expressed as:

(13) $P_{DCB}=36C^2$(13)

where $C$ represents the channels of the feature map generated by the last downsampling. The first DCB receives a feature map with 64 channels, which will generate $36\ast {64}^2$ extra parameters. As the network depth increases, the number of additional parameters grows to four times or more. We compared the training time among different models. On the Massachusetts dataset, D-FusionNet completes one epoch in approximately 10.5 minutes, while D-LinkNet only takes 1.2 minutes, and LinkNet finishes in just 45 seconds. On the GF-2 dataset, D-FusionNet completes one epoch in approximately 1.22 minutes, while D-LinkNet only requires 18 seconds, and LinkNet finishes in just 11 seconds. Therefore, in order to mitigate feature loss and expand the receptive field, D-FusionNet sacrifices speed.

The impact of parameter number on road extraction was analyzed by comparing D-FusionNet with two lightweight models (LinkNet and D-LinkNet), as shown in Figure 8. Both LinkNet and D-LinkNet retain the encoder-decoder structure and utilize ResNet as the encoder to reduce parameters, achieving the lightweight segmentation. However, when compared to the parameter-rich D-FusionNet, the performance of LinkNet and D-LinkNet on multi-source platform datasets is not ideal. This improvement may be attributed to the parameter. Therefore, striking a balance between training cost and accuracy improvement is crucial for road extraction from remote sensing images.

Figure 8. The comparisons of road extraction among LinkNet, D-LinkNet and D-FusionNet models (for each model, the left shows the extracted road and the right shows the evaluation parameters: TP with green, FN with red, TN with black, and FP with blue. The red rectangles highlight the comparisons among LinkNet, D-LinkNet and D-FusionNet models).

4.3 Impact of DCBs on road extraction

D-FusionNet enhances the performance of FusionNet through the introduction of the DCB module. Comparisons of road extraction between FusionNet and D-FusionNet were carried out to discuss the impact of DCBs on road extraction (Figure 9). D-FusionNet exhibits the smaller number of FP compared to FusionNet, indicating that the main error in FusionNet is from the road misclassifications as background. Analysis of Figure 9 reveals that the improvements of D-FusionNet primarily focus on enhancing road integrity and capturing a wider range of roads. Unlike FusionNet, D-FusionNet demonstrates the ability to connect road intersection points. This can be attributed to the feature buffering mechanism of the DCB, which effectively preserves detailed information in the image. FusionNet faces challenges in extracting some large continuous roads, whereas D-FusionNet excels in compensating for discontinuous roads. It is due to the larger receptive field of DCB, enabling D-FusionNet to explore road areas from a broader perspective.

Figure 9. The comparisons of road extraction between FusionNet and D-FusionNet models (For the each model, the left shows the extracted road and the right shows the evaluation parameters: TP with green, FN with red, TN with black, and FP with blue. The yellow rectangles highlight the comparisons among LinkNet, D-LinkNet and D-FusionNet).

4.4 Impact of data quality on road extraction

An analysis of the extraction results of D-FusionNet reveals that it produces more FP and FN in areas with high vegetation or dense urban buildings, leading to a decrease in overall accuracy (Figure 10). This phenomenon can be attributed to object occlusion or the similarity in radiation characteristics between roads and surrounding objects. In such cases, D-FusionNet may struggle to distinguish between foreground and background, resulting in the failure to recognize roads in the image. Additionally, there is a possibility of incorrectly identifying the space between buildings as roads. Some studies have taken note of these issues, and current related research is exploring “intervention” methods outside the network to further enhance the accuracy of road segmentation. For instance, some studies use data augmentation techniques to enrich the original feature information within the dataset. Others utilize auxiliary constraint tasks to address connectivity issues arising from the image quality (Li et al. 2021). Furthermore, adaptive histogram equalization techniques have been used in certain studies to mitigate the influence of external factors (Xu et al. 2021). Therefore, it is recommended that some constraints or image enhancement methods can be introduced to constrain the road area from the level of data in future research.

Figure 10. The roads that the D-FusionNet is difficult to extract.

4.5 Spatial transferability

To investigate the spatial transferability of the proposed approach, we performed a new road extraction task on another scene image using a model trained on the GF-2 dataset. In order to consider a wide range of potential issues that may arise in practical scenarios, our image selection criteria were as follows: Firstly, we prioritized images with high vegetation coverage. Secondly, we gave preference to images with a certain level of cloud coverage. Additionally, we took into account the practical significance of updating road network data and prioritized regions with low urbanization where there might be potential issues of slow road network data updates. Lastly, we prioritized images that included linear features other than roads, such as rivers. Ultimately, we selected a summer image from Enshi Tujia and Miao Autonomous Prefecture, Hubei Province, China (Product Serial Number: 4952204).

The image has a resolution of 7300 × 6908, and the results are shown in Figure 11. Observing the left side of the image (Results), we can see that D-FusionNet successfully extracts the road areas in the image at this moment, and the extraction results exhibit good continuity. We further analyze the extraction results by comparing the extraction capabilities of D-FusionNet and FusionNet (Group I, a~h). It can be observed that D-FusionNet improves upon the extraction results of FusionNet: the road areas (blue) that FusionNet can capture but D-FusionNet cannot are almost non-existent, indicating that D-FusionNet inherits the extraction capabilities of FusionNet. Additionally, the red areas, which are the additional road areas obtained by D-FusionNet, effectively connect the previously fragmented locations in FusionNet, demonstrating that D-FusionNet optimizes the extraction of FusionNet to some extent.

Figure 11. Spatial transferability test of D-FusionNet (The figure consists of three parts: on the left is the extraction result of D-FusionNet in the summer imagery of Enshi Tujia and Miao Autonomous Prefecture, Hubei Province. The extracted road areas are highlighted in yellow. In group I, we randomly selected eight image tiles from the imagery and compared the extraction results of D-FusionNet and FusionNet. Yellow represents the common areas extracted by both networks, red indicates the additional road areas extracted by D-FusionNet compared to FusionNet, and the meaning of blue is opposite to that of red. In group II, we demonstrate several error sources of D-FusionNet. Among them, i shows the influence of rivers on network extraction, using the same annotation method as group I. j and k demonstrate the impact of cloud and fog cover on network extraction. l~o illustrate the impact of imaging obstruction from surrounding buildings, vegetation, and other factors on network extraction. The annotation method for j~o is the same as that on the left side of the figure).

However, we should also pay attention to the errors that occur during the extraction task (Group II). Firstly, rivers near the roads pose challenges to the extraction process. CNNs mistakenly identify some river areas with similar morphology to roads as road areas. This error is present in both FusionNet and D-FusionNet, but it is amplified in the results of D-FusionNet due to its larger detection range for capturing area continuity. Secondly, the presence of clouds above the roads also poses difficulties for extraction. The roads that should appear in certain areas are covered by clouds, making it challenging for the network to imagine what the covered roads should look like. Lastly, imaging obstruction from surrounding buildings, vegetation, and other factors also has a similar impact to the challenges posed by clouds.

Certainly, we have proposed some data processing optimization suggestions for these three types of errors. Firstly, for the river areas, we recommend using masks to remove them from the image. Secondly, regarding the challenges posed by cloud coverage, we believe that cloud removal is an essential preprocessing step for road extraction from remote sensing images. Lastly, for imaging obstruction from surrounding buildings, vegetation, and other factors, we suggest considering a multi-source data fusion approach to eliminate them. For example, we can incorporate terrain factors by using DSM (Digital Surface Model) obtained through Lidar scans in the study area to constrain the learning process of the network (Ma et al. 2022).

5. Conclusions

In this study, we propose a novel semantic segmentation network called D-FusionNet to address the challenges in road extraction tasks. In contrast to other studies, we simultaneously consider the benefits brought by enlarging the receptive field and utilizing feature buffering in the forward propagation for road extraction. D-FusionNet incorporates the use of DCB to reconstruct the encoding process and extract road information. With the inclusion of DCB, D-FusionNet obtains a larger receptive field, addressing road discontinuity and enhances the capability of capturing roads. Additionally, similar to the residual structures, DCB facilitates tighter transmission of contextual information within D-FusionNet, reducing feature loss caused by data downsampling. The conclusions are summarized as follows:

1. Considering the impact of feature loss in the network design is necessary. This study demonstrates that denser networks yield superior road extraction performance. This phenomenon can be attributed to the effective minimization of feature loss achieved through a dense connection pattern, which in turn improves extraction accuracy.

2. It becomes crucial to introduce mechanisms that effectively expand the receptive field in the network. The DCB module, in addition to buffering feature losses, also expands the receptive field of D-FusionNet, enabling it to adapt to different datasets and possess spatial transferability in road extraction tasks.

Although D-FusionNet demonstrates good performance in various experiments, it has two limitations. Firstly, the inclusion of DCB introduces additional parameters to D-FusionNet. This leads to increased training costs and potential limitations on its scalability. Secondly, there is a possibility of D-FusionNet misidentifying foreground and background, resulting in failures to accurately recognize roads in the image. Therefore, future studies will prioritize finding a balance between training cost and accuracy improvement, as well as exploring image enhancement methods for road extraction from remote sensing images. Certainly, during the transition of our research to practical implementation, we will further refine the task by taking into account factors such as the type of coating and class (main, secondary, or vicinal).

Acknowledgments

The authors would like to express their sincere gratitude to the reviewers and editors for their constructive and high-quality revision suggestions on this manuscript.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The public dataset used in this study can be obtained from https://www.cs.toronto.edu/~vmnih/data/. The satellite imagery data involved in the study can be acquired from the China Center for Resources Satellite Data and Application at https://www.cresda.com/zgzywxyyzx/index.html.

Additional information

Funding

This research was funded by the National Key R&D Program of China (2020YFC1512001), Natural Science Foundation of China (NSFC) (Nos.42074040), Natural Science Basic Research Plan in Shaanxi Province of China (2023-JC-JQ-24), Innovative Talents Promotion Plan of Shaanxi Province (Grant No. 2022KJXX-22), and Fundamental Research Funds for the Central Universities of CHD (Nos: 300102262902, 300102263401, 300102262512, 300102263502).