Dense Connected Edge Feature Enhancement Network for Building Edge Detection from High Resolution Remote Sensing Imagery

Abstract

Deep-learning-based methods for building-edge-detection have been widely researched and applied in the field of image processing. However, these methods often emphasis the analysis of deep features, which may result in neglecting crucial shallow information representation. Furthermore, abstract features in the deep layers can potentially interfere with the accuracy of edge extraction. To address these challenges, we propose a densely connected edge-detection enhancement network (DCEFE-Net) for building-edge-detection in high-resolution remote sensing images. Firstly, by introducing spatial land channel attention modules, we effectively captured low-level spatial information and high-level semantic information from the input image. Secondly, the proposed edge-aware feature enhancement (EAFE) module emphasis the representation of informative edge features. By alliteratively generating multiple layers of edge-detection maps, it addresses the issue of edge detail loss and enhances edge-detection accuracy. Finally, the dense connectivity blocks strengthen the connections between the convolutional layers, thereby preventing the loss of edge features. Experimental results on the WHU and the Inria Aerial Image Labeling datasets validate the effectiveness of DCEFE-Net, as it consistently produces clear and reliable building-edge results.

Résumé

Les méthodes basées sur l’apprentissage profond pour la détection des contours de bâtiments ont fait l’objet de nombreuses recherches et applications dans le domaine du traitement d’images. Cependant, ces méthodes mettent souvent l’accent sur l’analyze des caractéristiques profondes, ce qui peut conduire à négliger la représentation de l’information superficielle. De plus, les caractéristiques abstraites dans les couches profondes peuvent potentiellement interférer avec la précision de l’extraction des contours. Pour relever ces défis, nous proposons un réseau densément connecté de rehaussement de la détection de contours (DCEFE-Net) pour la détection des contours des bâtiments dans les images de télédétection à haute résolution. Tout d’abord, en introduisant des modules d’attention pour les canaux spatiaux, nous avons capturé efficacement des informations spatiales de bas niveau et des informations sémantiques de haut niveau à partir de l’image d’entrée. Deuxièmement, le module proposé de rehaussement des caractéristiques tient compte des arêtes (EAFE) et met l’accent sur la représentation des particularités informatives des contours. En générant par allitérations plusieurs couches de cartes de détection des contours, le module résout le problème de la perte de détails aux contours et améliore la précision de leur détection. Enfin, les blocs de connectivité denses renforcent les connexions entre les couches convolutives, empêchant ainsi la perte des particularités des contours. Les résultats expérimentaux sur les jeux de données WHU et Inria Aerial Image Labeling valident l’efficacité du DCEFE-Net, car il produit systématiquement des résultats clairs et fiables en bordure de bâtiments.

Introduction

Buildings are closely related to factors such as human socioeconomic status and the ecological environments. The detection of building-edge in high-resolution remote sensing imagery is an important component of geographical information databases, which plays a crucial role in urban expansion (Durieux et al. 2008), disaster warnings (Li et al. 2017), and urban planning and construction (He et al. 2021). Due to differences in roof materials and the presence of skylights, as well as adjacency of buildings to roads and potential obstruction by tall trees, rapid and accurate building-edge-detection is a significant focus area for experts and scholars studying urban development.

With the ongoing advancements in science and technology, remote sensing imagery has improved. High-resolution remote sensing imagery (HRRSI) can capture detailed textural information and the semantic context of buildings. Early edge-detection algorithms were characterized by long processing times, low efficiency, and poor accuracy (Ahmadi et al. 2010; Cui et al. 2012; Partovi et al. 2017; Peng et al. 2005; Su et al. 2018). New building extraction methods based on HRRSI have academic value and practical applications.

With the rapid development of artificial intelligence, deep-learning technology has been widely used across various subfields of image processing owing to its ability to efficiently and accurately analyze and decipher data. Remote sensing building data are characterized by massive, diverse, and complex attribute features, and deep-learning methods can automatically and extract complex abstract features from images at multiple levels. Therefore, many researchers have conducted studies on deep-learning-based edge extraction. Lu et al. (2018) retrained a richer convolutional features (RCF) network to create the RCF-building model and improved the edge probability map using topographic concepts. This approach effectively leverages high-level semantic information, although the post-processing stage encounters unavoidable issues with building discontinuities. Wen et al. (2021) proposed a multiscale erosion network (ME-Net) to address the thickness issue of edge extraction; however, its numerous convolutional and pooling operations can cause a loss of edge pixel values. Xia et al. (2021) innovatively combined semi-supervised learning with edge-detection networks to create a semi-supervised deep learning based on edge-detection (SDLED) network to minimize labeled sample requirements while achieving a more accurate extraction of rooftop boundaries in remote sensing images. Hamaguchi et al. (2018) proposed a local feature extraction (LFE) module to address the segmentation of small, densely distributed objects in HRRSI.

In CNNs deep information is some coarse-grained and more abstract features that tend to describe the image as a whole with stronger semantic information. However, this approach overlooked the deep information generated during network convolution. Ahmed and Byun (2019) utilized the automatic feature extraction capability of convolutional neural networks (CNNs) for edge-detection of buildings with different volume sizes. In addition, a dense extreme inception network (DexiNed) (Soria et al.2020) is a dense extreme initial network that improves holistic nested (HED) networks to obtain thin edge maps without pre-trained weights. Although this has been successful in datasets, its effectiveness in remote sensing image research, specifically for building-edge extraction, was questioned in the initial results (Bousias Alexakis and Armenakis 2021).

The aforementioned CNN-based methods have facilitated significant progress in building-edge-detection. Nevertheless, limitations arise due to the gradual loss of multiscale features in remote sensing images with decreasing spatial resolution. Consequently, these approaches can result in incomplete boundary extraction, diminished accuracy, and low efficiency. Furthermore, traditional CNNs frequently disregard local textures and other low-level features. Recognizing the importance of both deep and shallow information, it is imperative to introduce a network structure that unifies low and high-level features.

Fang et al. (2021) utilized a feature decoder to analyze the relationships and differences between low-level edges and high-level semantic features. They devised a multi-layer edge-feature-guided network (MLEFGN) that simultaneously de-noises and predicts precise object edges. However, the applicability of this method to natural images contrasts with the distinct feature presentation of remote sensing imagery. To address these variations in remote sensing characteristics, Sun et al. (2017) enhanced building detection performance by creating a two-stage CNN model that extracts low and high-level features of image objects, effectively mitigating complex background interference. Bai et al. (2022) proposed an end-to-end edge-guided recurrent convolutional neural network (EGRCNN) that can generate accurate building boundaries by combining prior knowledge. Zhao et al. (2023) introduced a multiscale receptive field network (MSRF-Net) that employs attention mechanisms to capture multiscale receptive field features and enhance building detection through a feature fusion module.

This study extensively considered the link between low- and high-level features. Low-level features provide accurate target localization, whereas high-level features contain rich semantic information. Notably directly merging low- and high-level features may result in the loss of edge feature details and the inability to eliminate background noise present in low-level features. In addition, the consecutive convolution processes in the network structure may lead to a loss of edge information and the problem of building-edge breakpoints. Moreover, consecutive convolutional processes in the network architecture can lead to edge information loss and create discontinuities in the building-edges.

To address these challenges, we present a novel approach called a densely connected edge-detection enhancement network (DCEFE-Net), which efficiently generates precise building-edges in HRRSI. The main contributions of this study are summarized as follows.

1. We propose a DCEFE-Net capable of effectively considering both low-level spatial features and high-level semantic features to address the issue of boundary details blurring during building-edge extraction.

2. We propose an edge-detection enhancement (EAFE) module, which addresses the scale variations among buildings by iteratively stacking edge features and up-sampling them based on the network depth.

3. To further preserve the extracted edge information during the convolutional process, we enhance the connectivity between the network modules by adding a dense convolutional layer. This enables stronger connections between the modules and improves the robustness of the network for edge extraction.

Related work

This section describes related work in three aspects: transformer – based salient object detection (SOD), CNN – based SOD and edge-detection methods.

Transformer -based SOD

The transformer model, proposed by Vaswani et al. (2023), was initially aimed at modeling the long-range dependencies between inputs and outputs in machine translation. Leveraging its strong contextual information representation, the transformer handles crucial multiscale features using cross-attention layers. Consequently, this model has been applied in across diverse visual tasks, including image segmentation (Strudel et al. 2021; Yan et al. 2022), object tracking (Shan et al. 2023), image recognition (Dosovitskiy et al. 2021; Touvron et al. 2021), and object detection (Carion et al. 2020; Chen et al. 2021).

In addition, transformers have proven to be effective in different applications of salient object detection (SOD) (Liu et al. 2021, 2022; Wang et al. 2022). For example, Wang et al. (2023) created a transformer-enhanced module to expand the receptive field and enhance salient object recognition. Similarly, Min et al. (2022) embedded a transformer in a network structure, facilitating the distant propagation of contextual dependencies, thereby improving the accuracy of in video-based saliency detection. Pei et al. (2023) introduced a salient instance segmentation framework with a transformer, emphasizing global features from the encoder and query features. Wang et al. (2023) designed a transformer comprising feature enhancement and fusion modules to effectively extract valuable information across multiple scales for saliency detection in red, green, blue and depth (RGB-D) images.

Although transformer-based salient object detection (SOD) models excel, they often lack precise spatial cues and structural information, causing challenges such as pixel confusion between salient objects and backgrounds. In addition, for remote sensing imagery, CNNs adept at handling spatial features are better suited for image data processing.

CNN-based SOD

Researchers have introduced human visual attention (HVA) mechanisms into image- processing tasks through visual image saliency detection (Itti et al. 1998), which filters and screens important information in images via computer simulations. Deep-learning has significantly facilitated the application of saliency target detection research in semantic segmentation (Han et al. 2006; Ko and Nam 2006), target detection (Gao et al. 2012), image indexing (Zheng et al. 2015), image classification (Benjilali et al. 2019; Zhang et al. 2019), and super-pixels (Lin et al. 2018), overcoming the time-consuming nature and poor results produced by manual feature extraction using traditional methods. The main methods are outlined as follows:

1. Significant target detection methods based on auxiliary networks.

This approach utilizes models from other domains as auxiliary networks to improve salient object-detection performance. The multiscale deep features (MDF) algorithm (Li and Yu 2015) was the first to apply deep neural networks to saliency detection. CNNs were used to extract features at multiple scales to enhance the spatial coherence of the saliency results. The algorithm also constructs a large saliency annotation dataset, MSRA-B, to facilitate research on saliency models. Li et al. (2018) proposed a contour-to-saliency network (C2SNet) algorithm consisting of two branches. One branch predicts the contour of the target object, and the other estimates the saliency score for each pixel. This method can automatically convert contour detection in deep models into salient object detection. Wu et al. (2019) proposed a mutual learning supervised learning network (MLSLNet) that combines foreground contour detection, edge-detection, and saliency object detection to effectively reduce image noise and achieve accurate foreground contour prediction.

• 2. Boundary and semantic enhancement based on saliency target detection methods.

High-level features contain rich semantic information, whereas low-to-middle -level features contain good spatial information. Semantic enhancement can locate salient targets better (Liu et al. 2018; Wu et al. 2019), whereas boundary enhancement can obtain clearer boundaries for salient targets (Feng et al. 2019; Qin et al. 2019). However, enhancing high-level features can cause blurring of low-to-middle-level features, and vice versa. Simultaneously enhancing both can play a more positive role in edge extraction. Wang et al. (2017) introduced a feed-forward network (FFN) improvement method that utilized pyramid pooling and multi-stage refinement to extract high-level semantic and detailed underlying features. Wang et al. (2019) proposed PAGE-Net to refine edge-detection information using multiscale saliency information. Zhao and Wu (2019) developed the pyramid feature attention (PFA) network, that combines a context-aware pyramid feature extraction (CPFE) module to gather contextual semantic information and a spatial attention (SA) module to precisely identify salient object boundaries, resulting in enhanced performance.

• 3. Prominence target detection methods based on a combination of global and local information.

By using recursive operations and attention mechanisms, global information such as color, texture, and foreground or background, is integrated with local information for boundary detection. This approach improved performance in detecting salient objects. Wang et al. (2018) introduced a global recursive localization network (RLN) capable of adaptively learning image contextual information and accurately refining object boundaries. Zhang et al. (2018) incorporated multi-path recursive feedback into a network and transferred top-level semantic information toshallow layers. This allowed the attention-guided network to effectively filter out background interference in the generated features. Additionally, several improved algorithms have been proposed (Liu and Han 2016; Luo et al. 2017).

Edge-detection methods

Image edge-detection is considered one of the most crucial components of both image processing and computer vision. It plays a vital role in a wide range of applications, including image improvement, restoration, and segmentation, as well as pattern recognition, feature description, and object tracking (Ma et al. 2018; Pirzada and Siddiqui 2013). By accurately identifying and extracting edge-feature information from remote sensing images, edge-detection methods can quickly locate important features. There are two main types of edge-detection algorithms: traditional and deep-learning. Traditional edge-detection methods rely on analyzing the underlying information within an image, such as color, luminance, texture, and gradient. These methods determine edges in images by analyzing the grey-scale jumps.

Traditional edge-detection methods are classified into two main categories, first and second-order edge-detection operators. There are several first-order edge-detection operators, including the Robert (Marr and Hildreth 1980), Prewitt (Chaudhuri and Chanda 1984), Sobel (Gao et al. 2010), and improved isotropic Sobel operators (Harris and Stephens 1988). While these operators can identify pixel locations with luminance variations, the edges they extract may be coarse and imprecise, and these operators can also produce pseudo-edges.

second-order edge-detection methods, such as the Laplacian operator (Lecun et al. 1998), Canny operator (Canny 1986), Marr-Hildreth operator, the Laplacian of Gaussian (LOG) operator, and Difference of Gaussian (DOG) operator (Ren 2008), are more effective in identifying edges with a strong response in textured regions. However, these methods are sensitive to noise, which is a major disadvantage.

With the significant progress made by CNNs in computer vision (Chen et al. 2018; He et al. 2016), edge-detection methods have improved by incorporating their ability to automatically extract high-quality semantic information (Shen et al. 2015; Xie and Tu 2015). Popular CNNs used in edge-detection algorithms include AlexNet (Krizhevsky, Sutskever, and Hinton 2017), visual geometry group network (VGGNet) (Simonyan and Zisserman 2015), and deep residual networks (ResNet) (He et al. 2016).

Earlier fully convolutional network (FCN) (Shelhamer et al. 2017) and holistically-nested edge detection (HED) (Xie and Tu 2015) models were proposed as binary classification models for edges and non-edges by fuzing multi-level information based on classical networks. Yu et al. (2017) modified ResNet to enable the extraction of multi-category edges, achieving good segmentation results in the SBD and Cityscapes datasets. Liu et al. (2019) used VGGNet as the base network to synthesize multiscale and multi-level image information and achieved accurate edge-detection using rich convolutional features (RCF). Furthermore, many datasets have been proposed to evaluate the reliability of edge-detection algorithms (Liu et al. 2019).

Methodology

In this paper, we propose a DCEFE-Net designed to extract precise building-edges from HRRSI. HRRSI contains abundant information, with shallow features capturing spatial structural characteristics for edge localization, and deep features accurately identifying boundary regions. Both types of features are essential for multiscale building-edge-detection. Therefore, we propose an edge-detection architecture consisting of four blocks. To extract spatial structural information, we incorporated a spatial attention (SA) module into block 1. In block 4, we introduced a channel attention (CA) module to capture channel dependencies and recalibrate feature responses, thereby enhancing the accuracy of edge-detection. Furthermore, to achieve a better representation of multiscale edge features, feature aggregation is necessary. However, fuzing edge features directly from different levels can introduce noise from the lower-level features. To effectively preserve the edge features, we employed dense connections between and within the main blocks to prevent the loss of edge information. In addition, we propose an edge-aware feature enhancement (EAFE) module, that enhances edge features using dilated convolutions. This module combines the enhanced features with the upsampled output from the subsequent layer, thereby reducing background noise in the shallow features. This approach enables the network to focus on extracting precise building-edge features.

Architecture overview

Based on the HED (Xie and Tu 2015): for image edge-detection, we propose an end-to-end network structure called DCEFE-Net that can extract fine building-edges from HRRSI. The overall framework is illustrated in Figure 1.

Figure 1. Overview of the DCEFE-Net architecture.

DCEFE-Net comprises four main encoder blocks, each comprising a stack of 3 × 3 convolutional layers with batch normalization and a ReLU function. To better capture both the shallow and deep features of the image, we introduce the SA module in block 1, which extracts spatial structural information from the lower layers through three convolutional operations. Additionally, we incorporated the CA module into block 4 to capture high-level semantic information toward the end of the network structure. To address the issue of blurry edge prediction caused by pooling layers incorrectly identifying non edge points in the background of HRRSI, we introduced dense connections between the outputs of different sections. This was achieved by incorporating 1 × 1 convolutional layers to establish dense connections, effectively addressing the potential loss of edge information during the convolutional process and improving the network’s ability to preserve edge information. The output of each main block was then passed through the EAFE module. This module enhances the building- edge features and combines the edge feature maps from different scales to produce the final building-edge-detection results. The structure of the EAFE module is depicted in Figure 2.

Figure 2. The structure of the EAFE module.

The EAFE module uses the output of each main block as input. Because the edge feature scales vary across different main blocks, shallow features may not capture fine building-edges, whereas deep features may disperse the edge information owing to a large number of parameters. By employing the EAFE module, we can enhance the edge features within the convolutional layers, enabling the effective utilization of multiscale feature layers. This ensures that both fine-scale and high-level edge information are preserved and utilized in the network. To handle single-scale feature inputs in the EAFE module, we incorporated N dilated convolutional layers with different dilation rates to extract the edge features at multiple scales. These features were then combined with the original feature residuals to fuze the multi-layered edge information. The dilation rate for the n^th dilated convolutional layer was determined as r_n = max (1, r₀ × k), where r₀ represents the base dilation rate. Subsequently, based on the network depth where the enhanced edge features were located, a multi-level upsampling process was performed to obtain the edge extraction results at that particular scale. Multi-level upsampling involves two subblocks, each consisting of a convolutional layer and a transposed convolutional layer. In particular, subblock 1 includes a 1 × 1 convolutional layer followed by the ReLU function, whereas the transposed convolutional layer has a size of s × s, and the number of filters is set to 16. When the difference in scale (DS) between the feature values and ground truth exceeds 2, the enhanced features are input into subblock 1. Through multiple iterations with the filters in subblock 1, the DS is reduced to 2 and is further propagated to sub-block 2. The only difference between subblocks 1 and 2 is that subblock 2 had a single filter. When DS reaches 2, the convolutional and transposed convolutional layers in subblock 2 generate an edge-enhanced layer of the same size as the ground truth, effectively enhancing the edges.

Feature extraction block

This section introduces two modules in DCEFE-Net: the CA module, which scores the channels, and the SA module, which extracts low-level spatial structural features. The SA module was introduced after two convolutional layers and a pooling layer in block 1. Its purpose is to enhance the building details while minimizing edge loss. The CA module was introduced after the convolutional operations in block 4. It ranks the importance of the features and selects effective high-level features during the edge-detection process. Finally, the dense connection module effectively combines features from different levels and generates building-edge feature maps. High-level semantic features play a crucial role in improving edge-detection accuracy (Bell et al. 2015; Marcu and Leordeanu 2016). However, in deep feature learning, the receptive field of CNNs often exceeds the effective receptive field, leading to the underutilization of global image information (Chen et al. 2018). To address this issue, we input the HRRSI into DCEFE-Net and performed basic feature extraction using the first two convolutional layers and the SA module. Subsequently, after obtaining multiscale deep features through ten convolutional layers, we utilized the CA module to selectively filter the acquired deep features, effectively harnessing the deep information that is relevant for accurate edge-detection.

The CA module learns the significance of each channel feature by enabling channel interaction, and adjusts the strength representation of the channel features accordingly. This module is illustrated in Figure 3. The input features were preserved to the maximum extent through global average pooling, and the resulting channel descriptors are fed into two convolutional layers to capture channel dependencies. The CA module assigned greater weights to channels that demonstrated a high response to edge information. The formula for computing channel weights in the CA module is as follows: (1) $$\omega =\sigma \left({\mathit{F}}_{\left\{f_j\right\}}\left(f\right)\right)$$(1)

Figure 3. CA module structure for high level information filtering, and SA module structure for extracting the underlying information.

Here, σ represents the sigmoid activation function, and F denotes the convolution operation. The input image is divided along the channel dimension into standard convolution layers $f_j$ of size 1 × 1 with normalized weight values denoted by j. (2) $$\mathit{\text{CA}}=\sigma \left(f_{j2}\left(\xi \left(f_{j1}\left({\mathit{v}}^h,P_1\right)\right),P_2\right)\right)$$(2)

Here, ξ represents the ReLU function, and $f_{j1}\left(.,P_1\right)$ and $f_{j2}\left(.,P_2\right)$ correspond to Conv 1 and Conv 2, respectively. Conv 1 serves as the dimension-reduction layer, whereas Conv 2 acts as the dimension-augmentation layer. $P_1$ and $P_2$ represent the parameters of the CA module. By performing global average pooling on individual feature channels, the channel statistics ${\mathit{v}}^h$ were obtained. The final output of the high-level features (denoted as $\tilde{C_{\mathit{\text{high}}}}$) was obtained by weighting the CA module using the following formula: (3) $$\tilde{C_{\mathit{\text{high}}}}=\mathit{\text{CA}}·C_{\mathit{\text{high}}}$$(3)

Remote sensing images typically contain abundant foreground and complex background information. Relying solely on the deep features extracted by the CA block may lead to a significant loss of detailed information after the upsampling and convolution operations. An imbalance between the edge and deep features may also result in suboptimal edge prediction results. To address these issues and obtain more effective spatial information, the low-level features obtained after the convolution layer in block 1 were directly fed into the SA block to capture the spatial details of the image. This helped generate more distinctive features that compensate for the loss of spatial information caused by stacked convolutions and upsampling operations. The structure of the SA block is shown in Figure 3.

First, the spatial information of the low-level feature maps was concentrated and compressed using max-pooling and average-pooling operations. The resulting features were then enhanced by applying a sharp filter of size s × s to emphasize the building- edge features. Next, a convolution operation and sigmoid activation function were applied to reduce the number of feature maps in each channel. Here, ${\mathrm{P}}_3$ represents a parameter in the SA modules. Finally, the output is the weighted sum of the SA module on the feature maps of $\tilde{C_{\mathit{\text{low}}}},$ as shown in the following formula: (4) $$\mathit{\text{SA}}=\sigma \left(P_3\left(f_s\left(\left[I_{\mathit{\text{avg}}}^s;I_{\mathrm{\text{max}}}^s\right]\right)\right)\right)$$(4) (5) $$\tilde{C_{\mathit{\text{low}}}}=\mathit{\text{SA}}·{\mathrm{C}}_{\mathit{\text{low}}}$$(5)

Loss function

To evaluate the relationship between the model effects and losses, we used the loss function proposed by He et al. (2022) to optimize the network learning process. The loss function for DCEFE-Net is expressed as follows: (6) $${\mathcal{L}}_{\mathrm{\text{DCEFE}}}={\sum }_{z=1}^Z w_1\mathcal{L}\left(R_z,Y_z\right)+w_2\mathcal{L}(R,Y)$$(6)

The variable z represents the number of convolutional layers in the DCEFE-Net, where 1 ≤ z ≤ Z. $R_z$ denotes the edge-detection outcome of the side output layer z in each convolutional layer at a specific scale, while R represents the multiscale edge outcome of the fused intermediate detection results. $\mathcal{L}\left(.\right)$ denotes the difference function between the predicted and ground truth values of edge mapping, and $w_1,$ $w_2$ denote the fusion weights of the z-layer edge-detection and fused results, respectively. (7) $$Y={\sum }_{z\hspace{0.17em}=\hspace{0.17em}1}^Z{Y}_z$$(7) where Y denotes the ground truth value for the building-edge detection, which can be decomposed into z binary edge maps. Z denotes the scale size and $Y_z$ denotes the ground truth edge map, resulting in the side output layer z for a particular scale.

Experiments

Datasets

We used two challenging datasets, the WHU Aerial image dataset (Ji et al. 2019) and the Inria Aerial image labeling dataset (Maggiori et al. 2017), to train and evaluate the model.

1. WHU Aerial Image Dataset:

The WHU building dataset comprise aerial and satellite imagery data, including building samples. The dataset covers 450 km² of Christchurch, New Zealand, and consists of more than 220,000 independent buildings, as illustrated in Figure 4. We used HRRSI with a spatial resolution of 0.3 m. The dataset comprised 8189 RGB images, each with a size of 512 × 512. Out of these images, 4000 containing buildings were filtered and divided into a training dataset of 3400 images, a validation dataset of 600 images, and a test dataset of 550 images.

• 2. Inria Aerial Image Labeling Dataset (IAIL):

The IAIL dataset contains 180 HRRSI of five cities, namely West Tyrol, Chicago, Vienna, Kitsap County, and Austin, each with an image size of 5000 × 5000 pixels. Because the IAIL dataset includes images from Europe and the WHU dataset comprises images collected from the Southern Hemisphere, they complement each other. The IAIL dataset has a similar spatial resolution and surface coverage to the WHU dataset; however, it covers a more complex landscape and architectural style. Therefore, the IAIL dataset was cropped to the same 512 × 512 pixel size as the WHU dataset sample and filtered to include 8019 images as training data, 1419 images as validation data, and 576 images as test data. It was used to evaluate the generalization ability of DCEFE-Net under different image acquisition methods, scene complexities, and building structures. Samples of the cropped images are shown in Figure 5.

Figure 4. Samples of the WHU aerial image data set.

Figure 5. Samples of the IAIL aerial image data set.

Comparative methods and evaluation metrics

Based on the WHU and IAIL datasets, DCEFE-Net’s performance was compared with that of five state-of-the-art deep models. The main conclusions of these comparisons’ are summarized as follows:

HED (Xie and Tu 2015): The HED network is based on the VGGNet architecture and combines fully CNNs and a deep supervised model to achieve edge-detection in using natural images. This solves the problem of the difficult representation of multiscale and multilevel features present in the overall image, which ultimately improves the network’s learning efficiency.

DexiNed (Soria et al. 2020): The dense extreme inception network (DexiNed) is inspired by the HED and the Xception networks. DexiNed consists of six main blocks, each of which is closely connected to a subblock. The output of each main block is passed through the upsampling module to obtain an intermediate edge mapping map, which finally constitutes the fused edge mapping result.

LPCB (Deng et al. 2018): The learning to predict crisp boundaries (LPCB) addresses the problem of over-predicting thick edges in the edge-detection process. It introduces a new edge-detection loss function to balance the representation of the boundary and background classes in the training data. A hierarchical network architecture enables the generation of thin edges without post processing requirement.

RCF (Liu et al. 2019): A richer convolutional feature (RCF) network detect image edges using a rich convolutional feature hierarchy that captures multilevel and multiscale information. This allows for overall edge prediction from image to image, making full use of the available information.

BDCN (He et al. 2022): The bidirectional cascade network (BDCN)utilizes labeled edge pixels for supervision at specific scales. The network also introduces a scale enhancement module to efficiently learn multiscale representations at each layer during the training process.

Results and discussion

In this paper, we assessed the performance of DCEFE-Net using various metrics: overall accuracy (OA), recall, precision, F-measure, and intersection over union (IoU). We used two evaluation methods, strict and relaxed, for each model tested on the WHU and IAIL datasets, respectively. Strict evaluation calculates the results based on the exact correspondence between the predicted and test data. Relaxed evaluation has two components, precision and recall. Relaxed precision refers to the proportion of actual building-edge pixels that are detected as building-edge pixels within ρ pixels, while relaxed recall refers to the proportion of predicted building-edge pixels that are actual building-edge pixels within ρ pixels (Arbeláez et al. 2011). In all of the experimental models, the relaxation parameter ρ was set to 3 pixels.

Evaluation metrics

Inspired by Wen et al. (2021), DCEFE -Net’s effectiveness in extracting building-edges is measured using the aforementioned five metrics, specifically: OA, recall, precision, F-measure, and IoU. The evaluation metrics can be described using the following equations: (8) $$\text{Recall =}\frac{\mathit{\text{TP}}}{\mathit{\text{TP}}+\mathit{\text{FP}}}$$(8) (9) $$\text{Precision =}\frac{\mathit{\text{TP}}}{\mathit{\text{TP}}+\mathit{\text{FN}}}$$(9) (10) $$\mathit{\text{OA}}=\frac{\mathit{\text{TP}}+\mathit{\text{TN}}}{\mathit{\text{TP}}+\mathit{\text{FP}}+\mathit{\text{TN}}+\mathit{\text{FN}}}$$(10) (11) $$\mathit{\text{IoU}}=\frac{\mathit{\text{TP}}}{\mathit{\text{TP}}+\mathit{\text{FP}}+\mathit{\text{FN}}}$$(11) (12) $$\text{F‐measure}=\frac{2\times \text{Precision}\times \text{Recall}}{\text{Precison}+\text{Recall}}$$(12)

The following definitions were used to evaluate the performance of the algorithm in detecting building-edges: true positives (TP), false positives (FP), false negatives (FN), and true negatives (TN). TP indicates the number of pixels that agree between the detected edges and the true value of the ground building-edge. FP indicates the number of pixels that are inconsistent between the detected edge and the true value of the ground-building-edge. FN indicates the number of non-conforming pixels between the non-building and non-building-edges detected in the ground truth. TN indicates the number of detected background pixels that agree with the true value of the non-building-edges of the ground surface. The F-measure takes into account both the optimal dataset scale (ODS) and the optimal image scale (OIS). ODS involves selecting a fixed threshold and applying it to all images, whereas OIS involves selecting a threshold for each image that maximizes the F-measure for the entire dataset.

Training details

PyTorch 1.11.0, PyTorchvision 0.12.0, and CUDA 10.2 were used to implement MSD-Net in this study. All the experiments were conducted on an NVIDIA 2070 GPU with 16 GB of memory. The training data were resized to 512 × 512 pixels, and the batch size was set to 4. The AdamW optimizer was used to train all network structures because it is more effective than the commonly used Adam optimizer in terms of convergence and regularization.

To optimize the model, several hyper parameters were set: the learning rate was set to 1e-3, betas were set to values between 0.9 and 0.999, eps was set to 1e-6, weight-decay was set to 0, and correct-bias was set to true. Although the model’s performance improved initially, after several iterations the improvement stopped. To address this issue, the ReduceLROnPlateau callback function was implemented, which further reduced the learning rate. This function relies on four key parameters: the monitor, set to val_accuracy to track the metric; the factor, set to 0.2 scale the learning rate; the patience, set to 20 determine the number of stalled epochs before activation; and min_lr, set to 0.001 to prevent unnecessary and unhelpful reductions in the learning rate.

The experiments revealed that higher epoch counts improved accuracy. However, at 120 epochs, the rate of improvement in the accuracy decreased and then gradually leveled off. To reduce training costs, the final epoch count was set to 100. The implementation also set verbose to 1, mode to auto, and threshold epsilon to 0.0001 to determine the “plain zone” of detection. The optimization method employed was WarmUP, which prioritizes the use of a smaller learning rate and gradually increases it based on changes in the input data. When the model was stabilized, the method used a lower learning rate to obtain the local optimal point.

Comparative experiments

To quantitatively evaluate the performance of DCEFE-Net, we compared it with several existing methods on two datasets, WHU and IAIL. The results of the comparisons, including specific metrics and building-edge-detection plots, are presented in Sections 5.3.1 and 5.3.2, respectively. To ensure experimental fairness and consistency of the metrics, we set the binarization threshold to 0.5 and performed post-processing using the non-maximum suppression (NMS) method.

1. Results on WHU Dataset

Table 1 presents the building-edge detection metrics for the six models LPCB, HED, RCF, DexiNed, BDCN and DCEFE-Net, on the WHU dataset. All networks achieved strict OA values above 90%, with DCEFE -Net achieving the highest value of 98.33%, outperforming the other models. Moreover, after relaxed processing, DCEFE-Net performed well for all metrics except the recall index. Specifically, the relaxed OA and relaxed precision were 98.2% and 80.11%, respectively, with a precision improvement of 7.37% compared to DexiNed. DCEFE-Net has a lower recall than LPCB because of its thinner building-edges and a significant reduction in the number of FP samples. However, the F1 score of the DCEFE-Net was 72.94%, which was 26.56% better than that of the LPCB model. This suggests that DCEFE-Net is more effective in handling non-building-edge noise constraints. In addition, DCEFE-Net outperformed DexiNed in terms of the relaxed F1 score and IoU metrics, with improvements of 5.93% and 6.71%, respectively. Moreover, the strict-OA, prediction, and F1 values of the DCEFE-Net were 1.41%, 4.83% and 4.38% higher than those of the recently proposed BDCN, respectively. This suggests that DCEFE-Net is effective in enhancing the edge-detection performance of DexiNed.

Figure 6 shows the building-edge-detection results for the six networks on the WHU dataset. LPCB and RCF produced noisy edges and were too sensitive to impervious surfaces such as roads. The HED and DexiNed performed better, producing more accurate building contours. However, as show in Figure 7, HED and DexiNed continued to exhibit omissions and misclassifications in detecting fine building-edges, inaccurately controlled of building inflection points, and the possibility of misclassifying impervious surfaces as building-edges. By contrast, the proposed DCEFE -Net achieved better performance on the WHU dataset, reducing misclassification while preserving thin building-edges, accurately detecting complex building-edge inflection points, and matching labeled data in the dataset.

• 2. Results on the IAIL dataset

Figure 6. Example of building-edge mapping generated by six models in the WHU dataset. From left to right are the original image, ground truth labels, DexiNed, LPCB, HED, RCF, BDCN and DCEFE -Net.

Figure 7. The plot of the final building-edge-detection results generated by DexiNed, LPCB, HED, RCF, BDCN and DCEFE -Net after NMS processing. The red in the figure indicates the building-edge-detection labels, and the green shows the detection results of each model.

Table 1. For the evaluation results of the test dataset in the WHU dataset.

Model	Scheme	OA(%)	Prediction (%)	Recall(%)	F1(%)	IoU(%)
LPCB	strict	92.90	33.09	95.63	46.38	32.18
	relaxed	97.52	75.98	63.89	68.63	53.71
HED	strict	97.76	41.95	81.03	53.98	38.82
	relaxed	97.66	69.61	62.06	69.23	54.32
RCF	strict	97.38	38.96	83.38	51.60	36.60
	relaxed	97.83	79.22	68.37	67.77	52.78
BDCN	strict	96.92	44.59	83.29	52.87	32.76
	relaxed	97.28	68.90	68.18	67.76	52.79
DexiNed	strict	98.23	46.31	75.99	59.14	43.68
	relaxed	98.17	72.74	62.80	67.01	51.90
DCEFE -Net	strict	98.33	49.42	79.17	57.25	41.94
	relaxed	98.20	80.11	67.36	72.94	58.61

Each model was subjected to strict and relaxed. The best results are marked in bold.

Table 2 presents the building-edge-detection metrics for the six models: LPCB, HED, RCF, DexiNed, BDCN and DCEFE -Net, on the IAIL dataset, which included buildings with vegetation occlusion and more complex city types. As a result, the relaxed OA and relaxed precision are 0.37% and 0.84% lower, respectively, than those on the WHU dataset. DCEFE-Net nonetheless still achieved the best detection performance, with higher precision than LPCB, HED, RCF, BDCN and DexiNed by 18.27%, 18.80%, 19.65%, 20.51%, and 19.58%, respectively, even under strict metrics. In addition, the strict and relaxed OA values of DCEFE-Net were 6.7% and 4.76% higher than those of the BDCN, respectively. Although the strict recall of DCEFE-Net was 0.15% lower than that of HED, the overall results were better.

In Figure 8, the building-edge detection results for the five networks on the IAIL dataset are presented. DexiNed, LPCB, and RCF are sensitive to noise, resulting in rough building-edge-detection. Although the HED performs better in suppressing noise, the details of the roof are not sufficiently expressed, and the building-edges are confusing. DCEFE -Net has a better response to noise and can detect clear building-edge patterns while determining fine independent building-edges more accurately. The details are shown in Figure 9. Owing to the concealment of vegetation and the complexity of the background, the building-edges in the IAIL dataset cannot be completely represented. DexiNed, LPCB, HED, and RCF cannot achieve clear and accurate identification of low freestanding building-edges, resulting in blurred and rough building-edges. For the first column of large independent buildings with complex boundary morphology, the correct detailed building outline is also not predicted, and there are a large number of FP pixels. By contrast, DCEFE-Net excels at accurately edge detection for independent low buildings and distinguish them from neighboring buildings without merging them into larger segments. In addition, it can achieve good edge-detection results for of large buildings with complex edges, providing the most accurate and complete representations of the buildings’ thin edges.

Figure 8. Example of building-edge mapping generated by six models in the IAIL dataset. From left to right are the original image, ground truth labels, DexiNed, LPCB, HED, RCF, BDCN and DCEFE -Net.

Figure 9. Plot of the final building-edge-detection results generated by DexiNed, LPCB, HED, RCF, BDCN, and DCEFE-Net after NMS processing. Red indicates the building-edge-detection labels and the green shows detection results of each model.

Table 2. For the evaluation results of the test dataset in the Inria dataset.

Model	Scheme	OA(%)	Prediction (%)	Recall(%)	F1(%)	IoU(%)
LPCB	strict	93.26	30.32	79.86	39.97	28.52
	relaxed	95.05	60.15	77.18	67.12	52.41
HED	strict	92.02	29.79	93.86	40.07	28.51
	relaxed	94.65	57.50	88.55	68.57	54.00
RCF	strict	92.14	28.94	85.21	38.45	27.35
	relaxed	94.28	56.05	81.77	65.63	50.71
BDCN	strict	90.56	28.08	88.15	36.42	26.44
	relaxed	93.07	53.26	81.52	64.91	49.95
DexiNed	strict	91.75	29.01	91.29	38.85	27.63
	relaxed	94.36	56.21	87.53	67.27	52.54
DCEFE -Net	strict	97.26	48.59	93.71	62.09	47.29
	relaxed	97.83	79.72	89.16	83.53	72.63

Each model was subjected to strict and relaxed, and the best results are marked in bold.

Ablation experiments

The proposed DCEFE-Net incorporates two attention modules, namely the SA and CA modules, as well as the EAFE module. Processing of the WHU dataset using DCEFE -Net is illustrated in Figure 10. To validate further the performance of these modules, different combinations were experimentally evaluated using DCEFE-Net, as listed in Table 3. The baseline represents the performance of the backbone network without any of the three modules. Compared to the baseline, adding the SA module to block 1 significantly improved the edge-detection performance, with an increase of 3.59% in the OA and 2.66% in the F1 score. Adding the CA module alone to block 4 improved the OA by 3.71% and the F1 score by 2.71%. When both the SA and CA modules were simultaneously included, the OA improved by 4.28% and the F1 score improved by 3.54%. When the EAFE module was added alone, an OA increase of 4.04% and an F1 increase of 3.16% were achieved. The network performance is best when the value of the expansion convolution r in the EAFE module is taken as (3, 6, 9), and the network accuracy starts to show a decreasing trend as the value of r increases. Integrating all three modules into the network structure resulted in an OA improvement of 5.41% and an F1 improvement of 4.18%, respectively. In summary, incorporating attention modules and the EAFE module into the backbone network effectively enhanced the edge-detection performance.

Figure 10. The edge mapping of DCEFE -Net was tested in the WHU dataset. Block 1 to block 4 are the outputs of the four main modules that went through the EAFE module, fusion denote the connection and fusion between the outputs, and the average is the average of all predicted values.

Table 3. The ablation study of DCEFE-Net on the WHU dataset investigates the impact of different components: SA (Spatial Attention) module, CA (Channel Attention) module, and EAFE (Edge-Aware Feature Enhancement) module.

Method	SA	CA	EAFE	OA	F1
Baseline				92.26	66.31
DCEFE-Net	✓			95.85	68.97
DCEFE-Net		✓		95.97	69.04
DCEFE-Net			✓	96.30	69.47
DCEFE-Net	✓	✓		96.54	69.76
DCEFE-Net	✓		✓	96.72	69.87
DCEFE-Net	✓	✓	✓	97.67	70.49

By training DCEFE-Net on the WHU dataset, the effect of the dilation rate factor r₀ on the EAFE module was determined. The experimental results, as shown in Table 4, reveal that the network achieves optimal performance when r₀ is set to three, with a fixed number of three layers for the dilated convolutions. As the value of r₀ increased beyond three, the network performance started to gradually decline.

Table 4. Investigating the impact of EAFE module parameters on edge detection performance on the WHU dataset.

r0	Rate	OA	F1
0	(1,1,1)	95.06	68.69
2	(2,4,6)	95.39	68.77
3	(3,6,9)	95.48	68.89
4	(4,8,12)	95.27	68.71

Conclusion

We propose DCEFE-Net to address the issue of small and independent buildings being easily overlooked in building-edge-detection due to scale differences and blurred boundary details.

The main concept of DCEFE-Net is to enhance the detection of separate building structures and separate closely connected building-edges by employing multi-level dense connectivity blocks and an EAFE. Dense connectivity blocks were designed to prevent the loss of architectural edge features in the HRRSI. The inclusion of the SA and CA modules in the main module ensures a better focus on the multiscale edge features. Moreover, the proposed edge-aware EAFE effectively combines edge features from different levels and suppresses background noise in the images. Validation using the WHU dataset and the IAIL dataset indicates that the proposed method enhances edge-detection performance, enabling the generation of more precise building boundary masks and capturing detailed local structural information of building-edges. DCEFE-Net fuses several different scales of building edge features, and by analyzing the similarities and differences between the multi-scale edge features, it effectively solves the adhesion problem between small-scale buildings and obtains a better extraction effect of small buildings. However, it should be noted that DCEFE-Net may exhibit some sensitivity to building shadows during the edge-detection process, which warrants further investigation in future research.

Disclosure statement

The authors declare no competing financial interests.