MFFNet: a building change detection method based on fusion of spectral and geometric information

Abstract

Accurate detection and extraction of changes in buildings heights is important in monitoring construction (both legal and illegal) and assessing disasters. It is also important information for updating real 3D scenes. However, when using remote sensing images, shadows, vegetation and objects with similar spectral and morphological characteristics as buildings can cause false detections, omissions and incomplete patch edges. To address this issue, we develop the multiscale feature fusion network for dual-modal data (MFFNet), which has two main aspects: (1) The multi-dual-modal feature fusion module detects changes in features with similar spectral and morphological characteristics as buildings. This mitigates false detections by making the model more aware of areas where the elevation has changed over time. (2) Because building extraction is affected by shadows and vegetation, we designed a multiscale feature shuffle module. It takes multiscale features and establishes relationships between neighbouring pixels using the pixel-shuffle algorithm, then fuses and reorganizes the multiscale features to highlight the relationships between global contexts, thereby mitigating the problem of building occlusion by shadows. Comparative experiments show that MFFNet achieves better results on GF7-CD and 3DCD datasets than other similar methods. The proposed method can more accurately monitor building changes over large areas.

Keywords:

• Building change detection
• digital surface model
• dual-modal data fusion
• multiscale feature shuffle

1. Introduction

With the rapid development of cities, there is a need for up-to-date building information. This is crucial for various purposes, including detecting illegal construction (Zhu et al. 2022), assessing disasters (Shafique et al. 2022), monitoring construction progress and updating real 3D scenes(He and Chen 2019). Currently, building change detection (BCD) primarily relies on 2D remote sensing images (RSIs). There are several effective methods; for example, RSICD relies on geographically aligned multitemporal RSIs to detect changes in the ground surface (Singh 1989). High-resolution remote sensing imagery (HRRSI) can be used to quickly identify building changes at a large scale; however, it is inadequate for applications that require height change data, such as for detecting illegal private constructions, disaster damage analysis, construction progress assessment and realistic 3D scene updating.

According to the data source, BCD can be based on spectral information or geometric information. Using spectral information for change detection (CD) requires the extraction of features from RSIs and analysis of the differences between images taken at different times. In the early years, due to the limitations in RSI resolution, independent pixels were commonly used as the basic unit. For example, a pixel-based CD algorithm would perform algebraic operations on a pixel-by-pixel basis to obtain a change difference map. Then, a threshold value would be selected to determine the corresponding changes, such as by image differencing (Coppin et al. 2004) or regression analysis (Lu et al. 2004). With the commercialization of high-resolution and submetre remote sensing satellites, object-oriented analysis techniques have been introduced into the field of RSICD. The basic unit of CD has transitioned from pixels to objects (Chen Z et al. 2022). Object-oriented CD primarily utilizes spectral, texture and spatial context information to obtain change information at the object level (Jabari et al. 2019). However, CD based on spectral information suffers from the following two problems: (1) Objects that exhibit spectral and morphological characteristics similar to buildings, such as roads, can be erroneously identified as buildings, leading to false detections. When it comes to changes in spectral characteristics, both houses and roads cause changes in surface spectral features. The same applies to morphological characteristics. (2) When shadows and vegetation are present around the buildings in an image, the edges of the extracted patches tend to be incomplete. This is primarily due to noticeable differences in spectral values between areas with shadows and those without, as well as between areas with and without vegetation. These variations, which commonly occur at the edges of buildings, contribute to incomplete and inaccurate edge detection in CD results.

Change detection based on geometric information extracts elevation features from 3D data and analyses their differences across time (Zuxun et al. 2022). This can be used for tree growth monitoring, earthquake damage assessment and terrain and building CD. Geometric CD can be based on geometric information alone or a fusion of geometric and spectral information. Geometric-only CD methods, such as pixel-based height-difference methods, utilize a single grid point as a unit for identifying changes. These techniques have various applications, including 3D CD in urban areas (Chaabouni-Chouayakh et al. 2010, 2011) and building change analysis (Jung 2004; Sasagawa et al. 2013). They identify potential changes by deriving height difference scores. Geometric information plays a crucial role in CD and enhances its accuracy. However, in many cases, errors can make it difficult to obtain good results using geometric information alone. Therefore, the use of additional spectral information can help to compensate for such errors, further enhancing CD reliability. Pang et al. (2018) proposed a CD method based on a digital surface model (DSMs) and original image. It uses the graph cut optimization algorithm to extract the feature change area, which is then combined with the original image data to exclude the influence of trees. Finally, four building change categories were obtained: new construction, heightening, demolition and lowering. However, the initial results of these methods rely heavily on geometric comparisons and missed detections cannot be retrieved in the subsequent refinement process. In other words, these methods do not effectively combine geometric and spectral information. Qin et al. (2016) proposed an object-based 3D BCD method for multiperiod stereo images. The MeanShift algorithm is applied for segmentation to obtain the objects for each phase of data. This is combined with feature extraction with decision trees and support vector machine (SVM) for supervised classification. Finally, comparative analysis is performed. In this type of method, DSMs are typically integrated as an additional channel in the classification or detection method. The classification methods used include SVM, decision trees and others. In summary, BCD encounters challenges when relying on geometric information: (1) The presence of geometric errors (e.g. example, registration error) reduces accuracy; (2) combining spectral and geometric information is more accurate but is challenging to perform.

Accordingly, we propose a multiscale feature fusion network (MFFNet) for dual-modal data, which includes the dual-path feature extraction network (DPEN), the multi-dual-modal feature fusion module (MDFM) and the multiscale feature shuffle module (MFSM). It conducts CD of height-varying buildings by effectively combining spectral and geometrical information and considering the relationship between global contexts. The main contributions of this work are as follows:

• Geometric information is used as a crucial complement to spectral information for BCD. To address challenges in BCD results such as false detections, omissions and incomplete edges, we propose a method that combines spectral and geometric information through the extraction of building elevation features.

• We developed MFFNet to effectively combine dual-modal data. MFFNet consists of three main modules: the dual-path feature extraction network (DPEN), the MDFM and the MFSM. The DPEN is designed to extract features separately from the dual-modal data. Subsequently, the MDFM is utilized to combine the extracted image and elevation features more effectively. Finally, the MFSM is designed to make the model aware of the relationships between global contexts.

• To validate the reliability of MFFNet, we conducted experiments using two datasets. Additionally, we conducted ablation experiments for each module to verify the reliability and stability of our model.

2. Related work

In recent years, the rapid development of Earth observation technology has provided increasing amounts of high-resolution imagery from various satellite sensors, which has greatly enriched CD data sources (Chen et al. 2020). Different data sources contain different features; for example, optical images contain rich spectral, texture and shape features of ground objects, while nonoptical image sources, such as LIDAR or SAR, can provide information based on different ground physical mechanisms and are less affected by weather conditions than optical sensors (Zhang et al. 2021). We can categorize CD according to whether it is based on single or multiple data sources.

2.1. Change detection based on a single data source

We introduce CD based on a single data source from two aspects: (1) CD based on geometric information and (2) CD based on spectral information. CD based on geometric information can calculate metrics in two common ways: height difference and Euclidean distance. Turker and Cetinkaya (2005) utilized DSMs generated from pre- and post-earthquake stereo aerial images to detect collapsed buildings. However, pixel-based height-difference methods are prone to false detections due to alignment errors and DSM inaccuracy. To address this issue, researchers have proposed window/object-based methods. For example Tian et al. (2010) used the minimum height difference in a moving window to reduce DSM noise at object boundaries. In their subsequent work, the use of objects obtained from panchromatic images as height-difference units further reduced false detections. Another problem with height-difference methods is that they are sensitive to alignment and matching errors. The more theoretically rigorous surface Euclidean distance methods can compensate for such shortcomings to some extent. Another approach, presented by Waser et al. (2008), applies the Euclidean distance between surfaces to estimate forest volume changes based on DSMs obtained from image matching. This inter-surface Euclidean distance-based method is robust to alignment errors in top-view 3D data and can be applied to full 3D data, although it is computationally complex and time-consuming. Therefore, although using inter-zone surface Euclidean distances is theoretically more rigorous than using height differences, the latter is still the most convenient and effective method.

While different types of images (optical, SAR, multispectral, etc.) are used in CD, most of the deep learning algorithms used for urban CD only use optical images (Hafner et al. 2022). Spectral information is now increasingly used in CD. These methods include the decision tree (Ye et al. 2018; Xie et al. 2019), Random Forest (Désir et al. 2012; Bai et al. 2018; Feng and Li 2019) and SVM methods (Zhigao et al. 2006; Bovolo et al. 2008; Wei et al. 2015). With the rise of deep neural network research, many scholars have applied deep learning methods to CD. Gedara et al. (2022) proposed a method that integrates a hierarchically structured transformer encoder with a multilayer perception (MLP) decoder in a Siamese network architecture. This innovative approach enables efficient processing of multiscale long-range details, facilitating accurate CD. Similarly, Zhang et al. (2023) introduced a unique global-aware Siamese network (GAS-Net) aimed at generating global-aware features for efficient CD. Their approach incorporates an understanding of the relationships between scenes and foregrounds. Hou et al. (2021) developed the high-resolution triplet network (HRTNet) framework, incorporating a dynamic inception module to enhance multiscale feature information. In a separate study, Chen Z et al. (2022) proposed EDGE-Net, a pioneering method that prioritizes boundary accuracy and the integrity of change regions.

Although all these methods have achieved good detection results in specific scenarios, it is difficult to obtain ideal CD results with geometric information alone due to the presence of geometric errors. Meanwhile, spectral information can be affected by changes in features with similar spectral and morphological characteristics, as well as shadows and vegetation around buildings, leading to false detections, omissions and incomplete building edges. Therefore, some scholars have attempted to combine spectral and geometric information to improve CD accuracy.

2.2. Change detection based on multiple data sources

The different data sources that can be used for CD contain different features. This paper focuses on combining optical imagery and DSM for CD. Some methods consider both geometric and spectral information; for example, rule-based categorization (Tian et al. 2011, 2015; Nebiker et al. 2014; Lak et al. 2016), SVM (Chaabouni-Chouayakh and Reinartz 2011; Malpica et al. 2013; Tu et al. 2017), the graph cut method (Du et al. 2016) and random forest (Chen et al. 2016). However, these have strict parameter requirements, and incorrect parameter settings can lead to errors in the CD results. Yang Y et al. (2021) proposed a CD method based on multilevel segmentation of densely matched point clouds extracted from UAV images, achieving multilevel segmentation and change extraction through chromatic heterogeneity. This post-refinement method is relatively flexible and effective, with parameters that are easy to understand and adjust. Tian et al. (2013) fused changes in elevation and spectral information into a modelling framework that only requires a single metric to be adjusted to obtain CD results. Subsequently, Tian et al. (2014) employed Dempster–Shafer fusion theory to extract building changes by combining DSM elevation changes with Kullback–Leibler divergence similarity measure derived from original images. Qin (2014) proposed a CD method based on high-resolution stereo images and the LoD2 model. It utilizes an unsupervised Self-Organizing Map (SOM) to fuse multichannel metrics, including DSM and spectral features, enabling classification of different categories. Another 3D CD method (Pang et al. 2019) uses a joint hyper-pixel graph cut optimization approach. This method transforms BCD into a binary classification problem. First, a joint hyper-pixel object is acquired using the SLIC hyper-pixel segmentation method. Then, the hyper-pixel object is utilized as a processing unit to extract multidimensional change features. Finally, the optimal solution is obtained by employing the graph cut optimization framework. Zhang et al. (2021) proposed a CD framework called W-Net, which is capable of handling both homogeneous and heterogeneous remote sensing data for BCD. The bidirectionally symmetric end-to-end network architecture of W-Net supports the input of homogeneous or heterogeneous remote sensing data for 2D or 3D BCD. MAHNet, as proposed by Pan, Li, et al. (2022), combines two data sources, DSM and GF7, to perform 3D CD tasks.

The above methods combine geometric and spectral information in different ways to accomplish CD from RSIs. Yet challenges remain in effectively combining the two types of information; for example, how to effectively combine two types of information with different physical mechanisms is a challenge. MFFNet is designed to optimize the fusion of spectral and geometric information and establish robust associations with global contextual information. By doing so, we can effectively reduce false detections and omissions while improving the completeness of building edges.

3. Methodology

In this section, we introduce the proposed method in three parts: (1) The basic network architecture of MFFNet, (2) the MDFM and (3) the MFSM.

3.1. Description of the MFFNet framework

The network structure of MFFNet is shown in Figure 1, which is mainly composed of an encoder, the MDFM and a decoder. The encoder uses a dual-path feature-extraction network (DPEN), MDFM fuses image and elevation features at different scales and the decoder uses the MFSM to consider the global contextual relationships.

Figure 1. Schematic diagram of the proposed MFFNet, which is comprised of (left) an encoder consisting of the dual-path feature extraction network (DPEN), (Middle) the MDFM and (right), a decoder consisting of the MFSM.

3.1.1. Encoder

To extract features more fully in the encoding stage and to consider different data structures, we adopt the DPEN to extract the deep-level features of the HRRSI and DSM. The HRRSI can provide building shape, edge and texture details that are richer than those of ordinary optical images. Therefore, feature extraction from HRRSI uses ResNet-34 (He et al. 2016), a powerful feature extractor, to mine multilayer features such as spectral, geometric and texture features. It has four layers of residual blocks constituting {R1, R2, R3, R4}. The DSM data structure is relatively simple, covering surface information elevations other than ground-level in a single band. To prevent overfitting (Yang X et al. 2021) we use the FCN (Long et al. 2015) instead of ResNet-34 for feature extraction. It contains four layers of standard convolution {C1, C2, C3, C4}. The validity of the dual-path extraction network was verified in Subsection 4.6. To optimize the utilization of the extracted features, we pass them through the MDFM. The purpose of doing this is to effectively combine image and elevation features and highlight their differences, thereby mitigating false detections caused by different objects that exhibit the same spectral and morphological characteristics as buildings.

3.1.2. Decoder

The features from the MDFM are sent to the MFSM, which proceeds to fuse and shuffle the features at different scales, enabling the model to capture global contextual relationships. Finally, a convolutional layer and a sigmoid activation function are utilized to complete the CD task.

3.2. The multi-dual-modal feature fusion module

After going through the DPEN, rich image and elevation features are extracted from the image. However, these two feature types are not combined effectively. In MAHNet (Pan, Li, et al. 2022), they are combined using channel stacking. This combines all the extracted features, although not all features are helpful for image disparity discrimination (Hu et al. 2018). Excessive use of irrelevant features makes model training more difficult (Ronneberger et al. 2015). Therefore, it is very important to know how to efficiently utilize the two extracted different feature types in combination. We are inspired by the Convolutional Attention Block (CBAM) (Woo et al. 2018) and propose using MDFM to combine the two features more efficiently. MDFM consists of the multiple dual-modal feature-fusion module (DFM), which is described in the following section (Figure 2).

Figure 2. The DFM architecture incorporates a fusion layer, channel attention module and spatial attention module-m.

Let $x_{\mathrm{\text{Image}}}$ and $x_{\mathrm{\text{DSM}}}$ represent the image and elevation features, respectively. The dual-modality fusion is defined as: (1) $$F=f_{\mathit{\text{FL}}}\left(x_{\mathrm{\text{Image}}},x_{\mathrm{\text{DSM}}}\right)$$(1) (2) $$F^{\mathrm{\prime }}=f_{\mathrm{\text{CAM}}}\left(\mathrm{F}\right)\otimes \mathrm{F}$$(2) (3) $$F^{\mathrm{″}}=f_{\mathrm{\text{SAM}}-m}\left(F^{\mathrm{\prime }}\right)\otimes F^{\mathrm{\prime }}$$(3) where $f_{\mathrm{\text{FL}}}$ stands for the Fusion Layer, $f_{\mathrm{\text{CAM}}}$ stands for the Channel Attention function and $f_{\mathrm{\text{SAM}}-m}$ stands for the Spatial Attention-m function. The symbol $\otimes$ denotes multiplication by elements, F represents the result of the fusion layer, $F^{\mathrm{\prime }}$ denotes the result of CAM and $F^{\mathrm{″}}$ denotes the final output.

The details of the Fusion Layer are shown in Figure 3. The $f_{\mathrm{\text{FL}}}$ is defined as follows: (4) $$x_{\mathrm{\text{concat}}}=x_{\mathrm{\text{Image}}}\oplus x_{\mathrm{\text{DSM}}}$$(4) where $\oplus$ represents the concatenation operation and $x_{\mathrm{\text{Image}}}$ and $x_{\mathrm{\text{DSM}}}$ represent the features on two different branches. (5) $$f_{\mathrm{\text{FL}}}\left(x_{\mathrm{\text{Image}}},x_{\mathrm{\text{DSM}}}\right)=f_{\mathrm{\text{Dropout}}}^{\mathrm{\text{BatchNorm}}}\left({f_{\mathrm{\text{Dropout}}}^{\mathrm{\text{BatchNorm}}}(f}_{\mathrm{\text{Relu}}}^{\left(3\times 3\right)}\left({(f}_{\mathrm{\text{Dropout}}}^{\mathrm{\text{BathNorm}}}\left(f_{\mathrm{\text{Relu}}}^{\left(3\times 3\right)}\left(x_{\mathrm{\text{concat}}}\right)\right)\right)\right)$$(5) where $f_{\mathrm{\text{Relu}}}^{\left(3\times 3\right)}$ represents performing a 3 × 3 convolution layer and then activating the Relu activation function. $f_{\mathrm{\text{Dropout}}}^{\mathrm{\text{BathNorm}}}$ represents a BatchNorm layer followed by a Dropout layer.

Figure 3. Details of the fusion layer.

The Channel Attention Module is defined as: (6) $$\begin{array}{c}{f}_{\mathrm{\text{CAM}}}\left(F\right)=\sigma \left(\mathrm{\text{MLP}}\left(\mathrm{\text{Avgpool}}\left(F\right)\right)+\mathrm{\text{MLP}}\left(\mathrm{\text{Maxpool}}\left(F\right)\right)\right)\end{array}$$(6) where MLP is the weight shared and $\sigma$ represents a sigmoid function. Avgpool stands for average pooling function and Maxpool stands for maximum pooling function.

Spatial Attention Module-m(SAM-m). Inspired by CBAM-S (Pan, Cui, et al. 2022), we adapted the spatial attention module to make a better combination of the two different types of features. The formula for SAM-m is as follows: (7) $$\begin{array}{c}{f}_{\mathrm{\text{SAM}}‐\mathrm{m}}\left(F^{\mathrm{\prime }}\right)=f_{\sigma }^{\left(3\times 3\right)}\left(f_{\sigma }^{\left(3\times 3\right)}\left(f_{\sigma }^{\left(3\times 3\right)}\left(\left[F_{\mathrm{\text{avg}}}^{\prime };F_{\mathrm{\text{max}}}^{\prime }\right]\right)\right)\right)\end{array}$$(7) where $f_{\sigma }^{\left(3\times 3\right)}$ means that a 3 × 3 convolution operation is performed first, then, the sigmoid is activated as the activation function. $F_{\mathrm{\text{avg}}}$ is the Average pooling function, $F_{\mathrm{\text{avg}}}$ stands for the maximum Pooling function and [] stands for a concatenation operation.

MDFM to better control the semantic gap and reduce information redundancy, we add DFM to the skip junctions of each layer of the two branching encoders, as shown in Figure 4. The advantage of this approach is that it allows a better combination of features at the different scales of the two modalities and reduces the effect of irrelevant information. The dimensions of the feature map are: $D_i$ (64 × 64 × 64,128 × 32 × 32,256 × 16 × 16,512 × 8 × 8).

Figure 4. Structural diagram of the MDFM.

The attentional visualization of $D_i$ is shown in Figure 5, where yellow areas are those that the model has noticed. We can see that each DFM layer makes the model notice different places.

Figure 5. Visual attention maps.

3.3. Multiscale feature shuffle module

The previous section introduced the MDFM, as shown in Figure 5. After the features pass through the MDFM, the model can determine the differences. Therefore, it is crucial to explore how to fully utilize these features of different scales. However, current upsampling methods do not sufficiently consider the relationships between pixels, which may result in the loss of important information. To fully exploit the image and elevation features, we introduce the pixel-shuffle technique (Shi et al. 2016), which preserves more geometric and spectral information, thereby yielding higher quality upsampling results. The algorithmic mechanism of pixel-shuffle is shown in Figure 6.

Figure 6. Details of the pixel-shuffle operation.

In feature extraction networks (Simonyan and Zisserman 2014; Huang et al. 2017), deep convolution can extract high dimensional features; however, due to the lack of target location information, high dimensional features may not provide accurate location information. To solve this problem, the feature pyramid network (FPN) (Lin et al. 2017) has been proposed for RSICD. While some other scholars have adopted the multiscale fusion strategy (Peng et al. 2019), none of these methods can make the connections between global pixels. To solve this problem, we propose the MFSM method, as shown in Figure 7. First, multiscale features consider the relationships between neighbouring pixels by using pixel-shuffle. Second, recovered to the original size by transposed convolution of different sizes. Finally, multiscale features are fused and shuffled to capture global contextual relationships. The MFSM formulas are shown below. From left to right, the dimensions of the grey feature maps are $\{X_0^1\in R^{C_1\times \frac{H}{16}\times \frac{W}{16}},X_0^2\in R^{C_2\times \frac{H}{8}\times \frac{W}{8}},X_0^3\in R^{C_3\times \frac{H}{4}\times \frac{W}{4}},X_0^4\in R^{C_4\times \frac{H}{2}\times \frac{W}{2}}\}.$ The left-to-right yellow feature maps are $\{X_1^1\in R^{64\times \frac{H}{2}\times \frac{W}{2}},X_1^2\in R^{32\times \frac{H}{2}\times \frac{W}{2}},X_1^3\in R^{16\times \frac{H}{2}\times \frac{W}{2}},X_1^4\in R^{8\times \frac{H}{2}\times \frac{W}{2}}\}.$ The green chart sizes, in order from left to right, are $\{X_2^1\in R^{1\times H\times W},X_2^2\in R^{1\times H\times W},X_2^3\in R^{1\times H\times W},X_2^4\in R^{1\times H\times W}\}.$ The dimensions of the resulting graph are $X_{\mathrm{\text{total}}}\in R^{1\times H\times W}.$ The constraint requirements for MFSM are shown in Eq. (8)(8) $$\{\begin{array}{c} \\ X_2^1:X_0^1\stackrel{P(16)}{\to }{X}_1^1\stackrel{T(4)+\mathrm{\text{conv}}}{\to }{X}_2^1 \\ X_2^2:X_0^2\stackrel{p(8)}{\to }{X}_1^2\stackrel{T(8)+\mathrm{\text{conv}}}{\to }{X}_2^2 \\ X_2^3:X_0^3\stackrel{P(4)}{\to }{X}_1^3\stackrel{T(16)+\mathrm{\text{conv}}}{\to }{X}_2^3 \\ X_2^4:X_0^4\stackrel{P(2)}{\to }{X}_1^4\stackrel{T(32)+\mathrm{\text{conv}}}{\to }{X}_2^4 \\ X_{\mathrm{\text{total}}}=\mathit{\text{conv}}({\sum }_{n=0}^4{X}_2^n)\end{array}$$(8) : (8) $$\{\begin{array}{c} \\ X_2^1:X_0^1\stackrel{P(16)}{\to }{X}_1^1\stackrel{T(4)+\mathrm{\text{conv}}}{\to }{X}_2^1 \\ X_2^2:X_0^2\stackrel{p(8)}{\to }{X}_1^2\stackrel{T(8)+\mathrm{\text{conv}}}{\to }{X}_2^2 \\ X_2^3:X_0^3\stackrel{P(4)}{\to }{X}_1^3\stackrel{T(16)+\mathrm{\text{conv}}}{\to }{X}_2^3 \\ X_2^4:X_0^4\stackrel{P(2)}{\to }{X}_1^4\stackrel{T(32)+\mathrm{\text{conv}}}{\to }{X}_2^4 \\ X_{\mathrm{\text{total}}}=\mathit{\text{conv}}({\sum }_{n=0}^4{X}_2^n)\end{array}$$(8) where $P()$ represents the pixel-shuffle operation; ∑ represents the concatenation operation; T() represents the transposed convolution; conv represents 1 × 1 convolution; and $X_1^1,X_1^2,X_1^3,X_1^4$} represents the feature maps that have been adjusted by the pixel-shuffle algorithm.

Figure 7. Structural diagram of the MFSM.

4. Experiments

4.1. Datasets

Validation was carried out on two datasets: (1) GF7-CD, which was produced for this study and (2) 3DCD, which is a publicly available dataset. The two datasets result from different sensors, which is helpful for validating the paradigm ability of the proposed model.

4.1.1. GF7-CD dataset

This dataset comprised RSIs of Jinhua City, Zhejiang Province, China. They cover an area of 47.76 $\mathrm{k}{\mathrm{m}}^2$ and contain multiple types of features and complex surfaces. After strict pre-processing of the experimental data, a set of BCD datasets containing both HRRSI and DSM was produced. The spatial resolution of the optical image is 0.65 m. The spatial resolution of the DSM extracted from the GF-7 satellite is 1 m. To ensure the synergy of the training data, we resampled the 1m DSM data to the same 0.65 m as the optical image, and performed spatial resampling. Finally, the elevation value distribution of the DSM data was normalized to between 0 and 255. The training set contained 5040 sets of data, the validation set contained 990 sets of data and the test set contained 114 sets of data.

4.1.2. 3DCD dataset

The 3DCD dataset was proposed by Marsocci et al. (2023). According to Qin et al. (2016), this type of dataset can be categorized as an overhead dataset. It contains 472 pairs of images intercepted from optical orthophotographs obtained through two different aerial surveys and 472 pairs of DSMs generated by rasterization of point clouds obtained from two different LIDAR flights. This data only covers the urban centre of the city of Valladolid in Spain, including the surrounding commercial area. In addition, only changes in images of man-made structures, due to events such as the construction and demolition of buildings/roads/bridges, are included in this dataset.

4.2. Evaluation metrics

We used evaluation metrics that are commonly used in binary classification problems: F1-score, Precision (P), Recall (R) and overall accuracy (OA). The calculation formulas are as follows: (9) $$F_1–\mathrm{\text{score}}=\frac{2\times \mathrm{\text{Precision}}\times \mathrm{\text{Recall}}}{\mathrm{\text{Precision}}+\mathrm{\text{Recall}}}$$(9) (10) $$\mathrm{\text{Precision}}=\frac{\mathrm{\text{TP}}}{\mathrm{\text{TP}}+\mathrm{\text{FP}}}$$(10) (11) $$\mathrm{\text{Recall}}=\frac{\mathrm{\text{TP}}}{\mathrm{\text{TP}}+\mathrm{\text{FN}}}$$(11) (12) $$\mathrm{\text{OA}}=\frac{\mathrm{\text{TP}}+\mathrm{\text{TN}}}{\mathrm{\text{TP}}+\mathrm{\text{FP}}+\mathrm{\text{TN}}+\mathrm{\text{FN}}}$$(12) where TP and TN denote the numbers of correctly categorized changed and unchanged pixels, respectively, and FP and FN denote the numbers of incorrectly categorized changed and unchanged pixels.

4.3. Implementation details

The experiments were run on a workstation with a Windows 10 operating system, Intel(R) i7-9850 CPU and NVIDIA Quadro RTX5000 graphics card with 11 GB of memory. We used Pytorch 1.7.1 as the deep learning framework and programmed in Python 3.6.13. For the optimization and loss function, we employed the Adam optimizer and used the binary cross-entropy loss function to update the model parameters. The initial learning rate was set to $1\times {10}^{-5}.$ Additionally, we set the batch size to 2, with the program configured to stop when the F1-score of the validation dataset remained stable for 20 cycles.

We resized the starting input image to 256 × 256 pixels. To perform feature extraction, we utilized Resnet-34 and FCN. This allowed us to generate four scaled features: Image (64 × 64 × 64, 128 × 32 × 32, 256 × 16 × 16, 512 × 8 × 8) and DSM (64 × 64 × 64, 128 × 32 × 32, 256 × 16 × 16, 512 × 8 × 8).

4.4. Comparison methods

To validate the effectiveness of the proposed method, we selected several current popular CD networks for comparison: MapsNet, STANet, FC_EFNet, IFNet, SNUNet and BIT. The details are as follows:

1. MapsNet (Pan, Cui, et al. 2022): A multiattention module that provides good results and robustness in complex urban scenarios.

2. STANet (Chen and Shi 2020): This network embeds spatial-temporal attention to better accomplish CD tasks by capturing spatial-temporal dependencies at different scales.

3. FC_EFNet (Daudt et al. 2018): Directly adds before/after images to the channel dimension as inputs. This method is based on an early fusion convolutional network.

4. IFNet (Zhang et al. 2020): This network first extracts highly representative deep features of dual-time images using a full convolutional dual-stream architecture. Then, the extracted deep features are input into a deep supervised disparity discriminant network for change detection.

5. SNUNet (Fang et al. 2022): This network mitigates the loss of deep localization information in the neural network through compact information transfer between the encoder and decoder and between the decoder and decoder for better CD results.

6. BIT (Chen H et al. 2022): This network performs the CD Test better by representing the diachronic image as several tokens and modelling the over-the-air context using a transformer encoder.

4.5. Experimental results

The performance of MFFNet on two datasets was assessed both qualitatively and quantitatively. The areas of change were highlighted in white, while unchanged areas were highlighted in black. The evaluation metrics mentioned in Subsection 2.3 were used to compare MFFNet with six other methods. The quantitative evaluation results for the GF7-CD and 3DCD datasets are shown in Tables 1 and 2, respectively, with the best results highlighted in bold.

Table 1. Quantitative assessment of different methods on the GF7-CD dataset.

Classes	Method	OA (%)	Precision (%)	Recall (%)	F1-Score (%)
Image	MapsNet	96.71	89.88	90.37	90.12
	STANet	89.16	97.66	35.62	52.20
	FC_EFNet	92.29	70.89	90.29	79.42
	IFNet	95.12	92.04	77.34	84.05
	SNUNet	93.43	74.25	92.57	82.41
	BIT	95.21	81.51	92.01	86.48
Image + DSM	MFFNet	97.76	91.39	91.48	91.44

Table 2. Quantitative assessment of different methods on the 3DCD dataset.

Classes	Method	OA (%)	Precision (%)	Recall (%)	F1-Score (%)
Image	MapsNet	97.45	67.32	55.83	61.04
	STANet	96.05	45.10	47.77	46.40
	FC_EFNet	96.27	47.88	47.51	47.69
	IFNet	97.40	67.45	52.72	59.19
	SNUNet	96.84	57.69	44.52	50.25
	BIT	97.34	74.69	38.74	51.02
Image + DSM	MFFNet	98.59	86.05	72.50	78.70

4.5.1. GF7-CD dataset

Table 1 compares the accuracy evaluation results with those of other CD methods. MFFNet is superior to the other methods in almost all evaluation indexes. In terms of F1-Score, our method is 1.32–12.02% better than the other four methods. The MapsNet model achieves the second-highest F1-Score after MFFNet, which can be attributed to it having an effective design for detecting minor changes in complex scenes. STANet achieves the highest Precision of 97.66%, resulting from a higher number of omissions, and it yields a higher F1-Score than the other methods. FC-EFNet shows lower performance across the various evaluation metrics. Although the IFNet and SNUNet models slightly outperform our method in terms of Precision and Recall, their F1-Scores are significantly lower. The experimental results demonstrate that MFFNet effectively reduces model leakages and misdetections. The experimental results are visualized in Figure 9, where regions with significant improvement are highlighted with dashed boxes. MapsNet, FC_EFNet, IFNet, SNUNet and BIT produce a higher number of misdetections, as shown in the dashed box regions shown in Figure 8(a–d). The figure clearly illustrates the significant improvements of MFFNet compared to other CD methods in terms of reducing false detections and omissions, while also achieving more complete edges. The traditional method misclassifies playgrounds and flower beds as building changes occur, due to its exclusive focus on image features and neglect of crucial elevation features, leading to the generation of false detections.

Figure 8. Visualization of building change detection results on the GF7-CD dataset.

4.5.2. 3DCD dataset

Table 2 presents the quantitative evaluation results. MFFNet achieves an OA of 98.59%, Precision of 86.05%, Recall of 72.50% and F1-Score of 78.70%, thereby outperforming the other methods across all evaluation metrics. The visualization results are shown in Figure 9. In Figure 9(a), the middle part of the bridge is similar to the surface spectral information, so there are false detections by all methods that do not consider elevation information. In Figure 9(b,d), due to the change in the surface spectral information around the building, there is a false detection by the traditional method, while the edges of the building extracted by MFFNet (with consideration of elevation information) are complete. In Figure 9(c), spectral differences due to illumination and thus misdetection occur in all other methods that do not consider elevation information. In the same building shown in the dashed box of Figure 9(e), the problem of pseudo-variation due to differences in spectral information leads to false detections in all six remaining methods.

Figure 9. Visualization of building change detection results on the 3DCD dataset.

Overall, in terms of quantitative results, MFFNet shows the best performance on both datasets, with significant reductions in false detections and omissions. This is mainly due to the fact that the HRRSI provides rich shape, edge and texture details of buildings combined with the surface elevation information provided by DSM. The dual-path extraction network module proposed in this paper can extract image features and elevation features well, which are then effectively combined by the DFM. Then, the MFSM considers the relationships with the global context to better accomplish the task of BCD.

4.6. Dual-path feature-extraction network comparison experiment

This study introduces the DPEN for feature extraction of optical images and DSM. We designed specialized encoders to accommodate the distinct data structures. When dealing with input data from two different structures, the dual- path feature- extraction network is more efficient in accomplishing complex feature learning compared to the siamese structure feature extraction network (SSEN). To identify the most suitable DPEN for both optical image and DSM data structures, we performed comparative experiments on the GF7-CD dataset using various feature extraction networks.

4.6.1. Siamese structure feature extraction network

The proposed network consists of two identical branches, each dedicated to processing dual time-phase optical images and DSMs. Both branches utilize identical feature-extraction networks and share weights to ensure consistent feature extraction for both types of data.

4.6.2. Dual-path feature-extraction network

This network is comprised of two distinct extraction networks. One primarily utilizes ResNet, while the other employs the simpler FCN. To assess the impact of combining different forms of ResNet and FCN, we conducted comparative experiments by pairing ResNet-18 and ResNet-34 with FCN.

As indicated in Table 3, SSEN-18 achieves the highest R-value among the four methods, reaching 92.13%. However, the remaining evaluation metrics indicate lower performance. Conversely, DPEN-34 exhibits superior performance in the other three metrics, except for R-value. DPEN-34 utilizes ResNet-34 for optical image extraction and FCN for DSM extraction. As illustrated in Figure 10, DPEN-34 demonstrates clearer boundaries and minimal discontinuity within the changed region. When comparing the experimental accuracy and visualization results, DPEN-34 proves to be the most suitable method for this paper.

Figure 10. Visualization of the comparative experiments of dual-path extraction networks applied to the GF7-CD dataset.

Table 3. Results of the dual-path extraction network comparative experiments.

Encoder method	OA (%)	Precision (%)	Recall (%)	F1-Score (%)
SSEN-18	96.70	88.49	92.13	90.28
SSEN-34	96.87	89.71	91.70	90.70
DPEN-18	95.61	95.48	85.00	89.93
DPEN-34	97.76	91.39	91.48	91.44

The bold characters in the table represent the best performing results among different methods.

4.7. Ablation experiment

To assess the impacts of DFM, MDFM and MFSM on the overall performance of the proposed MFFNet, we conducted ablation experiments using the GF7-CD dataset. The Baseline configuration involved the removal of the three modules (DFM, MDFM and MFSM) from MFFNet, resulting in a significant decline across nearly all evaluation metrics compared to standard MFFNet. The quantitative results of these ablation experiments are presented in Table 4.

Table 4. Accuracy evaluation results of the ablation experiment.

Model	DFM	MDFM	MFSM	OA (%)	P (%)	R (%)	F1 (%)
Baseline	×	×	×	93.35	79.59	80.63	80.11
A	×	×	√	96.67	88.15	92.33	90.20
B	×	√	×	95.82	85.93	89.54	87.10
C	√	√	×	96.83	90.45	90.51	90.48
MFFNet	√	√	√	97.76	91.39	91.48	91.44

Note: √ indicates that the module was included, × indicates it was not included. The bold characters in the table represent the best performing results among different methods.

4.7.1. Effectiveness of MFSM

Model A is an extension of the Baseline model, incorporating the MFSM. As depicted in Table 4, Model A exhibits improvements in total precision, accuracy, recall and F1-Score of 3.32%, 8.56%, 11.7% and 10.09%, respectively, compared to the Baseline model. Furthermore, as illustrated in Figure 11, Model A shows significantly enhanced building outlines in comparison to the Baseline model. This improvement can be attributed to the MFSM module, which not only integrates multilevel features but also considers the pixel relationships within the image.

Figure 11. Comparison of ablation experiment results.

4.7.2. Effectiveness of MFM

Model B introduces the MDFM module to the Baseline model. As shown in Table 4, all quantitative metrics indicate improvements compared to the Baseline model. Additionally, as depicted in Figure 11, a noticeable reduction in misdetections and omissions can be observed when comparing Model B with the Baseline model. This improvement can be attributed to the MDFM module’s effective management of the semantic gap and reduction of information redundancy.

4.7.3. Effectiveness of DFM

Model C incorporates both the DFM and MFM modules into the Baseline model. As shown in Table 4, all quantitative indicators indicate improvement compared to Model B. Analysis of Figure 11 reveals that Model C exhibits enhanced sensitivity to small target changes and fewer false detections and omissions than Model B. This can be attributed to the DFM’s more effective integration of geometric and spectral building information.

As shown in Table 4, MFFNet is best in all quantitative indicators except for the recall rate, which is slightly lower than that of Model A. The four sets of ablation experiments show that three modules, DFM, MDFM and MFSM, provide significant improvements of the final CD effect, which proves their validity and reliability.

5. Discussion

In this section, we discuss the role of geometric information in the models presented in this paper. To better understand the significance of geometric information, we conducted comparative experiments on two datasets: 3DCD and GF7-CD. We utilized MFFNet as the baseline network and removed DSMs from the data sources to construct a comparison model D. The results are presented in Tables 5 and 6.

Table 5. Experimental results comparing Model D and MFFNet on 3DCD.

Classes	Method	OA (%)	Precision (%)	Recall (%)	F1-Score (%)
Image	D	96.74	58.36	31.40	40.83
Image + DSM	MFFNet	98.59	86.05	72.50	78.70

The bold characters in the table represent the best performing results among different methods.

Table 6. Experimental results comparing Model D and MFFNet on GF7-CD.

Classes	Method	OA (%)	Precision (%)	Recall (%)	F1-Score (%)
Image	D	94.94	85.33	83.96	84.63
Image + DSM	MFFNet	97.76	91.39	91.48	91.44

The bold characters in the table represent the best performing results among different methods.

The results of the quantitative experiments indicate that the removal of the DSM data source leads to decreases in all quantitative indicators. For instance, F1-Scores are 6.81% and 37.83% lower on the GF7-CD and 3DCD datasets, respectively. This further supports the significant role of geometric information in BCD. To provide a more intuitive demonstration of the impact of geometric information on the models, we conducted attention visualization on their resulting maps. In this visualization, red represents focused attention while blue indicates distracted attention. As shown in Figure 12(a), the colour of the bridge in Image2 is similar to that of the road in Image1. Consequently, model D, which does not consider geometric information, only focuses on the white area surrounding the bridge. As a result, a significant amount of leakage occurs. However, MFFNet mitigates this issue by incorporating elevation information—the change in bridge height between DSM1 and DSM2. This allows the model to identify the specific area where the height changes, leading to a reduction in leakage. As shown in Figure 12(b), there are noticeable changes in the houses, surrounding roads and vegetation between Image 1 and Image 2. However, model D is adversely affected by these irrelevant changes, as well as shadows, which impairs its ability to concentrate on the houses. In contrast, MFFNet effectively directs attention towards the houses by leveraging the information on changes in house elevation present in DSM1 and DSM2. As shown in Figure 12(c), the buildings in Image 2 are obscured by shadows, making their boundaries unclear. However, the buildings in DSM2 are not affected by shadows. Leveraging this insight, MFFNet allocates more attention to capturing the building outlines, leading to sharper and more distinct building outlines. As shown in Figure 12(d), both Image 1 and Image 2 contain buildings (with height differences) and flower beds (without height differences). The attention map of model D reveals that it continues to concentrate its attention on the centre and outlines of the flower beds, while the outlines of the buildings remain indistinct. In contrast, MFFNet, which incorporates geometric information, demonstrates superior attention towards building contours, consequently minimizing false detections (such as mistaking flower beds for buildings if they have similar shape characteristics).

Figure 12. Experimental results visualizing the attention of Model D and MFFNet.

The quantitative and qualitative results demonstrate that incorporating geometric information effectively enhances the accuracy of BCD. This approach reduces false detections and omissions caused by features with similar spectral and shape characteristics as buildings, and also mitigates the impacts of shadows and vegetation. Furthermore, it enables the extraction of more comprehensive building edges.

6. Conclusions

BCD based on RSI is an important means of obtaining information on ground changes. This study proposes MFFNet to solve the problems caused by shadows, vegetation and changes in objects with similar spectral and morphological characteristics as buildings. Such problems can easily lead to issues in BCD such as false detections and omissions, and incomplete patch edges. The MFFNet consists of a DPEN, a MDFM and a MFSM. Here, the DPEN allows better extraction of spectral and geometric features, the MDFM enables efficient combination of extracted features and the MFSM considers the relationship between global contexts. This can provide a more efficient combination of spectral and geometric information. The effectiveness of MFFNet in BCD was demonstrated experimentally.

To evaluate the performance of our proposed MFFNet, we conducted a series of experiments, including comparative experiments on two datasets and ablation experiments on important modules. The results show that MFFNet significantly reduces false detections and omissions compared to other current network models. Specifically, it achieves F1-Scores of 91.44% and 78.70% on the GF7-CD and 3DCD datasets. We believe that obtaining additional information about buildings will help to improve the BCD results.

The experiments have the following shortcomings: since MFFNet requires both optical images and DSM, and because popular public datasets usually contain only optical images, validation was only performed on two datasets. Therefore, the model’s paradigm ability remains to be verified. For this reason, we will continue the paradigmatic study of this model.

Authors’ contributions

Conceptualization, Z.H.G. and J.P.P.; methodology, Z.H.G. and J.P.P. and P.X; validation, Z.H.G.; formal analysis P.X. and C.Q and X.X.W and Y.H.Y and Y.W and L.Z; resources, H.J.Z. and H.Z.R.; writing—original draft preparation, Z.H.G.; writing—review and editing, J.P.P. and Z.H.G.; All authors have read and agreed to the published version of the manuscript.

Acknowledgement

We thank the authors of the 3DCD dataset.

Disclosure statement

The authors declare no conflict of interest. And we sincerely appreciate that academic editors and reviewers give their help comments and suggestions.

Additional information

Funding

This work is supported by the Key R&D Program of Ningxia Autonomous Region: Ecological environment monitoring and platform development of ecological barrier protection system for Helan Mountain(2022CMG02014)and Chongqing Graduate Joint Training Base Construction Project (JDLHPYJD2019004) and The Project Supported by the Open Fund of Key Laboratory of Monitoring, Evaluation and Early Warning of Territorial Spatial Planning Implementation, Ministry of Natural Resources(No: LMEE-KF2023001).