Integrating bi-temporal VHR optical and long-term SAR images for built-up area change detection

ABSTRACT

With the rapid expansion of urbanization, it is imperative to monitor built-up areas changes to promote the sustainable development of cities, aligning with the goals of Sustainable Development Goal 11(SDG 11). Remote sensing big data is valuable for automatically mapping these changes. Bi-temporal very high-resolution (Bi-VHR) optical images have been widely utilized for fine-grained change detection (CD). However, the significant spatiotemporal inconsistency due to imaging conditions and seasonal variations poses challenges for VHR optical CD. Unlike optical images, synthetic aperture radar (SAR) images are unaffected by atmospheric interference and provide robust spatiotemporal features as a supplement. Previous CD algorithms with SAR overlooked the exploration of long-term features of time series. In this study, we propose a novel CD framework combining long-term SAR with Bi-VHR images. It incorporates a spatial-frequency learning module to enhance SAR temporal features and a multisource feature fusion module to adaptively fuse the heterogeneous features. The experiments are conducted on OS-BCD dataset, which is the first dataset specifically designed for this task. The results demonstrate that our proposal outperforms advanced CD methods with F1 score, IoU and OA of 64.99%, 48.13%, and 95.11%, validating the efficacy of our proposal in accurately detecting changes in built-up areas.

KEYWORDS:

• Built-up area
• change detection
• deep learning
• long-term SAR
• very high-resolution image

1. Introduction

With the accelerating process of global urbanization, the expansion of built-up areas poses a threat to arable land quality (Kong 2014; Li, Zhang, and Wang 2023; Zeren Cetin et al. 2023) and leads to environmental degradation (Degerli and Çetin 2022a; Liu, Zhang, and Zhou 2018). The dynamic changes of built-up areas and the cascading impacts present significant challenges to the achievement of the United Nations Sustainable Development Goals (SDGs) especially SDG11 Sustainable Cities and Communities (Cetin 2016; Han et al. 2022). Timely and accurate identification of changes in built-up areas is crucial for formulating urban policies and promoting sustainable urban development(Cetin 2019; Cetin et al. 2021). Remote sensing data, with its large-scale and multi-temporal observation capabilities, enables precise and convenient monitoring of built-up area changes, thereby contributing to the achievement of SDG11 (Derakhshan, Cutter, and Wang 2020). In comparison to the time-consuming manual vectorization method, the advancement of change detection (CD) technology (Lv et al. 2023) with remote sensing (Degerli and Çetin 2022b) provides an opportunity for automated mapping of change areas (Cetin et al. 2023; Huang et al. 2023; Zeren Cetin et al. 2023).

In recent years, many studies focused on conducting CD tasks based on bi-temporal very high-resolution (Bi-VHR) images. These images, with spatial resolutions in the range of meters or even sub-meters, provide detailed spatial information about ground objects (Zhao et al. 2023). Deep learning models, known for their robust ability to extract high-level semantic features (Zhang, Zhang, and Du 2016), have been widely adopted for Bi-VHR change detection (Chen et al. 2020; Zhu et al. 2021). Convolutional neural networks (CNNs) combined with Siamese structures have emerged as popular approaches for learning hierarchical representations from Bi-VHR data, with a focus on capturing geometric change characteristics through shared parameters in encoders (Chen et al. 2022; Fang, Pan, and Kou 2019; Li et al. 2023; Song, Hua, and Li 2023). Recently, attention-based models have been introduced in change detection (CD) tasks to explore the relevance of object representations (Chen and Shi 2020; Liu and Shi 2021). The continual improvement of network structures, such as expanding receptive fields (Xia, Xu, and Pu 2022) and integrating multi-layer features (Yuan et al. 2022; Zhuo et al. 2021), has further enhanced the accuracy of fine-grained CD based on VHR images.

However, the effectiveness of CD is still limited by the substantial spatiotemporal inconsistency present in VHR images. Firstly, VHR images exhibit significant spatial inconsistency, resulting in diverse data domains across different regions or time periods (Teng et al. 2023). This spatial variability poses challenges to the generalization performance of CD models (Tang, Wang, and Zhang 2022). For instance in Figure 1, factors like the captured angle (Matasci et al. 2015), sensor type and solar irradiance (Wang et al. 2017) can influence the clarity and visibility of VHR images, leading to considerable variations in the data domain. Secondly, the temporal inconsistency of the background introduces semantic ambiguity (Lv et al. 2023) due to seasonal changes and limited shooting time (Li et al. 2022). For example, the farmland depicted in Figure 1 undergoes visible changes as the seasons progress (e.g. crop harvesting). Such changes in visual appearance, while maintaining unchanged geographical semantics, increase the likelihood of model misdetection in CD networks. Therefore, it is difficult to break through the uncertainty of data sources mentioned above solely by optimizing the network.

Figure 1. The spatiotemporal inconsistency inherent in VHR images poses a challenge to the robustness improvement of CD models.

To address the issue of Bi-VHR data inconsistency in CD, we propose the integration of Synthetic Aperture Radar (SAR) with optical images. SAR images provide long-term scattering features that complement the visual features captured by the Bi-VHR images. The backscatter feature of SAR images characterizes the structural information of ground objects (Saha, Bovolo, and Bruzzone 2021), and has been successfully applied in tasks such as built-up mapping (Liang et al. 2023). Unlike optical images, SAR data is not affected by atmospheric conditions such as clouds and rain, making it suitable for long-term periodic modeling (Hu and Ban 2014; Pirrone, Bovolo, and Bruzzone 2020). This allows us to account for heterogeneity caused by image quality and seasonal changes. Previous studies have demonstrated the effectiveness of using bi-temporal SAR data for CD, but only leveraging its spatial features (Fang, Pan, and Kou 2019; Geng et al. 2019; Jia et al. 2015). They ignore the potential of SAR images in capturing phenological information, which has not been extensively explored.

In order to couple VHR and long-term SAR data into built-up CD tasks, two key problems still need to be addressed. Firstly, it is necessary to extract global spatiotemporal information from the SAR time series. CNN typically extract features through sliding convolution operations in the spatial domain (Gao et al. 2018), which makes it difficult to capture the global features of both time and space. In contrast, frequency domain features can highlight the key features of long time series by embedding global features into each frequency component (Shi et al. 2020). Therefore, we propose the frequency learning to effectively extract spatiotemporal semantic information from the SAR time series. Secondly, the effective integration of VHR and SAR images presents certain challenges. These two types of images differ in terms of resolution, imaging principles, and feature semantics, making their integration a formidable task. In this article, we propose a novel approach that leverages a multi-level attention mechanism to fuse discriminative features extracted from VHR and SAR images.

The main contributions of this article are given as follows.

1. To address the issue of spatiotemporal inconsistency in Bi-VHR optical CD task, we propose a novel deep learning network combining VHR optical and long-term SAR data specially for CD (HOLS-CDnet).

2. In HOLS-CDnet, a spatiotemporal-frequency learning module (SFLM) and an attention-based multisource feature fusion module (MFFM) are employed to enhance the long-term feature representation capability and integrate heterogeneous discriminative features adaptively.

3. We create and publicly release a new CD dataset, which firstly comprises both Bi-VHR and long-term SAR images for CD, named Optical-SAR Built-up Change Detection Dataset (OS-BCD).

The rest of this article is organized as follows: Section 2 introduces some existing techniques in bi-temporal VHR change detection and multi-source remote sensing data change detection. Section 3 describes the basic information of the built-up change detection data set used in our article, which is built by ourselves. Section 4 introduces the structures and evaluation criteria of our proposed network. Section 5 presents the results of experiments we conducted with our network and compares them with some classical methods. Section 6 discusses the effectiveness of SAR + VHR, and Section 7 gives our conclusions.

2. Related work

2.1. Bi-VHR change detection

In recent years, deep learning has been widely used to extract VHR fine-grained change information. Siamese neural networks are popular for their ability to extract features through differences or splicing flexibly (Wang, Tan, et al. 2020). In (Liu et al. 2018), an unsupervised deep convolutional coupling network is proposed for change detection of VHR heterogeneous images. Two Siamese extensions of fully convolutional networks were first proposed by Daudt, Caye, and Boulch (2018), which achieved the best results on two open change detection datasets. Zhan et al. (2017) introduced a deep Siamese convolutional network for aerial image change detection and the Siamese network learned to extract features directly from the image pairs. To further improve the performance of Siamese neural networks, Chen et al. (2019) incorporated spatial and channel attention mechanisms and denser connections to fuse features. DASNet (Chen, Yuan, et al. 2021b), by using the dual-attention mechanism to capture long-range dependencies, could overcome the lack of resistance to pseudo-changes. A densely connected Siamese network for change detection, namely SNUNet-CD (Fang et al. 2022), reduced the loss of localization information in neural network deep layers by using compact information communication between encoder and decoder. Du et al. (2019) designed DSFANet, where two symmetric deep networks were used to project the input data of dual-temporal imagery. And a Multi-scale Context Aggregation module (MSCA) in MSCANet was intended to adaptively gather and aggregate contextual data from various crowd sizes (Wang, Zhao, et al. 2020b). In a nutshell, these studies focus on highlighting foreground changes and suppressing background information by enhancing model representation capabilities.

2.2. Multisource remote sensing data change detection

Multisource remote sensing data change detection combines multiple types of remote sensing data, such as multispectral images, hyperspectral images, light detection and ranging (LiDAR) data (Lv et al. 2022) and radar images, to more accurately detect object changes. The combination of multisource data provides richer information for ground object changes. Some studies have integrated SAR and optical images in CD. DTCDN (Li, Du, et al. 2021) is a deep translation-based change detection network, which combines optical and SAR images and uses deep context features to separate unchanged pixels from changed pixels. A Siamese network (Saha et al. 2022) is proposed to cope with the absence of post-temporal optical images by processing dual-temporal SAR input and optical pre-temporal input respectively. SAR images are also used to alleviate the problem of optical image cloud coverage (Li et al. 2022). In one of the works, optical images and SAR images are used to segment the geographical patches and automatically generate samples (Zhou et al. 2021). These studies have preliminarily demonstrated the feasibility of using multisource data for change detection. However, there are few models fusing long-term SAR and VHR optical data to obtain fine-grained CD.

3. OS-BCD: the first built-up CD dataset contained both Bi-VHR optical and long-term SAR images

3.1. Study area

The study area is Guigang City, Guangxi Zhuang Autonomous Region, China, as shown in Figure 2. It is located between 109°11'56” and 110°39'36"E, 22°39'16” and 24°2'42"N, with a total area of about 1.06 × 10⁴ km² and a built-up area of more than 80 km². Due to the latitude and terrain factors, Guangxi is prone to cloudy and rainy weather, making it difficult for optical images to cover all its areas at once (Yuan, Xu, et al. 2022). Multiple sources and different capturing times result in significant differences in the VHR optical image data domain of the region.

Figure 2. The study area for collecting images and labeling of OS-BCD.

3.2. OS-BCD dataset

We constructed a dataset including 2188 pairs of Bi-VHR images and long-term SAR data with built-up area changes. The basic information of the dataset is shown in Table 1. Long-term SAR data consist of 48 bands, which correspond to two imaging phases. Each phase includes data captured over a period of 12 months, with two polarization modes recorded each month.

Table 1. Basic information of OS-BCD dataset.

Dataset Information	Attribute	Value
Overview	Image Pairs	2188 pairs
	Time Period of the Image	2018, 2022
	Training: Validation: Testing Set Ratio	7:1:2
Bi-VHR images	Source	Google Earth
	Pixel Resolution	2m
	Image Size	3 × 256 × 256
Long-term SAR data	Source	Sentinel-1 GRD products
	Pixel Resolution	10m
	Image Size	48 × 64 × 64
	Time Period	12 months per year
	Polarization Modes	VV and VH

The samples are shown in Figure 3, we can see that the built-up area changes are mainly divided into three categories (a) from vegetation to built-up area (b) from vegetation to bare area (c) from bare area to built-up area. It is obvious that the data domain of Bi-VHR is inconsistent, and some of the built-up area changes are subtle and susceptible to background changes. For example, the first row of Figure 3 is a scenario of arable land harvesting that is easily identified to be a transition from vegetation to bare area. OS-BCD dataset and the code of HOLS-CDnet will be made publicly available.

Figure 3. Dataset sample examples, showing Bi-VHR and bitemporal VH of long-term SAR data and labels. (a) from vegetation to built-up area (b) from vegetation to bare area (c) from bare area to built-up area.

4. Methodology

4.1. Overview of HOLS-CDnet

This section describes the novel network (HOLS-CDnet) that integrates Bi-VHR images and long-term SAR data for built-up area change detection. As is illustrated in Figure 4, the overall framework consists of three modules:

1. Bi-VHR Siamese CD module (BSCM): we construct a Siamese convolutional neural network to extract multilevel features of Bi-VHR images. By calculating the difference between low-level features and concatenating high-level features, the module gets both fine-grained texture $V_s$ and semantic representatives $V_d$.

Figure 4. The framework of HOLS-CDnet (a) Bi-VHR Siamese CD module (BSCM) (b) SAR spatiotemporal-frequency learning module (SFLM) (c) multisource feature fusion module (MFFM).

(2)

SAR spatiotemporal-frequency learning module (SFLM): to extract long-term features of long-term SAR data, a spatiotemporal-frequency learning block (SFLB) is introduced to combine spatial and frequency domains. Through global frequency domain operations and local spatial convolution operations, SFLB enhances the ability to distinguish spatiotemporal features of long-term SAR data. We use SFLBs to extract the low-level difference features $S_s$ and high-level difference features $S_d$.

(3)

Multisource feature fusion module (MFFM): In order to integrate multi-source features from optical and SAR sensors, we utilize three attention modules to fuse multisource difference features. First, the low-high level attention is used to combine $V_s$ and $V_d$ to obtain both texture and semantic representatives of VHR images. Second, multisource fusion attention is utilized respectively on shallow and deep features of multisource data to fuse heterogeneous features. Then, we build a change enhancement module with attention mechanism to adaptively explore effective information from feature stacks. The change enhancement attention module adaptively selects appropriate changing or invariant features from different sensors based on channel self-attention, so as to achieve robust fusion of heterogeneous features. Finally, through a decoder, we get and fuse the ‘$\mathrm{VHR}\mathrm{output}$’ and ‘$\mathrm{Fusion}\mathrm{output}$’ to obtain the finally predicted change map.

These three modules will be introduced in detail in the following sections.

4.2. Bi-VHR siamese CD module (BSCM)

Since VHR images have delicate spatial information, it is significant to use an efficient Siamese convolutional network to extract bi-temporal features. Considering the outstanding performance of Resnet (He et al. 2016), we use Resnet-50 as the backbone of the Siamese network. Resnet-50 can be divided into one convolutional block and four residual blocks.

First, the Bi-VHR images $I_{T1}$ and $I_{T2}$ are fed into ResNet-50 to get two groups of features $\{f_T^1,\dots ,f_T^4\}$, $T=\mathrm{T}1$, T2. In order to enhance changes between low-level features, including fine-grained spatial information, we use the absolute difference to extract the shallow change information. Specifically, we grasp the shallow change feature $V_s$ by applying the difference function $|f_{T1}^2-f_{T2}^2|$. In order to explore high-level features, usually containing semantic representatives, we obtain the deep change feature $V_d$ by performing the concatenation function $[f_{T1}^4{f}_{T2}^4]$.

4.3. SAR spatiotemporal-frequency learning module (SFLM)

CNN which learns in spatial domain is widely used in local information extraction, but is not efficient in capturing global information. However, when transitioning an image from spatial domain to frequency domain by Fourier analysis, global information is embedded into each frequency component. Learning in frequency domain means learning a self-adaptive filter to preserve useful information globally, which is cost-efficient for grasping global information (Jia and Yao 2023; Zhang et al. 2022). Considering the advantage of extracting local and global information simultaneously, some studies employed spatial-frequency joint learning into deep learning (Wu et al. 2019; Yang et al. 2023; Zhou et al. 2022). Since long-term SAR data have enormous spatiotemporal information, it’s critical to effectively extract the periodic and mutative characteristics. In the frequency domain, the periodic feature of the background is considered as a low-frequency signal, while the changing feature is considered as a high-frequency signal. Therefore, temporal global features can be extracted from overall time series by frequency domain feature calculation. Meanwhile, local features in spatial domain can effectively locate changing positions. Therefore, we introduce the SFLB, which is a classical block to joint frequency-spatial learning (Jia and Yao 2023) to extract the global and local information efficiently.

First, long-term SAR is first divided into two temporal phases. Then the data from the two temporal phases are processed using SFLB. To enhance change features between time 1 and time 2, the absolute difference between the results of SFLB is computed to get $S_s$. To combine semantic representatives of deep features, a concatenation is also performed on the results, after which the SFLB is utilized again to obtain $S_d$.

The process of SFLB is described in detail in Figure 5. The SFLB is divided into two main parts. One part learns parameters in the spatial domain without Fourier transformation and the data are directly fed into two layers of convolution to get change information in local spatial domain. The other part learns filter parameters in the frequency domain. Initially, the SAR data are transformed from the spatial domain to the frequency domain using the Fast Fourier Transformation (FFT). Then real and imaginary parts are concatenated along the channel dimension. Subsequently, concatenated data are forwarded into two layers of convolution. After that, the data are transformed back to the spatial domain using the Inverse Fast Fourier Transformation (IFFT). Finally, the outputs from the spatial domain and frequency domain are concatenated, and convolution is utilized to compress and restore the results to their original dimensions.

Figure 5. Detailed description of spatiotemporal-frequency learning block (SFLB) in SFLM. FFT and IFFT mean Fast Fourier Transformation and Inverse Fast Fourier Transformation.

4.4. Multisource feature fusion module (MFFM)

To obtain discriminated change information from multisource data, MFFM including low-high level attention, multisource fusion attention and change enhancement attention is proposed. First, to adaptively integrate shallow and deep change information of Bi-VHR, low-high level attention is used. Then, aimed at combining multisource features obtained from Bi-VHR and long-term SAR data, we use multisource fusion attention and change enhancement attention to adaptively integrate them. The low-high level attention and change enhancement attention are composed of channel attention (CA) and spatial attention (SA) (Woo et al. 2018), while the multisource fusion attention mainly consists of CA. CA and SA can model the correlations between multilevel and multisource features in space and location, by which we can get a more discriminative fusion feature.

To be specific, in low-high level attention, first, we concatenate the $V_s$ and the $V_d$ and forward them into CA and SA to get the channel-spatial-refined difference feature $F_V^{\mathrm{\prime \prime }}$. These processes can be described as follows: (1) $F_V=[V_s,V_d]$(1) (2) $F_V^{\mathrm{\prime }}=\mathrm{CA}(F_V)\otimes F_V$(2) (3) $F_V^{\mathrm{\prime \prime }}=\mathrm{SA}(F_V^{\mathrm{\prime }})\otimes F_V^{\mathrm{\prime }}$(3)

In multisource fusion attention, first, we concatenate $V_s$ and $S_s$, and feed them them to CA to get the channel-refined shallow feature $F_s^{\mathrm{\prime }}$. Then we perform the same on $V_d$ and $S_d$ to obtain channel-refined deep feature $F_d^{\mathrm{\prime }}$. These processes can be expressed as follows: (4) $F_s=[V_s,S_s]$(4) (5) $F_s^{\mathrm{\prime }}=\mathrm{CA}(F_s)\otimes F_s$(5) (6) $F_d=[V_d,S_d]$(6)

(7) $F_d^{\mathrm{\prime }}=\mathrm{CA}(F_d)\otimes F_d$(7) In change enhancement attention, we concatenate $F_s^{\mathrm{\prime }}$ and $F_d^{\mathrm{\prime }}$, and perform both CA and SA on them to get the channel-spatial-refined fusion feature $F_{\mathrm{fuse}}^{\mathrm{\prime \prime }}$, which can be described as follows: (8) $F_{\mathrm{fuse}}=[F_s^{\mathrm{\prime }},F_d^{\mathrm{\prime }}]$(8) (9) $F_{\mathrm{fuse}}^{\mathrm{\prime }}=\mathrm{CA}(F_{\mathrm{fuse}})\otimes F_{\mathrm{fuse}}$(9) (10) $F_{\mathrm{fuse}}^{\mathrm{\prime \prime }}=\mathrm{SA}(F_{\mathrm{fuse}}^{\mathrm{\prime }})\otimes F_{\mathrm{fuse}}^{\mathrm{\prime }}$(10)

The mechanisms of CA and SA are shown in Figure 6. The $\mathrm{CA}$ is calculated as the following formula: (11) $\mathrm{CA}=\sigma (\mathrm{ML}{\mathrm{P}}^r(\mathrm{AvgPool}(F))+\mathrm{ML}{\mathrm{P}}^r(\mathrm{MaxPool}(F)))$(11) where $F\in {\mathbb{R}}^{\mathrm{C}\times \mathrm{H}\times \mathrm{W}}$ denotes the input feature. $\mathrm{AvgPool}(\cdot )$ and $\mathrm{MaxPool}(\cdot )$ represent the average pooling layer and max pooling layer respectively. Through the two layers, we get two vectors of size $\mathrm{C}\times 1\times 1$. $\mathrm{ML}{\mathrm{P}}^{\mathrm{r}}(\cdot )$ is a weight sharing multi-layer perceptron with a hidden layer of channel reduction ratio $r$, after which the two new vectors are added and then activated by a sigmoid function $\sigma$ to get $\mathrm{CA}$. Similar to CA, SA can be described as the following formula: (12) $\mathrm{SA}=\sigma (f^{(k\times k)}(\mathrm{AvgPool}(F)\mathrm{MaxPool}(F)))$(12) where $f^{k\times k}$ denotes a convolution with a kernel size of $k$, and the average pooling layer and max pooling layer are utilized on channel dimension to obtain two vectors of size $1\times \mathrm{H}\times \mathrm{W}.$

Figure 6. Detailed description of channel attention and spatial attention in MFFM.

As for the decoder, we introduce a commonly used atrous spatial pyramid pooling (ASPP) in DeepLabV3 (Chen et al. 2017), which resamples features at different scales for identifying change areas of arbitrary scales accurately and efficiently. An improved ASPP includes one $1\times 1$ convolution and three $3\times 3$ convolutions with different atrous rates, as well as an image pooling block. Thus, we capture multilevel change features by using ASPP. Then we fuse the five outputs of improved ASPP by concatenation and feed them to a $1\times 1$ convolution and a bilinear upsampling to restore the size of label. Finally, we obtain the $\mathrm{Outpu}{t}_V$ and $\mathrm{Outpu}{t}_{\mathrm{fusion}}$.

Finally, in order to take advantage of VHR and multisource fusion simultaneously, a hyper-parameter $w$ is introduced to combine the $\mathrm{Outpu}{t}_V$ and $\mathrm{Outpu}{t}_{\mathrm{fusion}}$. We get the final change map $\mathrm{change}\mathrm{ma}{p}_{\mathrm{final}}$, which can be expressed as the following formula: (13) $\mathrm{change}\mathrm{ma}{p}_{\mathrm{final}}=w\times \mathrm{Outpu}{t}_V+(1-w)\times \mathrm{Outpu}{t}_{\mathrm{fusion}}$(13)

4.5. Loss function

Two loss functions are used as the objective function to optimize the performance of HOLS-CDnet (Li, He, et al. 2022). One is the Binary Cross Entropy Loss (BCELoss), which is usually utilized in measuring the error between the target and the input probabilities in binary classification. The other is the dice loss, which proved to be useful in overcoming the class imbalance in CD. They can be described as follows: (14) $\mathrm{BCELoss}=-(\mathrm{ylog}\hat{y}+(1-y)\log (1-\hat{y}))$(14) (15) $\mathrm{DiceLoss}=1-{\displaystyle \frac{1}{m}\sum _{j=1}^m{\displaystyle \frac{2{\sum }_{i=1}^N{\hat{y}}_{i,j}{y}_{i,j}}{{\sum }_{i=1}^N{\hat{y}}_{i,j}+{\sum }_{i=1}^N{y}_{i,j}}}}$(15) where $\hat{\mathrm{y}}$ and y respectively denote the predicted change map and the target, whose number are either 1 or 0. Finally, the objective function is as follows: (16) $\mathrm{Loss}=\lambda \mathrm{BCELoss}+(1-\lambda )\mathrm{DiceLoss}$(16) In the context of our paper, the symbol $\lambda$ serves as a balancing factor, regulating the proportion of weight assigned to BCELoss and DiceLoss within the overall loss function. The permissible range for the variable lies between 0 and 1.

5. Experimental result

5.1. Implementation details

We conducted all the experiments on the OS-BCD which was divided into training/validation/test sets by the ratio of 7:1:2 randomly, including 1531/219/438 pairs of Bi-VHR and long-term SAR images. The whole experiments were realized under the Pytorch framework, and used GeForce RTX 2080 GPU to train the proposed network. Besides, we adopted Adam to optimize the model, and the learning rate was initially set to 0.0005 and then decreased by Cosine annealing. When it came to the hyperparameters in attention mechanism, we set the reduction ratio r in CA and kernel size k in SA to 8 and 7 respectively, which is a useful setting proved in (Shi et al. 2022). We trained every network for 100 epochs and validated trained models on the validation set, on which the best model was chosen to be utilized on the test set.

5.2. Evaluation metrics

To evaluate the effectiveness of different methods on the test set, we chose some metrics including Precision, Recall, F1, IoU and OA. The metrics are described as follows:

(17) $\mathrm{Precision}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}}$(17) (18) $\mathrm{Recall}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}}$(18)

(19) $\mathrm{F}1={\displaystyle \frac{2\times \mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}$(19) (20) $\mathrm{IoU}={\displaystyle \frac{\mathrm{TP}}{\mathrm{FN}+\mathrm{TP}+\mathrm{FP}}}$(20) (21) $\mathrm{OA}={\displaystyle \frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}}$(21) where TP, FP, TN, and FN denote the numbers of true positive, false positive, true negative, and false negative pixels, respectively.

5.3. Comparative experiments

5.3.1. Comparative CD methods

To show the superiority of the proposed method, several state-of-the-art CD methods were selected to compare with. Given that those methods were designed for VHR images, we only used Bi-VHR as the input, and those methods are listed as follows:

1. MTCDN (Du et al. 2024): it integrates a CD network into a generative adversarial network(GAN) for optical and SAR images. It employs two generators and discriminators for each image domain, utilizing multitask learning to combine image identification and change detection. An end-to-end framework is also used in the prediction and training stage to reduce cost.

2. FC-Siam-diff (Daudt, Caye, and Boulch 2018): it uses a Siamese encoder to extract features from Bi-VHR, and utilizes an absolute difference to combine two temporal features, which are then connected to the decoder.

3. ISNet (Cheng, Wang, and Han 2022): it employs margin maximization to delineate the distinction between changed and unchanged semantics. It also implements a targeted arrangement of attention mechanisms to guide the utilization of channel attention (CA) and spatial attention (SA).

4. BIT-CD (Chen, Qi, and Shi 2021): it introduces a bitemporal image transformer (BIT) designed to proficiently capture contextual information within the spatial–temporal domain.

5. BiDateNet (Papadomanolaki et al. 2019): it is a fully convolutional network based on U-Net. It introduces Long Short-Term Memory (LSTM) convolutional blocks into the skip connection.

6. DASNet (Chen, Yuan, et al. 2021b): it employs a dual attention mechanism to capture long-range dependencies, and a weighted double-margin contrastive loss is proposed to address the problem of imbalanced samples.

7. SNUNet-CD (Fang et al. 2022): it compacts information transmission between encoder and decoder, and between decoder and decoder. In addition, an ensemble Channel Attention Module (ECAM) is proposed for deep supervision.

8. MSCANet (Liu et al. 2022): it utilizes a CNN-based feature extractor to capture hierarchical features, and designs a transformer-based MSCA to encode and aggregate context information. A deep supervision strategy is also used in the network.

5.3.2. Comparative experimental result

To display the superiority of HOLS-CDnet, it was compared with some representative CD methods on the test set. As shown in Table 2, HOLS-CDnet achieves the highest F1/IoU/OA of 64.99/48.13/95.11%, which is 2.79/3.00/0.17% higher than the second one, proving the effectiveness of integrating long-term SAR data with Bi-VHR images in built-up CD. Although the precision rate of HOLS-CDnet is comparatively lower than the recall, it is acceptable to recall more changed pixels at the expense of precision. MSCANet combining Transformer with CNN is the second-ranked method, and BiDateNet utilizing CNN and LSTM ranks third. Transformer and LSTM both capture long-range contextual information, which can strengthen feature representations better, improving the detection performance. ISNet has a high recall rate and a low precision rate, showing that margin maximization may capture more false alarms. The mediocre performance of BIT-CD indicates that it is not effective to enhance features extracted by Resnet using BIT. In the baseline Siamese methods, FC-Siam-diff outperforms SNUNet-CD on F1 and IoU. The reason may be that dense skip connections grasp excessive change information, resulting in a high recall rate and a low precision. DASNet, the only method using distance map to identify changes doesn’t exhibit a satisfactory result, demonstrating that distance map is hard to determine the boundary of change accurately. MTCDN, the only method that utilizes both optical and SAR data, is the worst one. The reason may be that it is hard for GAN in MTCDN to generate and fuse cross-resolution optical and SAR features.

Table 2. The result of different CD methods on the test set.

Method	Pre.	Rec.	F1	IoU	OA
MTCDN	45.41%	49.51%	33.01%	19.78%	92.95%
FC-Siam-diff	47.17%	63.65%	54.18%	37.16%	94.67%
ISNet	42.52%	78.34%	55.12%	38.04%	91.45%
BIT-CD	47.65%	65.10%	55.02%	37.95%	92.87%
SNUNet-CD	43.69%	65.83%	52.53%	35.62%	92.04%
MSCANet	62.14%	62.25%	62.19%	45.13%	94.94%
BiDateNet	59.64%	63.76%	61.63%	44.54%	94.69%
DASNet	49.51%	58.26%	53.53%	36.54%	93.24%
HOLS-CDnet	62.34%	67.86%	64.99%	48.13%	95.11%

The change detection result on test set is visualized in Figure 7. Row 1, and 5 contain some built-up area changes, with complex background which can be easily identified as pseudo changes. In these cases, HOLS-CDnet has the benefits of edge continuity and regional integrity, while the boundaries provided by (d)-(h) methods are more broken. Change areas detected by the (i)-(k) methods are not delicate enough, and there are more missed or false detections. Row 2–4 and the last row are simple changes with more obvious built-up area changes. The detection of HOLS-CDnet in these scenarios has robust background suppression, but there are obvious omissions or false alarms in other methods. In general, HOLS-CDnet has a good detection effect in simple change scenarios, less missed and false detection in complex change scenarios.

Figure 7. Visualization of change detection results of each method on the test set (a) image1 (b) image2 (c) label (d) MTCDN (e) FC-Siam-diff (f) ISNet (g) BIT (h) SNUNet-CD (i) MSCANet (j) BiDateNet (k) DASNet (l) HOLS-CDnet.

5.4. Ablation experiments

5.4.1 The effect of SAR, MFFM and SFLB

To verify the effectiveness of the introduction of SAR data, MFFM and SFLB, we design some ablation experiments. In Table 3, the ‘VHR’ denotes the network that only contains BSCM without MFFM and SFLM; The ‘SAR’ represents a simple CNN module instead of SFLB to extract long-term SAR features only in spatial domain; The ‘MFFM’ and ‘SFLM’ are modules described in section 4.4 and 4.3 respectively. The network that uses all the four modules is the proposed method, HOLS-CDnet.

Table 3. Ablation experiments on four modules.

VHR	SAR	MFFM	SFLM	Rec.	Pre	F1	IoU
✓✓				72.27%	51.50%	60.15%	43.01%
✓	✓			66.89%	59.19%	62.81%	45.78%
✓	✓	✓		64.42%	63.25%	63.83%	46.88%
✓	✓	✓	✓	67.86%	62.34%	64.99%	48.13%

The network that incorporates all the modules demonstrates superior performance, as evidenced by the results presented in Table 3. Compared with using ‘VHR’ only, the F1 and IoU of HOLS-CDnet are improved by 4.84% and 5.12%, verifying the validity of the whole modules. Though introducing ‘SAR’ directly, the recall rate and precision rate are balanced better. Utilizing ‘MFFM’ to combine ‘VHR’ and ‘SAR’ can further improve the performance. Introducing the ‘SFLM’ in the three modules above obtains the optimal result, indicating that integrating long-term SAR data with VHR images calls for special feature extractor and elaborate fusion module. The combination of MFFM and SFLM is more effective in improving the CD result of ‘VHR’ than simply introducing the long-term SAR data.

As depicted in Figure 8, the second and last rows illustrate that the utilization of all modules leads to improved regional integrity performance. This observation indicates that the incorporation of MFFM and SFLM modules enhances the recognition capabilities of the baseline network. The ‘VHR’ tends to detect more pseudo changes, and ‘VHR + SAR’ is inclined to omit some marginal changes, while the HOLS-CDnet captures built-up area changes more accurately. This result demonstrates that MFFM and SFLM can combine the advantages of Bi-VHR and long-term SAR data, obtaining more discriminant feature representations.

Figure 8. Visualization of ablation experiments’ result (a) image1 (b) image2 (c) label (d) 'VHR’ (e) 'VHR + SAR’ (f) 'VHR + SAR'+'MFFM’ (g) 'VHR + SAR'+'MFFM'+'SFLM’.

5.4.2 The effect of attention and ASPP

To verify the effectiveness of the introduction of Attention and ASPP in MFFM, we conducted some ablation experiments. Table 4 shows the results of the ablative experiments. It is evident that the combined approach integrating the ASPP and Attention modules yields the best performance. The F1 value reaches 64.99% and the IOU is 48.13%. In comparison to using only the ASPP module, both metrics demonstrate an improvement of 8%. The network without the inclusion of the Attention Module exhibits the poorest results, underscoring the significance of the Attention Module in enhancing network performance.

Table 4. Ablation experiments on Attention and ASPP.

Attention	ASPP	Rec.	Pre	F1	IoU
✓		57.87%	57.88%	57.88%	40.72%
	✓	64.82%	49.93%	56.41%	39.28%
✓	✓	67.86%	62.34%	64.99%	48.13%

Figure 9 illustrates the specific effects of the ablation experiments. From the second and fourth rows in the figure, it can be observed that the complete MFFM Module can detect more comprehensive changes, with more accurate edge information. In contrast, the changes detected using only the ASPP Module exhibit smoother edges but fail to effectively characterize the correct change regions. Results obtained solely with the Attention Module show more instances of missed detections, particularly evident in the fourth row, emphasizing the improved accuracy in model recognition achieved through the incorporation of the Attention module.

Figure 9. Visualization of ablation experiments’ result. ‘Attention'+'ASPP’ equals to ‘MFFM’ (a) image1 (b) image2 (c) label (d) 'VHR + SAR'+'SFLM'+'Attention’ (e) 'VHR + SAR'+'SFLM'+'ASPP’ (f) 'VHR + SAR'+’SFLM'+'MFFM’.

6. Discussion

6.1. The effectiveness of SAR combined with VHR in build-up CD

6.1.1. SAR in build-up CD

Some studies have shown that SAR data can be used for change detection (Jia and Yao 2023). We design experiments that utilize simple CNN or SFLM to process long-term SAR data. As shown in Table 5, although the experiments prove that SFLM can effectively improve the F1/IoU by 3.45/2.53%, neither of the two methods using long-term SAR data alone can achieve the accuracy of VHR images, which may be attributed to the coarse spatial resolution and speckle noises of long-term SAR data, as well as tremendous temporal information.

Table 5. Result of using long-term SAR alone as the input.

Method	Rec.	Pre.	F1	IoU
SAR	22.32%	61.93%	32.81%	19.62%
SAR + SFLM	36.57%	35.97%	36.26%	22.15%

6.1.2. ‘VHR + SAR’ in build-up CD

Currently, most networks are designed for Bi-VHR. None of them integrate long-term SAR data with Bi-VHR images for CD. The simplest way to realize it without making any changes to classic CD methods is concatenating long-term SAR data and Bi-VHR images. We design experiments to test the effectiveness of simple concatenation in CD methods and the result is shown in Table 6. ‘Concat’ indicates that long-term SAR data and Bi-VHR images are fused by concatenation as the input, and ‘proposed’ represents the proposed method in our study.

Table 6. The result of methods integrating long-term SAR data, ‘↓’ and ‘↑’ denote the decline and rise compared with using Bi-VHR images alone, respectively.

Method	Rec.(%)	Pre.(%)	F1(%)	IoU(%)
BiDateNet (concat)	18.34 (↓41.31)	51.51 (↓12.25)	27.04 (↓34.59)	15.64 (↓28.91)
BIT-CD (concat)	19.51 (↓28.14)	48.30 (↓16.79)	27.80 (↓27.22)	16.14 (↓21.81)
ISNet (concat)	25.75 (↓16.77)	42.13 (↓36.21)	31.96 (↓23.16)	19.02 (↓19.02)
FC-Siam-diff (concat)	2.45 (↓44.72)	47.66 (↓15.98)	4.66 (↓49.52)	2.39 (↓34.77)
SNUNet-CD (concat)	46.39 (↑2.69)	47.10 (↓18.73)	46.74 (↓5.78)	30.50 (↓5.12)
HOLS-CDnet (concat)	61.01(↓5.77)	35.81(↓24.91)	45.13(↓18.48)	29.14(↓17.50)
HOLS-CDnet (proposed)	67.86 (↓4.41)	62.34 (↑10.84)	64.99 (↑4.84)	48.13 (↑5.12)

It is obvious that simple concatenation decreases the performance of CD tremendously. The reason might be that since long-term SAR data have numerous channels, simple concatenation enables the network to pay more attention to long-term SAR data, making it difficult to fully utilize the important spatial information of Bi-VHR images. However, the proposed method improves the result on all metrics compared with using Bi-VHR images alone, which proves that using multi branches to process the two modal data is more suitable and MFFM is useful for integrating Bi-VHR images and long-term SAR data.

6.1.3. Bi-SAR or long-term SAR in build-up CD

To further prove the superiority of long-term SAR data, we conducted experiments based on the HOLS-CDnet framework. In these experiments, ‘SAR(1VV, 1VH)’ denotes the use of bi-temporal optical and bi-temporal short-term SAR data, with each short-term SAR time phase comprising one VV and one VH band acquired in December of the corresponding year. In contrast, ‘long-term SAR’ indicates the use of bi-temporal optical and bi-temporal long-term SAR data, where each long-term SAR time phase comprises 24 bands (VV and VH bands for each of the 12 months), totaling 48 bands. The experimental result presented in Table 7 below clearly demonstrates a significant improvement in F1 and IOU values for the long-term SAR method compared to the bi-temporal SAR approach, supporting the validity of our experimental design.

Table 7. Experimental results of using long-term SAR and short-term SAR.

SAR Data	Pre.	Rec.	F1	IoU	OA
SAR (1VV, 1VH)	55.48%	59.62%	57.48%	40.33%	94.09%
long-term SAR	62.34%	67.86%	64.99%	48.13%	95.11%

The change detection result of two methods is visualized in Figure 10. As clearly evident from the first and last rows, the change areas detected solely using bi-temporal short-term SAR appear noticeably fragmented, and the detected edges lack precision. In comparison, long-term SAR captures more comprehensive change information, suppressing false positives. This result effectively demonstrates the necessity of employing the long-term SAR method.

Figure 10. Visualization of change detection results of using long-term SAR and short-term SAR. (a) image1 (b) image2 (c) label (d) short-term SAR with 1VV, 1VH for each temporal phase (e) long-term SAR.

6.2. Sensitivity of the hyper-parameters $\mathit{w}$ and $\mathit{\lambda }$

$w$ is a hyperparameter used to fuse $\mathrm{change}\mathrm{ma}{p}_{\mathrm{VHR}}$ and $\mathrm{change}\mathrm{ma}{p}_{\mathrm{fuse}}$ in equation (13). To get the best $w$, we set it from 0.1-0.9 with an interval of 0.1. Table 8 reveals that the optimum overall performance occurs when $w$ is 0.8. When $w$ is less than 0.4, the result becomes better with the increase of $w$. When the parameter is greater than 0.4, the results fluctuate with the increase of the value.

Table 8. Sensitivity experiment of $w$_.

$\mathrm{w}$	F1	IoU
0.1	64.08%	47.15%
0.2	63.99%	47.04%
0.3	63.87%	46.92%
0.4	62.93%	45.92%
0.5	64.51%	47.61%
0.6	64.95%	48.10%
0.7	63.80%	46.84%
0.8	64.99%	48.13%
0.9	64.50%	47.60%

$\lambda$ is the balance factor in the Loss function of Equation (16), which is used to adjust the weight between BCELoss and DiceLoss. In order to get the best value for $\lambda$, we choose it from 0.1 and 0.9 in steps of 0.1. Table 9 shows that when $\lambda$ is 0.5, F1 and IOU have the highest values, reaching 64.99% and 48.13% respectively. When the value of $\lambda$ is less than 0.4, with the increase of $\lambda$ the result becomes worse. When $\lambda$ is greater than 0.5, F1 and IOU fluctuate around 64.5% and 47.5%, respectively.

Table 9. Sensitivity experiment of $\lambda$_.

$\lambda$	F1	IoU
0.1	62.38%	45.33%
0.2	61.86%	44.79%
0.3	60.65%	43.52%
0.4	63.33%	46.33%
0.5	64.99%	48.13%
0.6	64.45%	47.54%
0.7	64.54%	47.65%
0.8	64.68%	47.79%
0.9	64.35%	47.44%

6.3. Time costs and number of parameters

In our experiments, we conducted measurements of the Floating Point Operations per Second (FLOPs), the number of parameters, and the prediction time in the dataset. The FLOPs is a metric used to quantify the computational complexity of a neural network model, which represents the total number of floating-point operations, such as additions and multiplications, performed by the model per second. Table 10 presents the calculated results.

Table 10. Time costs and number of parameters.

Method	FLOPs(G)	params(M)	inference time (s)
MTCDN	56.88	47.82	0.06
FC-Siam-diff	20.37	27.51	0.01
ISNet	23.99	38.42	0.06
BIT-CD	10.63	3.50	0.03
SNUNet-CD	54.83	12.03	0.01
MSCANet	14.80	16.42	0.02
BiDateNet	45.80	13.40	0.01
DASNet	100.72	48.22	0.05
HOLS-CDnet	221.67	197.94	0.02

Among all the models compared, the value of FLOPs of HOLS-CDnet of 221.67G, which means that our model requires more computing resources. In terms of the number of parameters, our proposed network has far more parameters than all other networks. However, in terms of time consumption, our network only takes 0.02s. Our model has constructed a dual-branch input structure specifically designed for long-term SAR and optical image fusion, which requires significant computational resources. If the networks such as DASnet also adopt a dual branch encoder structure, the parameter quantity is also similar. Meanwhile, although our model has larger parameters, the actual inference time is not inferior to that of small networks.

7. Conclusion

In order to achieve the objectives of SDG11, accurate detection of changes in built-up areas is of vital importance for the sustainable development of cities. In this study, we address the challenge of data domain variations and background heterogeneity in built-up CD by combining long-term SAR with Bi-VHR images. Our proposal integrates multisource data through the use of a novel network called HOLS-CDnet, which employs SFLM to capture global changes from long-term SAR data, and MFFM to effectively integrate features from multiple sources. Experimental results demonstrate that our method outperforms state-of-the-art approaches, achieving the highest F1 score and IoU of 64.99% and 48.13%, respectively. Ablation experiments further validate the contributions of incorporating long-term SAR data, utilizing SFLM, and employing MFFM. These enhancements enable the network to effectively suppress background information and ensure regional integrity in the CD task. Furthermore, we acknowledge that relying solely on long-term SAR data for CD is challenging due to its coarse spatial resolution and speckle noise. Our approach achieves more robust performance by leveraging the complementary features of VHR and SAR data. Moving forward, we aim to further investigate the efficacy of integrating VHR images and SAR data in other change detection tasks. Additionally, we plan to delve deeper into the potential applications of built-up CD to support urban development decisions for SDG11.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The dataset and the code of HOLS-CDnet is available at https://github.com/Lihy256/HOLS-CDnet.

Additional information

Funding

This study was supported in part by the National Key R&D Program of China under Grant 2022YFB3903402, in part by the National Natural Science Foundation of China under Grant 42222106, in part by the National Natural Science Foundation of China under Grant 61976234, and in part by the Fundamental Research Funds for the Central Universities, Sun Yat-sen University under Grant 22lgqb12.