Real-time identification of collapsed buildings triggered by natural disasters using a modified object-detection network with quasi-panchromatic images

ABSTRACT

During disaster response, it is very important to obtain the information of collapsed building distribution accurately and quickly. However, limited by some practical factors, existed methods often suffer from the contradiction between the accuracy and efficiency of building damage extraction. This paper proposed a simple and effective framework to rapid recognize collapsed building objects using pre-disaster building distribution maps and post-disaster quasi-panchromatic remote sensing images. The proposed method is validated using several historical disasters in the xBD dataset and tested using three cases of earthquakes in terms of both effectiveness and efficiency. In addition, we have verified that the texture information of optical remote sensing images can be used as the main basis to judge whether a building is collapsed or not, so the panchromatic images are sufficient to enable the deep learning model to correctly recognize collapsed buildings. The experimental results indicate that using quasi-panchromatic images can alleviate the influence of style variations and diverse roof colors present in multi-spectral images on the model’s generalization performance, resulting in an average overall accuracy improvement of 2.4%. Additionally, the reduced data volume leads to an improvement in inference efficiency.

KEYWORDS:

• Building damage
• remote sensing
• object-level
• disaster response
• real-time
• deep learning

Introduction

Natural disasters occur frequently around the world. The destructive disasters may cause damage to buildings and threaten the safety of human life and property. After a disaster, it is very important to quickly obtain the distribution map of collapsed buildings accurately for disaster responses (Motosaka & Mitsuji, 2012). With the development of remote sensing technologies, disaster damage information extraction from remote sensing images on airborne or satellite platforms has gradually become a common means for post-disaster emergency response (Zheng et al., 2021).

At present, on-orbit and real-time remote sensing of surface anomalies has become a research hotspot in the field of earth observation (D. Li et al., 2021). On-board detection of building damage is an important part of future disaster emergency response. Due to the limited hardware conditions such as memory space on the satellites, how to reduce the dependence on multi-temporal remote sensing images and reduce the amount of data are key problems to be solved for the instant identification of building damage. Therefore, a lightweight and efficient detection model suitable for on-orbit satellites should be the focus of research (Q. Wang, 2022). In addition, making full use of pre-disaster key knowledge to improve the accuracy and efficiency of real-time response is also a direction worth exploring.

Nowadays, the deep learning technology represented by convolutional neural networks (CNNs) has achieved excellent performance in image pattern recognition tasks (Huang et al., 2022). For building damage extraction, although change detection methods using CNNs (Zheng et al., 2021) based on paired images can achieve higher accuracy, the reliance on both pre- and post-disaster homogeneous images reduces the efficiency of emergency response. Similar problem is also encountered in the methods based on the geometric changes of buildings pre- and post-disaster, such as the difficulty in quickly and large-scale acquisition of light detection and ranging (LiDAR) data (X. Wang & Li, 2020). In addition, the methods that only use post-disaster images can achieve rather high efficiency of emergency response, but might sacrifice accuracy due to the lack of pre-disaster information (Qing et al., 2022). It is still a practical challenge to well balance the accuracy and efficiency in terms of building damage identification for disaster response.

In general, the generalization ability of CNN models determines the recognition accuracy in real applications (Xu & Sun, 2018). In the process of disaster emergency, available remote sensing images might come from different data sources, which is very different from the images used for model training. The different spectral responses and imaging conditions of different sensors are important factors affecting the generalization performance of the CNN models (Hu & Tang, 2021).

The very high spatial resolution (VHR) optical remote sensing images can provide detailed surface texture information, which is important for extracting building damage. As shown in Figure 1, compared with the intact buildings, the texture of collapsed buildings in the post-disaster image is significantly different from that in the pre-disaster image, which may be the main features of CNN model to judge whether the buildings are collapsed or not. In addition, the styles of pre- and post-disaster images in this image are also obviously different. Compared with multispectral images, panchromatic (PAN) images are grayscale and have no spectral information but texture information can be improved (Shao & Cai, 2018). Currently, some studies use panchromatic images for classification (Seresht & Ghassemian, 2016) and object detection (Hou et al., 2019), but there is still a lack of research on the potential of panchromatic images in the task of collapsed buildings recognition.

Figure 1. Texture differences of building objects between images from pre- and post-disaster. The texture features are computed by histogram of oriented gradients (Dalal & Triggs, 2005). Pearson distance (Immink & Weber, 2014) measures the distance of feature vectors pre- and post-disaster. The similarity of features decreases as the value increases.

The building distribution can filter out complex background categories and provide the location and shape information of each building, which is crucial for accurately identifying collapsed building objects. The distribution maps of buildings can be collected and processed before the disaster occurs to prevent the need to extract buildings from pre-disaster images during disaster response, thereby reducing the dependence on the availability of pre-disaster images. At present, the building distribution data including building footprints and rooftops are becoming the basic geographic information, which is publically available in many parts of the world (Hu et al., 2022). We can obtain building distribution data from open access products, such as Bing maps of Microsoft (https://github.com/microsoft/GlobalMLBuildingFootprints), which provided building footprints from multi-source high-resolution remote sensing images in most regions of the world. The multi-annual China Building Rooftop Area (CBRA) dataset with 2.5 m resolution has also been released (Liu et al., 2023). In addition, the building footprints available on Open Street Map (http://www.openstreetmap.org/) covered many cities around the world, but the data in rural areas is particularly lacking at present. With the application of continuously optimized building recognition models and the increasing availability of high-resolution remote sensing imagery, the quality and completeness of building distribution products are improving rapidly. Therefore, it is necessary to make full use of the building distribution data to improve both the efficiency and effectiveness of future on-orbit disaster emergency tasks.

In this paper, we propose a deep learning-based framework for object-level recognition of collapsed buildings using pre-disaster building distribution maps and post-disaster panchromatic images (i.e. CopBud-PAN framework). Specifically, the building distribution map is used to provide the location of each building object to avoid the interference of complex backgrounds, which can be prepared in advance to save emergency time. The panchromatic images are used to extract post-disaster features and are expected to reduce the impact of sensor spectral response differences and data volume to further improve both accuracy and efficiency. The proposed method is dedicated to accumulating experience for future on-orbit real-time disaster emergency response.

The remainder of this paper is organized as follows. Section 2 summarizes the literatures of building damage extraction and the existing issues. Section 3 describes the data used. The specific methods are introduced in Section 4. The experimental results and a discussion are shown in Section 5. We draw conclusions in Section 6.

Related work

Building damage extraction methods

Morphological and geometric information, especially the changes in the height of buildings after disasters, can be used to accurately extract the building damage. Brunner et al. (2010) jointly used the SAR and optical images to assess building damage via geometric parameter estimation for the 2008 Wenchuan earthquake. Tong et al. (2012) used the IKONOS stereo image pairs to compare the height of buildings pre- and post-disaster to extract collapsed buildings. X. Wang and Li (2020) proposed a new framework to extract collapsed buildings by corner density and height difference using a combination of multiple types of images including LiDAR data. However, the specific data such as stereo image pairs and LiDAR are often inconvenient to obtain in a large area and in a short period of time after the disaster, so the emergency effect cannot be guaranteed (Zheng et al., 2021).

At present, most researchers used multi-band optical images to extract damaged buildings (Miura et al., 2020), in which the spectral features play an important role. Among them, the change detection method based on pre- and post-disaster images is a typical way, which mainly uses the changes of homogeneous images between two phases to identify the building damage (Bai et al., 2020; Shen et al., 2021; Weber & Kan, 2020). Specifically, the Siamese networks at the pixel- or object-level (Zheng et al., 2021) are often employed. However, it may be difficult to obtain the ideal paired data source under emergency conditions. The diversity of building roof colors and the spectral responses of different sensors may have a significant negative impact on the generalization performance of this kind of models (Ge et al., 2022). There are also studies using only post-disaster images to improve the efficiency of emergency response (Galanis et al., 2021; X. Li et al., 2011; Ma & Qin, 2012). In fact, without the assistance of pre-disaster information, the complex background categories and building damage features usually limit the recognition accuracy (Ge et al., 2023; Qing et al., 2022).

In addition, some studies have shown that the detailed textures in VHR images are important features for effectively extracting building damage (Dubois & Lepage, 2013; Song et al., 2012). For example, Klonus et al. (2012) applied a principal component analysis to multispectral texture images, and performed subtraction to obtain texture change information to extract a map of building changes. Janalipour and Mohammadzadeh (2018) extracted texture features from LiDAR data and applied the fuzzy inference systems to correlate them with the building damage extents, achieving accuracy superior in the 2010 Haiti earthquake. Qing et al. (2022) employed various texture information including contrast and variance as feature-enhanced bands, which improved the positioning accuracy of the damaged area. In general, there is currently a lack of research on whether texture features on single-phase post-disaster panchromatic images have potential advantages in accuracy and efficiency for deep learning models to identify object-level collapsed buildings.

It is worth mentioning that the object detection models of deep neural networks are often employed to directly identify buildings or disaster-affected areas at the object level from remote sensing images. For example, OEC-RNN (Huang et al., 2022) is a building object delineation model composed of three parts: target detection, pixel classification of edges and corners, and roof outline delineation. There are also some works using object detection methods for automatic identification of landslides (Ju et al., 2022; H. Li et al., 2022) and post-disaster damaged road (Zhao et al., 2022). In addition, Lu et al. proposed a fully convolutional object detection network to identify refugee shelters and tents from worldview-2 satellite images (Lu et al., 2021). Unlike these methods, instead of using an object detection model, the input data of the pre-disaster building distribution maps are employed to provide the location and shape of each individual building to obtain the object-level recognition results of collapsed buildings in our method.

Issues in current research

Among the existed methods, there are obvious contradiction between the accuracy and efficiency in terms of collapsed building recognition for real-time disaster response:

1. Relying on homologous images or multiple types of auxiliary data is not conducive to rapid emergency response.

2. Insufficient use of pre-disaster data that can directly characterize building attributes or even complete abandonment of pre-disaster information is an important factor affecting the recognition accuracy.

3. Lack of a simple and effective on-orbit detection framework that can quickly extract collapsed building using only key pre-disaster information and post-disaster images.

To solve these problems, the proposed method uses building distribution maps combined with the post-disaster panchromatic images to simultaneously improve the accuracy and efficiency of building damage extraction. At the same time, we analyzed the effect of panchromatic images for suppressing differences in sensor spectral response and the importance of texture features for recognizing collapsed building objects.

Materials

This paper utilizes two sets of data: the xView2 Building Damage Assessment (xBD) dataset for model training and validation, accessible at [https://xview2.org/dataset], and three historical earthquake cases (Haiti earthquake, Yushu earthquake, and Nepal earthquake) as test data.

Training and validation data

The xBD (Gupta et al., 2019) is an open access large-scale dataset for building damage extraction. The dataset includes paired pre- and post-disaster images of multiple natural disaster cases worldwide, such as tornadoes, earthquakes, etc. Since the proposed method is specifically designed for natural disasters that cause structural damage to buildings’ roofs, flood-related hazards are removed from the dataset because in these hazards, buildings surrounded by floodwater but with intact roofs are labeled as collapsed. Therefore, a total of 3678 ×BD images from 13 disaster cases are used in the experiment, with each RGB image having a size of 1024 × 1024 pixels. The distribution of the 13 disaster cases selected in this research is shown in Figure 2. We randomly divided 90% of this dataset as the training set, and the remaining 10% as the validation set for monitoring the training process.

Figure 2. Disaster distribution of the xBD dataset used in this research.

The post-disaster building objects in xBD dataset are divided into four categories from no damaged to destroyed. Our study was conducted under the two classification criteria of collapsed or not collapsed, because the distribution of collapsed buildings is urgently needed for disaster emergency and rescue. The no damaged and minor damaged buildings were regarded as the not collapsed category, and the major damaged and destroyed buildings were regarded as the collapsed category. Some examples of both image and label are shown in Figure 3.

Figure 3. Examples of the xBD dataset. Blue: buildings, red: collapsed buildings.

pre-image, post-image, building, distribution, post-label

Test data

Haiti earthquake

As shown in Figure 4(a), the first test area is located in Port-au-Prince, Haiti. The Haiti earthquake occurred on 12 January 2010. This earthquake (Mw 7.0) killed approximately 230 000 people (Bilham, 2010), and destroyed a lot of infrastructures, such as hospitals and schools (Singh, 2010). The post-disaster image of this area was taken by the WASP camera system of aerial photography on 25 January 2010 (Yu et al., 2011), and the corresponding building distribution was obtained by visual interpretation of the GeoEye-1 satellite image collected on 26 August 2009. The Haiti test data was cropped to obtain 15 post-disaster images with a size of 1024 × 1024 pixels.

Figure 4. Location map and test data. Haiti test area (a), Yushu test area (b), and Nepal test area (c).

Yushu earthquake

As shown in Figure 4(b), the second test area is located in Yushu Tibetan Autonomous Prefecture, China. The Yushu earthquake occurred on 14 April 2010, with the epicenter at 96.6°E, 33.2°N (J. Wang et al., 2012). This earthquake (Mw 6.9) caused approximately 14,700 casualties and destroyed many houses in Jiegu town (J. Wang et al., 2012). The post-disaster VHR image of the Yushu test area was collected in April, 2010 by aerial platform, and the corresponding building distribution was obtained by referring to the visual interpretation results from the Quickbird satellite image collected on 6 November 2004. The Yushu test data was cropped to obtain 104 post-disaster images with a size of 1024 × 1024 pixels.

Nepal earthquake

Figure 4(c) shows the third earthquake (Mw 7.8), which is located in Kathmandu, Nepal. The Nepal earthquake occurred on 25 April 2015, causing about 8000 deaths. Many buildings including ordinary houses and historical landmarks in this area were damaged (Hossain et al., 2015). The post-disaster VHR image of this test area was taken by the Pléiades satellite on 3 May 2015, and the corresponding building distribution was obtained by visual interpretation of the GeoEye-1 satellite image collected on 12 March 2015. The Nepal test data was cropped to obtain 380 post-disaster images with a size of 1024 × 1024 pixels.

The detailed data information of the three cases is shown in Table 1. The specific label process for the test cases is as follows: Firstly, the pre-disaster images were imported into ArcMap 10.6 tools, and the outlines of the buildings were manually delineated, creating vector maps of the building objects. Then, the pre-disaster building distribution map was overlaid onto the post-disaster images. The authors visually interpreted each building and marked whether it collapsed or not. Finally, the shapefile format files were converted to TIFF format, serving as the ground-truth.

Table 1. Data information of the test cases.

Cases	Data	Source	Bands	Acquisition time	Resolution
Haiti	Post-image	WASP camera	RGB	January 25, 2010	0.3 m
	Pre-buildings	GeoEye-1	/	August 26, 2009
Yushu	Post-image	Aerial platform	RGB	April, 2010	0.5 m
	Pre-buildings	Quickbird	/	November 6, 2004
Nepal	Post-image	Pléiades	RGB	May 3, 2015	0.3 m
	Pre-buildings	GeoEye-1	/	March 12, 2015

Building and image features

There are obvious differences in the building characteristics of the three areas. The Haiti area has well planned blocks and large single buildings. The buildings in Yushu area are densely distributed, and the buildings in Nepal area are relatively small. For the image features, the post-disaster images of the three cases were taken by different sensors, so the spectral responses of the ground objects are different. The images of Haiti and Yushu have high definition, while the image of the Nepal area is somewhat blurred with low brightness.

Since building damage should be more concerned, the grayscale histograms of collapsed buildings in different bands on post-disaster images were counted. As shown in Figure 5, the distribution of gray values of different bands on the same case is similar, and the difference between the red band and the other two bands is slightly larger. The histograms change significantly between different cases, indicating that the image sources and the regional differences of buildings lead to the diversity of the spectral features of collapsed buildings.

Figure 5. The grayscale histograms of collapsed buildings in different bands on post-disaster images of Haiti, Yushu and Nepal, respectively. The horizontal axis is the gray level of the red, green and blue bands in the range of 0–255, and the vertical axis is the frequency of collapsed building pixels of each grayscale. The horizontal comparison shows the distribution difference of collapsed buildings in different bands, and the vertical comparison shows the distribution difference of image grayscale features in different cases.

Methods

Overview

Figure 6 shows the CopBud-PAN framework for recognizing collapsed building objects. Specifically, the building distribution maps and a pre-trained model, i.e. CopBudNet, should be prepared before a disaster. After the disaster, if panchromatic images are not directly available, the quasi-panchromatic images will be generated using the multispectral images that can come from a different source than the ideal data. The proposed CopBudNet model is used to quickly identify the building damage.

Figure 6. The CopBud-PAN framework. (a) illustrates the process of transforming building distribution maps from pixel-level to object-level before a disaster. (b) depicts the generation of post-disaster RGB images into quasi-panchromatic images, followed by the utilization of a pre-trained model to predict collapsed buildings.

Generate object-level building distribution

The purpose of our study is to identify damaged building objects in the building distribution map. Therefore, the pixel-level pre-disaster building distribution map needs to be preprocessed into an object-level representation to facilitate its immediate use by the model after a disaster occurs. As shown in Figure 6 (a), each building object is assigned a unique integer number to convert the pixel-level (left) to object-level (right) building distribution map. Different colors represent different individual buildings. The processed building distribution map can be directly input into the CopBudNet model. After feature extraction through convolutional layers, the assigned numbers can be used to locate the range of each building object and extracts the corresponding feature maps for classification.

Generate quasi-panchromatic images

The integral values of the spectral response curves are used to set weights for each band to generate the quasi-panchromatic (QPan) images from the original RGB images of the xBD and test data. A larger integral value indicates that the remote sensing image has accumulated more reflected energy from ground objects in this band. First, the integral values of the spectral response curve in the red, green, and blue bands are calculated as $I_r$, $I_g$ and $I_b$ respectively. Then normalize them to obtain the weight of each band. Take the weight of the red band $W_r$ as an example:

(1) $W_r=\frac{I_r}{I_r+I_g+I_b},$(1)

where $W_g$ and $W_b$ are calculated in a similar way. Finally, we get the gray value of the QPan image:

(2) $V_p=W_r\ast V_r+W_g\ast V_g+W_b\ast V_b,$(2)

where $V_r$, $V_g$ and $V_b$ are the gray values of red, green, and blue bands respectively.

The band weights of QPan images involved in this paper are shown in Table 2, and an example of generated QPan image from RGB is shown in Figure 6(b). Due to the absence of spectral response curves for aerial images of the Haiti and Yushu test areas, we synthesized their Qpan images using the same weight settings as the training data (i.e. Worldview). In fact, directly using the mean values of RGB channels is also a reasonable choice.

Table 2. Band weights for generating QPan images.

Data sources	Band weights (R: G: B)
Worldview	0.3289: 0.3605: 0.3106
Pléiades	0.3336: 0.3326: 0.3338
Aerial platform	0.3289: 0.3605: 0.3106

CopBudnet

For real-time building damage detection, it is important to make full use of building distribution data that can be prepared in advance to reduce the reliance on pre-disaster images obtained on site. The application scenario is to directly detect collapsed building objects based on the location information of buildings provided by building distribution maps. In order to suppress the adverse effect of sensor spectral response differences on the performance of the model and minimize the amount of data, the panchromatic images were used instead of multispectral images. Of course, we need to verify that the texture information provided by panchromatic images are sufficient for CNN models to correctly recognize the collapsed building objects.

According to the requirement, this paper proposed the collapsed building object recognition network (i.e. CopBudNet). The architecture of the network is shown in Figure 7. The CopBudNet is realized based on the backbone network NASNet-Mobile (Zoph et al., 2018), which has the characteristic of less parameters. The NASNet-Mobile model extracts image features from neural cells obtained by neural architecture searching (Zoph et al., 2018), which have higher computational efficiency and less GPU memory usage.

Figure 7. Architecture of CopBudNet. Stem conv performs initial feature extraction through 3x3 convolution; reduction cell halves feature map dimensions with convolution and pooling; normal cell further extracts features using convolution and pooling without size alteration. RoI (region of interest) is the minimum enclosing rectangle of building objects; fully connected layer maps RoI vectors into 2 dimensions; softmax (Jang et al., 2017) is an activation function used to derive probabilities for building collapse.

Specifically, the post-disaster QPan image and the corresponding pixel-level pre-disaster building distribution map are overlaid to create a two-channel image used as input to the model. The building distribution map has the potential to help the network extract the features of post-disaster buildings. This branch undergoes stem convolution, reduction cell, and normal cell, reducing the feature map size and extracting high-dimensional features. In another branch, the object-level building distribution map is directly downsampled, without undergoing convolution and other operations, to the corresponding size (128 × 128). This process serves to provide the specific locations (i.e. the minimum enclosing rectangle) of each building for clipping individual feature maps. Finally, the clipped features of each building object are resized, and then input to the fully connected layer to classify whether the building is collapsed or not. The final output of the model is building damage distribution maps.

Experimental settings

The AdamW optimizer (Loshchilov & Hutter, 2019) is used during the training process, and the cosine learning rate annealing schedule with a minimum learning rate of 1 × 10⁻⁵ and an original learning rate of 1 × 10⁻³ is employed. The batch size of model training is 4, and the epoch interval of the decay learning rate is 10 by default. The loss function is cross-entropy, and the label smoothing trick is employed for regularization. We re-served the model with the highest accuracy on the validation set and apply it to our test set to evaluate its generalization performance. The experiments were conducted on a Tesla v100 GPU.

The overall accuracy (OA) is employed to quantitatively evaluate the recognition accuracy of the proposed method. In addition, the metrics of $\mathrm{recal}{\mathrm{l}}_{\mathit{cop}}$, $\mathrm{precisio}{\mathrm{n}}_{\mathit{cop}}$ and $\mathrm{F}{1}_{\mathit{cop}}$ of the collapsed buildings of the collapsed buildings, and $\mathrm{recal}{\mathrm{l}}_{\mathit{notcop}}$, $\mathrm{precisio}{\mathrm{n}}_{\mathit{notcop}}$ and $\mathrm{F}{1}_{\mathit{notcop}}$ of the not collapsed buildings are provided for reference. We define the collapsed buildings as positive, and the not collapsed buildings as negative.

(3) $\mathrm{OA}=\mathit{\hspace{0.17em}}\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}$(3)

(4) $\mathrm{Recal}{\mathrm{l}}_{\mathit{cop}}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$(4)

(5) $\mathrm{Precisio}{\mathrm{n}}_{\mathit{cop}}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$(5)

(6) $\mathrm{F}{1}_{\mathit{cop}}=\frac{2\mathrm{TP}}{2\mathrm{TP}+\mathrm{FP}+\mathrm{FN}}$(6)

(7) $\mathrm{Recal}{\mathrm{l}}_{\mathit{notcop}}=\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FP}}$(7)

(8) $\mathrm{Precisio}{\mathrm{n}}_{\mathit{notcop}}=\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FN}}$(8)

(9) $\mathrm{F}{1}_{\mathit{notcop}}=\frac{2\mathrm{TN}}{2\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}$(9)

where TP, TN, FP, and FN are the numbers of true positive objects, true negative objects, false positive objects, and false negative objects, respectively.

Results and discussion

This section compared the accuracy and efficiency of the proposed method in collapsed building recognition on QPan data and RGB multiband data, respectively. We then discuss the rationality of the method design in the subsection 5.2.

Both the QPan and RGB data were trained and predicted using the CopBudNet. The models were trained using the xBD training set, and their generalization performance was compared on the three test cases and the validation set.

Experimental results

Qualitative analysis

Haiti case

Figure 8 shows examples of identification results for CopBudNet on QPan and RGB images in the Haiti case. We can see that the model correctly recognized some collapsed building objects on the QPan images that were not correctly recognized on the RGB images. The recognition results on the QPan images are closer to the ground-truth.

Figure 8. Examples of identification results on the Haiti case. The pre-disaster images (a), post-disaster images (b), results on QPan images (c), results on RGB images (d), and the ground-truth (e).

(a) pre-image, (b) post-image, (c) QPan, (d) RGB, (e) ground-truth

Figure 9 shows the collapsed building recognition results of CopBudNet on QPan and RGB images on the whole Haiti test area. Although the model had some misclassifications on QPan images, the results are much better than on RGB images. Overall, the model has relatively good recognition results for large individual buildings, but poor performance for small collapsed areas with dense buildings, which may be since the recognition of these areas is inherently more difficult, and the model has rarely learned the similar features. A possible solution is to generate more simulated data to alleviate the sample scarcity problem through data augmentation for similar disaster scenarios.

Figure 9. Collapsed building recognition results on the Haiti case. Results on QPan image (upper left), results on RGB image (upper right), the ground-truth (lower left). The three small images in the lower right are the partial image before the earthquake, the partial image after the earthquake, and the corresponding identification results on the QPan image.

Yushu case

Figure 10 shows examples of identification results for CopBudNet on QPan and RGB images in the Yushu case. We can see that in the severely damaged areas, the model recognized more intact buildings as collapsed buildings on the RGB images, while the model can more accurately recognize collapsed buildings on the QPan images. The tones of the pre- and post-disaster images were very different, due to differences in the sensors onboard the satellite and the aviation platform. The stylistic difference between the training data and the images acquired during a disaster emergency may lead to a decrease in accuracy.

Figure 10. Examples of identification results on the Yushu case.

(a) pre-image, (b) post-image, (c) QPan, (d) RGB, (e) ground-truth

Figure 11 shows the collapsed building recognition results of CopBudNet on QPan and RGB images on the whole Yushu test area. The model recognized damage distributions roughly consistent with the ground-truth on both QPan and RGB images. Buildings in the central and eastern parts of the test area were not severely collapsed, but the model missed more scattered collapsed buildings on the RGB imagery.

Figure 11. Collapsed building recognition results on the Yushu case. Results on QPan image (upper left), results on RGB image (upper right), the ground-truth (lower left).

Nepal case

Figure 12 shows examples of identification results for CopBudNet on QPan and RGB images in the Nepal case. We can see that the post-disaster buildings in these areas are severely damaged, most of which are houses constructed of adobe and stone. Compared to the ground-truth, the recognition results on RGB images misclassified some buildings that were apparently not collapsed, while on QPan images, such errors were reduced. The recognition results of the whole area of the Nepal case are not shown, because the buildings in this area are too small, so the subtle differences in the results are difficult to observe on a large scale, and are not necessary for comparison and analysis.

Figure 12. Examples of identification results on the Nepal case.

(a) pre-image (b) post-image (c) QPan (d) RGB (e) ground-truth

Validation set

Figure 13 shows examples of collapsed building recognition results for CopBudNet on QPan and RGB images in the validation set. We focus on the analysis of the recognition errors in different disasters. From the visual samples of tornado and tsunami, we found that buildings that were severely damaged but not completely collapsed were easy to be misidentified. In the visual samples of wildfire and volcanic eruption the model is prone to misidentify building objects with very small areas. This may be due to the loss of some features of small buildings in the convolution process. In addition, the dataset itself also has certain quality problems, such as label errors and incomplete registration with images, which will also affect the application effect of the model.

Figure 13. Examples of recognition results on the validation set. The first row to the fourth row respectively shows a tornado disaster, a wildfire disaster, a tsunami disaster, and a volcanic disaster.

(a) pre-image (b) post-image (c) QPan (d) RGB (e) ground-truth

Quantitative evaluation

The collapsed building recognition accuracy of CopBudNet is shown in Table 3. We can see that for each test case as well as the validation set, the model achieves higher OA accuracy on QPan images. Especially on the Haiti case, the OA value of the QPan images is 3.5% higher than that of the RGB images, which is related to the fact that the images are derived from the aerial platform but the training data comes from the satellite platform, and may be attributed to the single-band panchromatic images can alleviate the style problem caused by different sensors. In terms of the precision and recall metrics for the collapsed and not collapsed buildings, the multiband data sometimes achieved higher scores, but the QPan data maintained a higher F1 score.

Table 3. Collapsed building recognition accuracy on RGB and QPan images on the test cases and xBD validation set (xBD).

Cases	Data	OA	Precision	Recall	F1	Precision	Recall	F1
			Collapsed			Not collapsed
Haiti	RGB	0.771	0.704	0.511	0.593	0.792	0.896	0.841
	QPan	0.806	0.716	0.670	0.692	0.846	0.872	0.858
Yushu	RGB	0.830	0.715	0.697	0.706	0.875	0.884	0.880
	QPan	0.844	0.717	0.775	0.745	0.903	0.873	0.888
Nepal	RGB	0.723	0.844	0.638	0.727	0.630	0.840	0.720
	QPan	0.746	0.826	0.709	0.763	0.668	0.797	0.727
xBD	RGB	0.867	0.780	0.780	0.780	0.904	0.904	0.904
	QPan	0.884	0.813	0.803	0.808	0.915	0.920	0.917

In addition, we can see in most cases that the F1 score of not collapsed buildings is significantly higher than that of collapsed buildings. This suggests that it is easier for the model to correctly recognition the intact buildings, perhaps because the features of collapsed buildings are diverse and complex. Comparing the performance of the models on different cases, it can be seen that there is a large gap in term of their OA score. Among them, the model achieved the highest score of 0.884 on the validation set, and only achieved a score of 0.746 on the Nepalese test data. This shows that due to the huge diversity of buildings in different regions and the difference in data sources, directly applying a trained model to disaster areas is likely to lead to a decrease in generalization performance.

Experimental results show that the QPan images are sufficient for the model to recognize collapsed buildings when the locations of building objects are given. Moreover, QPan images reduce the stylistic diversity of images from different data sources, which makes it easier to obtain better performance. It is worth mentioning that in practical applications, real panchromatic images can be used directly. Panchromatic images generally have higher resolution than multispectral images, which facilitates greater accuracy.

Efficiency

For disaster responses, the efficiency should be considered in addition to the accuracy. A smaller amount of data can save time in data transmission and speed up the model’s inferential efficiency. Compared to multiband images, panchromatic images reduce the number of bands and usually have less data volume.

We compared the prediction times of CopBudNet on three types of images: RGB images, panchromatic (Pan) images, and quasi-panchromatic (QPan) images. The model’s prediction efficiency on Pan images simulates the inference time when directly using panchromatic images as the model’s input, assuming that panchromatic images are a single band of sensor imagery and do not need to be generated from RGB images. The prediction efficiency on QPan images includes the time required for synthesizing a single-band image from RGB.

The results of the efficiency evaluation are presented in Table 4. We can see that Pan images can significantly improve the predict efficiency of the model, which can save valuable emergency time. Compared to RGB images, the model exhibits slightly higher recognition efficiency on QPan images, with a significant reduction in data volume, which can alleviate the tension of satellite storage space resources. The faster prediction speed on single-band images is attributed to the fewer parameters in the model’s first convolutional layer. Therefore, to achieve real-time emergency response, it is feasible to directly utilize Pan images. In cases where Pan band is unavailable, synthesized QPan images or even a specific band should be considered. It’s worth noting that the degree of efficiency improvement is related to the coverage scope of images in the disaster area.

Table 4. Predict time of CopBudNet and the data volume on the test cases and xBD validation set (xBD).

Cases	Data	Predict time (s)	Efficiency improvement	Data volume (MB)
Haiti	RGB	26s	/	23
	Pan	24s	8.3%	9
	QPan	26s	0	9
Yushu	RGB	70s	/	147
	Pan	63s	11.1%	53
	QPan	68s	2.9%	53
Nepal	RGB	110s	/	346
	Pan	98s	12.2%	163
	QPan	106s	3.8%	163
xBD	RGB	348s	/	486
	Pan	307s	13.4%	177
	QPan	341s	2.1%	177

Discussion

In this section, a method comparison experiment, three ablation experiments and a statistical analysis are conducted to verify the rationality of our method. Specifically, The performance of change detection methods and the proposed method for building damage extraction is compared in subsection 5.2.1; the experimental results using another backbone network are discussed in subsection 5.2.2; the effect of building distribution on feature extraction, and the rationality of using the minimum enclosing rectangle to capture features are discussed in subsection 5.2.3; the importance of different image features is discussed in subsection 5.2.4; the emergency identification of collapsed buildings in 2023 Turkey earthquake is shown in subsection 5.2.5.

Comparison with change detection methods

Currently, most building damage extraction methods are proposed within the framework of change detection based on pre- and post-disaster paired images. Due to the dependence on multi-temporal images, change detection is difficult to achieve the goal of satellite autonomous real-time emergency response. In the proposed method, it is assumed that only key pre-disaster knowledges, such as building distribution data, can be deployed on satellites to assist missions due to limited memory space. In this way, after a disaster occurs, a single post-disaster image combined with the pre-disaster building distribution maps can be used for rapid disaster damage identification.

In order to highlight the advantages of the proposed method compared with traditional change detection methods, we compared the CopBudNet with both BDANet (Shen et al., 2021) and ChangeOS (Zheng et al., 2021), which represent the state-of-the-art change detection methods in the building damage assessment task using deep learning. The methods are trained on the same set of disaster cases as the CopBudNet, and both three test cases and xBD validate set were used to assess their generalization accuracy.

Specifically, the ChangeOS model takes pairs of pre- and post-disaster images as inputs and outputs building damage maps. The model integrates deep object localization networks and disaster damage classification networks into a unified semantic change detection network for end-to-end building damage assessment. To make a fair comparison, we perform post-processing on the outputs of the recognition results by using the pre-disaster building distribution maps. This involves overlaying the recognition results with the building distribution maps and assigning the most frequent class of pixels within each building as the final category for that object. Similarly, BDANet also takes pairs of pre- and post-disaster remote sensing images as inputs. The same post-processing procedure as in ChangeOS is also applied to BDANet to refine its building damage recognition results.

Both ChangeOS and BDANet require predicting the locations of buildings from pre-disaster images and utilize the change features between pre- and post-disaster images for damage classification. In contrast, CopBudNet directly takes the pre-disaster building distribution maps as input and uses the features from the post-disaster images to determine whether the building has collapsed. While such a comparison may not be perfectly fair due to the differences in their methodologies, it can demonstrate which technology has more advantages and the degree of differentiation between the two in emergency identification tasks. Table 5 presents the main variables of these methods.

Table 5. Main variables of different methods.

Methods	Backbone	Learning rate	Optimizer	How to produce pre-disaster building map
BDANet	ResNet	0.0002	AdamW	Building segmentation
ChangeOS		0.03	SGD
Ours	NASNet-Mobile	0.003	AdamW	Existing products

Table 6 shows the accuracy of the three comparison methods. It can be seen that our method achieved significantly higher value in almost all of the metrics. The main reason is that the CopBudNet employed the pre-disaster building distribution maps to filter the complex background so that the convolution layers can focus on the features of the buildings themselves. The ChangeOS did not achieve better overall accuracy than the DBANet because the object-level change detection model tends to treat multiple dense buildings as a single object and cannot separate the collapsed and intact parts, which introduces more errors. The experimental results show that the scheme of building damage identification based on the pre-disaster building distribution maps and post-disaster images has advantages in accuracy compared with the change detection methods, and has great application potential in the task of real-time disaster emergency response.

Table 6. Comparison of collapsed building recognition accuracy of different methods.

Cases	Methods	OA	Precision	Recall	F1	Precision	Recall	F1
			Collapsed			Not collapsed
Haiti	BDANet	0.703	0.572	0.346	0.431	0.735	0.875	0.799
	ChangeOS	0.608	0.379	0.317	0.345	0.694	0.749	0.721
	Ours	0.771	0.704	0.511	0.593	0.792	0.896	0.841
Yushu	BDANet	0.674	0.628	0.685	0.655	0.718	0.664	0.690
	ChangeOS	0.614	0.687	0.651	0.669	0.519	0.560	0.539
	Ours	0.830	0.715	0.697	0.706	0.875	0.884	0.880
Nepal	BDANet	0.663	0.678	0.792	0.711	0.633	0.488	0.551
	ChangeOS	0.594	0.760	0.472	0.583	0.494	0.776	0.604
	Ours	0.723	0.844	0.638	0.727	0.630	0.840	0.720
xBD	BDANet	0.834	0.736	0.786	0.760	0.865	0.848	0.856
	ChangeOS	0.817	0.817	0.676	0.741	0.817	0.872	0.844
	Ours	0.867	0.780	0.780	0.780	0.904	0.904	0.904

Replace the backbone network

In the CopBudNet model, NASNet-Mobile is used as the backbone network. To further verify the universality of the experimental results, the backbone network was replaced by the ResNet (He et al., 2016). Due to the superiority of the identity mapping structure, ResNet has been used as the backbone network in many studies. In this section, CopBudNet that uses ResNet as the backbone is called CopBudNet (Res).

Similarly, the accuracies of CopBudNet (Res) on QPan images and RGB images on the test cases and validation set were compared. As shown in Table 7, the model achieves higher OA and F1 scores on QPan images. This indicates that the accuracy advantage exhibited by QPan images is not caused by the backbone network, supporting the conclusion that panchromatic images can alleviate the negative impact of spectral response differences on model generalization ability. In addition, there is no significant difference in the accuracy of CopBudNet and CopBudNet (Res) on test cases, and CopBudNet is the recommended model since NASNet-Mobile has less parameters.

Table 7. Collapsed building recognition accuracy of CopBudNet (res) on RGB and QPan images.

Cases	Data	OA	Precision	Recall	F1	Precision	Recall	F1
			Collapsed			Not collapsed
Haiti	RGB	0.776	0.621	0.794	0.697	0.885	0.767	0.822
	QPan	0.806	0.742	0.618	0.674	0.829	0.896	0.861
Yushu	RGB	0.826	0.650	0.877	0.747	0.940	0.804	0.867
	QPan	0.857	0.712	0.860	0.779	0.936	0.856	0.894
Nepal	RGB	0.731	0.842	0.657	0.738	0.640	0.832	0.723
	QPan	0.749	0.812	0.736	0.772	0.681	0.768	0.722
xBD	RGB	0.861	0.732	0.852	0.788	0.931	0.864	0.896
	QPan	0.878	0.790	0.815	0.802	0.918	0.906	0.912

Ablation of building distribution and feature capture

In CopBudNet, a post-disaster image and the corresponding pre-disaster building distribution map are concatenated into a two-channel image and then fed into the network. We speculate that the building distribution maps can help convolutional layers for image feature extraction, making the network pay more attention to buildings and their surrounding areas. To verify the effectiveness of this approach, an ablation experiment in which the post-disaster images were not concatenated with the building distribution maps was conducted. That is, the network only performs convolution operations on post-disaster images, and the building distribution maps are only used to capture the features of building objects. Such a model is called CopBudNet (No concat) in the followings.

In addition, CopBudNet captures high-dimensional building features using the minimum enclosing rectangle of the building objects. This approach preserves background features around buildings within the rectangles. To confirm whether these background features are useful for classification of building objects, an additional ablation experiment was conducted. Specifically, the background features captured by the minimum enclosing rectangle will be masked out, and then only the building object features are fed into the fully connected layer. Such a model is called CopBudNet (Mask) in the followings.

The quantitative results of the two ablation experiments with the original experiment on the validation set were compared. As shown in Table 8, we can see that not using the building distribution map for feature extraction or removing background features around the building has caused the overall accuracy to decline to some degrees. For precision and recall metrics, the balance between them is more important. Experiments show that the building distribution map as the second band of the post-disaster image is helpful for feature extraction, and retaining some background features around the building is useful for the model to judge whether the building object has collapsed or not.

Table 8. Collapsed building recognition accuracy obtained from different ablation experiments on the validation set.

Methods	OA	Precision	Recall	F1	Precision	Recall	F1
		Collapsed			Not collapsed
CopBudNet	0.884	0.813	0.803	0.809	0.915	0.920	0.917
CopBudNet (No concat)	0.877	0.773	0.840	0.805	0.928	0.893	0.910
CopBudNet (Mask)	0.874	0.753	0.872	0.808	0.940	0.876	0.907

The specific method of utilizing background pixels around buildings to assist in recognizing collapsed structures is not limited to just the bounding rectangle. It can also involve setting buffers of different radii around buildings. As for determining the optimal method, further exploration is warranted.

Feature statistics of building objects

The experimental results confirm that the texture features of panchromatic images are important for the model to recognize collapsed buildings. This section is dedicated to further supporting the conclusion from the level of image statistical features, that is, texture changes rather than grayscale differences play a greater role in judging whether buildings are collapsed or not. Specifically, the histogram of oriented gradients (HOG) (Dalal & Triggs, 2005) is used to separately count the texture features of each building object from both pre- and post-disaster image, and Pearson distance (Immink & Weber, 2014) is used to measure the texture similarity between pre- and post-disaster. HOG is often used as a feature descriptor in urban areas (N. Wang et al., 2022). Similarly, the histogram of grayscale is used to count the grayscale features of each building object, and Pearson distance is also used to measure the grayscale similarity between pre- and post-disaster. Pearson distance is defined as

(10) $\mathit{Pearson}\mathit{distance}=1-\mathit{r}$(10)

(11) $\mathit{r}=\frac{\mathit{Cov}\left({\mathit{X}}_{\mathit{pre}},X_{\mathit{post}}\right)}{\sqrt{\mathit{Var}\left[X_{\mathit{pre}}\right]\mathit{Var}\left[X_{\mathit{post}}\right]}}$(11)

where $X_{\mathit{pre}}$ and $X_{\mathit{post}}$ are the feature vectors of pre- and post-disaster building objects, respectively. $\mathit{Cov}$ and $\mathit{Var}$ refer to the covariance and variance, respectively. The value of Pearson distance is distributed in [0,2].

Figure 14 shows the Pearson distance between pre- and post-disaster texture features and grayscale features for each object on QPan images of the test cases. We can see that the Pearson distance of texture features between pre- and post-disaster of the collapsed buildings is generally significantly larger than that of the intact buildings. This shows that the collapsed building objects have obvious texture changes. For grayscale features, the distribution of Pearson distance values between collapsed and not collapsed buildings has a similar range. Furthermore, we can see that the texture Pearson distance between different building objects is more concentrated, especially the collapsed buildings. This shows that the texture difference between pre- and post-disaster of the collapsed buildings is relatively stable. The grayscale Pearson distance variance is large, which is affected by the diversity of roof styles.

Figure 14. Pearson distance of pre- and post-disaster texture features and grayscale features of building objects on panchromatic images. The Haiti case (a), Yushu case (b), Nepal case (c).

(a) Haiti, (b) Yushu, (c) Nepal

In general, the post-disaster texture features of collapsed buildings are significantly different from those pre-disaster. In addition, it is evident that collapsed buildings undergo morphological changes. Therefore, texture and morphological features should be used as the main basis for judging the damage of building objects.

Emergency response to 2023 Turkey earthquake

On 6 February 2023, two destructive earthquakes occurred in southern Turkey near the Syrian border, with magnitudes of Mw 7.8 and Mw 7.5 respectively. These seismic events resulted in numerous casualties and heavy property losses. After the disaster, we carried out emergency identification of collapsed buildings in some hardest-hit areas of Turkey, in order to support the post-disaster response work, and also to further verify the practicability of the method in this paper. Specifically, the proposed CopBudNet model trained on the xBD dataset was directly employed to identify collapsed building objects from post-disaster WorldView-3 images. These images have a spatial resolution of 0.3 m and three bands of RGB, and the imaging time is approximately 9 February 2023. This is the almost all of immediate available optical remote sensing data after the earthquake. In addition, the pre-disaster building distribution maps was processed from the Microsoft’s building footprint product (https://github.com/microsoft/GlobalMLBuildingFootprints), which were fed into the model along with the generated QPan post-disaster imagery of WorldView-3. The coverage of the WorldView images is shown in the purple area of the upper right subplot in Figure 15.

Figure 15. Comparison of identification results of collapsed buildings in downtown islahiye. Microsoft’s results (left), our results (right). The basemap is post-disaster WorldView-3 imagery.

After the earthquake, a joint team from Microsoft AI for Good and other departments also assessed the damaged buildings in the hardest-hit areas of Turkey (Robinson et al., 2023). Specifically, they employed a semantic segmentation model trained on the xBD dataset as the basis for building damage recognition. The team members interpreted and manually labeled regions of interest in post-disaster imagery, and fine-tuned the original model using these post-disaster data with ground-truth. The pixel-level outputs of their model included three categories: not collapsed buildings, collapsed buildings, and background. Finally, the results of model identification were superimposed with the pre-disaster building distribution data to obtain more regular results (Robinson et al., 2023). In contrast, our method eliminates the need for multiple steps but outputs collapsed building objects in an end-to-end manner through a simplified pipeline.

The approach employed by the Microsoft team also exhibits similarities with our proposed method, as both utilize the xBD dataset for training and leverage pre-disaster building distribution maps to derive assessment results. Therefore, their results can be mutually verified to a certain extent. Specifically, the overlapping area in the two results, i.e. downtown Islahiye, was employed for qualitative comparison and analysis. We removed places that were obscured by clouds and had few buildings.

Figure 15 shows the complete identification results of collapsed buildings in the comparison area of Islahiye, Turkey. It can be seen that the difference between the two results is mainly concentrated in the central region. The Microsoft’s results show large areas of collapsed buildings in town centers, while our results have relatively scattered damaged buildings. The building damage recognition results output by the two methods are similar in most areas as a whole. It is worth noting that the quality of building distribution maps directly affects the reliability of the recognition results. The building footprint data we can obtain under emergency conditions may have problems such as omissions due to untimely updates.

Figure 16 shows some enlarged areas of the two results. We can see from the first row that Microsoft’s method misidentified some intact buildings in the town center after the earthquake as collapsed, which may be due to the complexity of the buildings themselves and the limited generalization ability of the semantic segmentation model for post-disaster images. The second row shows that both methods are effective in detecting areas where collapsed buildings are concentrated. The example in the last row shows that the two methods can achieve relatively consistent recognition results in the edge area of Islahiye town.

Figure 16. Comparison of collapsed building recognition results in partially zoomed-in areas. Post-disaster WorldView-3 images Microsoft’s results and our results.

It can be seen from Figure 16 that the outlines of the buildings did not completely coincide with the buildings on the post-disaster images, which is mainly caused by the side view angle of the satellite imaging. We recommend using building footprints rather than roof-generated building distribution maps to mitigate the impact of imaging angles. In fact, it can be observed that even with slight misalignment between the pre-event building distribution maps and post-disaster images, the model still remains effective. One of the reasons is that during the process of extracting high-dimensional features, the model can leverage the surrounding features of the buildings. Additionally, incorporating imperfect data with slightly shifted building distribution maps into the training set can help the model better adapt to this issue.

Under the framework of CopBud-PAN, the CopBudNet model had output the distribution of collapsed buildings. We made statistics on the identification results in Islahiye output by our model to provide more reference information. As shown in Figure 17, the statistical results show that the number of collapsed buildings accounts for 6.05% of all buildings, and the proportion of buildings with an area of less than 500 square meters collapsed was the largest, indicating that smaller buildings are vulnerable to serious damage in this earthquake. It is worth noting that some buildings obscured by clouds were easily identified as the collapsed category, resulting in an overestimation of the number of collapsed buildings, which is worthy of further improvement for the model.

Figure 17. Statistics on the identification results of CopBudNet in Islahiye. The proportion of collapsed buildings (left), and the collapsed proportion of different areas (right).

Conclusion

To solve the contradiction between the accuracy and efficiency of building damage recognition in the process of disaster emergency, this paper proposed a new framework to quickly identify collapsed building objects using pre-disaster building distribution data and post-disaster panchromatic images. The contributions of this paper are as follows, which can provide references for the real-time remote sensing recognition of building damage and accumulate experience for the direction of on-orbit emergency response.

1. We demonstrate that, given the location of the buildings, the texture and morphological information is crucial and sufficient for the model to correctly identify collapsed building objects. Due to the diversity of buildings, it is difficult to determine their damage through the spectral comparison between pre- and post-disaster images in a large area, and it is necessary to suppress the influence of image style and roof color differences.

2. Experimental results show that the panchromatic images can alleviate the influence of sensor spectral response differences on the generalization performance of the model, and improve the OA accuracy by 2.4% on average across three disaster cases. In addition, compared with multiband images, panchromatic images can reduce the data volume and increase the inference efficiency by about 10%, which is conducive to ensuring the timeliness of emergency response.

3. The CopBudNet is an effective and efficient model for recognizing collapsed building objects in the proposed CopBud-PAN framework. It takes pre-disaster building maps and post-disaster quasi-panchromatic images as input, which can reduce the dependence on pre-disaster images, avoid complex backgrounds and suppress the influence of building spectral differences

.Admittedly, the current method is far from being deployed on satellites for instant diagnosis of building damage. For instance, we rely on high-quality building distribution maps as pre-disaster knowledge. Due to the diversity of buildings and their damage features, the generalization ability of models is still limited. The applicability of the method in more diverse resolutions, especially in images with resolutions lower than 1 meter, requires further research and validation. In addition, the focus of this paper is on the feasibility verification of solutions rather than the specific optimization of model structures. It is essential to adopt cutting-edge deep learning techniques to develop more powerful models.

Acknowledgments

This research is carried out under programming in Python and deep learning framework Pytorch.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The xBD dataset is available at https://xview2.org/dataset.

Additional information

Funding

This research was supported by National Natural Science Foundation of China Major Program (42192580, 42192584)