A review of multi-class change detection for satellite remote sensing imagery

ABSTRACT

Change Detection (CD) provides a research basis for environmental monitoring, urban expansion and reconstruction as well as disaster assessment, by identifying the changes of ground objects in different time periods. Traditional CD focused on the Binary Change Detection (BCD), focusing solely on the change and no-change regions. Due to the dynamic progress of earth observation satellite techniques, the spatial resolution of remote sensing images continues to increase, Multi-class Change Detection (MCD) which can reflect more detailed land change has become a hot research direction in the field of CD. Although many scholars have reviewed change detection at present, most of the work still focuses on BCD. This paper focuses on the recent progress in MCD, which includes five major aspects: challenges, datasets, methods, applications and future research direction. Specifically, the background of MCD is first introduced. Then, the major difficulties and challenges in MCD are discussed and delineated. The benchmark datasets for MCD are described, and the available open datasets are listed. Moreover, MCD is further divided into three categories and the specific techniques are described, respectively. Subsequently, the common applications of MCD are described. Finally, the relevant literature in the main journals of remote sensing in the past five years are analyzed and the development and future research direction of MCD are discussed. This review will help researchers understand this field and provide a reference for the subsequent development of MCD. Our collections of MCD benchmark datasets are available at: https://zenodo.org/record/6809804#.YsfvxXZByUk

KEYWORDS:

• Remote sensing
• information extraction
• change detection
• multi-class change detection
• review

1. Introduction

Change Detection (CD) is an essential task in the field of remote sensing, which can be understood as a technology to identify ground changes by analyzing multi-temporal remote sensing images of the same geographical region taken at different times (Singh 1989). In recent decades, CD has been extensively researched and applied in a wide range of domains, including natural resource management (Song et al. 2018b; Chen et al. 2016), disaster monitoring (Lei et al. 2019; Sublime and Kalinicheva 2019) and urban planning (Iino et al. 2017; Lyu et al. 2018).

With the continuous growth of satellite remote sensing technology, a great amount of remote sensing data can be used in CD, such as High Spatial Resolution (HSR) and hyperspectral remote sensing images (Shi et al. 2020). Rich remote sensing data sources can provide abundant information to detect differences in Land Use and Land Cover (LULC) in particular areas (Van de Voorde et al. 2013). This is very crucial in urban expansion, LULC monitoring and disaster assessment. Traditional Binary Change Detection (BCD) just concerns change and no-change areas between the bi-temporal images, which causes limitations in applications (Lanza and Di Stefano 2011; Rengarajan et al. 2016). Therefore, it has become a hot research topic in recent years to study the Multi-class Change Detection (MCD), which can distinguish the changes of LULC classes (Liu et al. 2019). Figure 1 compares the BCD and MCD tasks.

Figure 1. Illustrations of BCD and MCD.

The most intuitive scheme to realize MCD is to classify images at different time separately, and then obtain multi-class change map by comparing the results of different classification maps. This scheme is also commonly referred to as the method of Post-Classification (PCC) (Lal and Margret Anouncia 2015). However, this method usually has the problem of low detection accuracy due to the accumulation of errors. Therefore, many Direct Classification (DC) methods as opposed to the PCC method have been proposed for MCD (Chen et al. 2012; Hao et al. 2016; Yu et al. 2016). Since the DC method classifies the processed multi-temporal images directly, there is no accumulation of errors, so it can obtain higher detection accuracy. However, the samples required by the DC method are “from-to” change samples, which require the knowledge of the pre- and post-change land cover classes, so that is always hard to acquire. In addition, there are many other different types of MCD methods, such as unsupervised Change Vector Analysis (CVA)-based methods (Bovolo, Marchesi, and Bruzzone 2012) and Deep Learning (DL)-based methods (Dong et al. 2018). Due to the increasing development of DL in recent years (Li et al. 2021), the DL-based methods have been extensively used in MCD in recent years (Shafique et al. 2022). Similar with the traditional methods, the DL-based methods can be divided into the PCC method and the DC method, and both methods tend to use different semantic segmentation DL networks to conduct classification (Xia et al. 2022). Since the MCD task often has two inputs, so the representative DL network framework currently used for MCD is the double-branch network framework, such as ASN framework (Yang et al. 2021), PCFN framework (Xia et al. 2022) and Bi-SRNet framework (Ding et al. 2022). Therefore, it is necessary to conduct a comprehensive and timely review of MCD to summarize new technologies and applications.

Although there are many reviews of CD, these literatures usually only discuss BCD and do not focus on MCD (Ban and Yousif 2016; Mishra, Shrivastava, and Dhurvey 2017; Zhu 2017; Asokan and Anitha 2019; Wen et al. 2021). Only a few articles have mentioned MCD (Liu et al. 2019) due. Therefore, this paper provides a comprehensive review of MCD, including challenges, datasets, methods and applications.

2. Difficulties and challenges in MCD

Although MCD has been studied by scholars as early as the beginning of the 21st century (Zhou, Troy, and Grove 2008) or even earlier. However, the task of MCD has not been well developed as it is more complex and the dataset is sparser compared with the BCD task. In this section, we synthesize the factors affecting detection performance, then the difficulties and challenges of MCD are outlined below:

2.1. Visual feature confusion

Due to the different types of interference present in imaging, features in image pairs may be different visually compared to the original situation, that is, there is confusion. For example, since there are differences in imaging conditions between image pairs (e.g. different capture angles or different seasons), there may also be large spectral differences between unchanged features in optical remote sensing imagery, which enhances the difficulty in identifying the change region, thus affecting the accuracy of MCD. As shown in Figure 2(a), the features in the blue boxes in the image pairs are all grassland. However, as shadow occlusion caused by different capture angles and times, they have highly different spectral characteristics. The Synthetic Aperture Radar (SAR) images, on the other hand, are not affected by the imaging conditions mentioned above, but the inherent speckle noise existing in SAR images also makes MCD more difficult. Figure 2(b) shows a pair of SAR images, the red boxes show the regions of change, but it is difficult to identify the regions of change correctly by visual observation due to disturbance from speckle noise.

Figure 2. Difficulties and challenges in MCD: (a) spectral difference between the same surface features; (b) speckle noise interference in SAR image pairs; (c) “salt-and-pepper” noise; (d) imbalanced change classes in MCD reference maps; (e) binary change reference map and image at time1.

2.2. “Salt-and-pepper” noise

Since the spectral variability exists in the remote-sensing image pairs, using the traditional pixel method for MCD tends to generate a large amount of “salt-and-pepper” noise. The MCD results obtained by a traditional pixel classification method are shown in Figure 2(c), where it can be seen obviously that a large amount of “salt-and-pepper” noise is contained in the orange circle. The reason for this is that traditional pixel classification methods tend to assume that each pixel is independent of each other and do not consider the spatial relationships between pixels. To deal with this issue, some scholars have considered utilizing the spatial contextual information in images to perform MCD, such as object-based approaches.

2.3. Imbalance of change classes

Different from the classification of change and no change in the BCD, the MCD also needs to distinguish different change classes. Figure 2(d) shows a real change map, where the black color refers to the no-change region and the other colors refers to different change classes. It is not hard to discover that the area proportion of each change class in the real change map is unbalanced, and some change classes are only concentrated in small localized areas. Therefore, the classes of small changes are often difficult to be mined accurately because of the small number of training samples. To address this kind of problem, some scholars have tried to combine object-based image processing with Convolutional Neural Network (CNN) (Blaschke 2010; Zhang et al. 2018; Zheng et al. 2020a; Liu, Yang, and Lunga 2021). However, since the information obtained from object-based image processing is usually non-semantic, the combination of object-based processing with CNNs still remains at a superficial process combination (lack of feature combination) and thus still has many application limitations.

2.4. Complex background

Since changes tend to occur in parts of the region, the MCD process is disturbed by excessive no-change regional background. Figure 2(e) show the binary change reference map and the image at time1. The black color is the no-change region; the white color is the change region. It is easy to find a large areal proportion of no-change region in the binary change reference map. Moreover, no-change region frequently has the distribution of complex ground features, as shown the right image in Figure 2(e), which leads to negative MCD results.

3. Benchmark datasets and performance evaluation

The benchmark dataset not only serves as the benchmark for evaluating and verifying the performance of MCD algorithms, but also performs a key role in promoting MCD research. In recent years, several MCD datasets of different application targets have been published in the field of remote sensing to solve different problems. Table 1 lists the properties of these datasets for comparison. This section describes each of the datasets listed in the table individually. The trends and characteristics of these datasets are shown in Figure 3.

Figure 3. The development trend of existing MCD dataset.

Table 1. Comparisons of the available benchmark MCD datasets.

Dataset name	Change classes	Total image pairs	Image size	Data source	Resolution	Year	Application
Taizhou	3	1	400×400	Landsat7	30m	2000/2003	City expansion
Nanjing	3	1	800×800	Landsat7	30m	2000/2002	City expansion
Mts-WH	-	1	7200×6000	IKONOS	1m	2002/2009	Urban scene change
Yancheng	5	2	220×430	EO-1	30m	2005/2007	Vegetation change
USA	6	1	180×225	EO-1	30m	2004/2007	Farmland change
HRSCD	-	291	10000×10000	BDORTHO database	0.5m	2005/2006/2012	Land cover change
xBD	4	11034	1024×1024	WorldView-3	0.3m	-	Building damage assessment
SECOND	30	4662	512×512	Several platforms and sensors	0.5 3m	-	Land cover change

3.1. Taizhou dataset

The Taizhou Dataset mainly shows changes related to city expansion (Wu, Du, and Zhang 2013; Lyu, Lu, and Mou 2016). It consists of two images of size 400 × 400 pixels, collected from Landsat7 with a spatial resolution of 30 m. The available manually annotated samples of this dataset for MCD cover four classes of interest, including unchanged area, city change, soil change, and water change. This dataset is available from: https://github.com/I-Hope-Peace/ChangeDetectionRepository/tree/master/Dataset/Landsat/Taizhou

3.2. Nanjing dataset

The major changes in the Nanjing dataset occur due to complex city expansion as well as farmland changes, which is more complex and challenging (Du et al. 2019). It contains two images of size 800 × 800 pixels. The rest details about Nanjing dataset are provided in the above images. This dataset is available from: https://github.com/I-Hope-Peace/ChangeDetectionRepository/tree/master/Dataset/Landsat/Nanjing

3.3. Mts-WH dataset

The Mts-WH dataset was created for the task of evaluating the performance of scene MCD (Wu, Zhang, and Zhang 2016). The multi-temporal images in Mts-WH dataset were collected by the IKONOS sensors at 2002 and 2009. The images size is 7200 × 6000 pixels with four bands, and the spatial resolution is 1 m. The Mts-WH dataset is available from: http://sigma.whu.edu.cn/newspage.php?q=2019_03_26

3.4. Yancheng dataset

This dataset includes hyperspectral EO-1 Hyperion bi-temporal images of two locations with 242 bands, taken in Yancheng, Jiangsu Province, China (Song et al. 2018a). The images have a spatial resolution of 30 m and size of 220 × 430 pixels. There are five main categories of land-cover changes, which are associated with changes in vegetation, bare soil and water, including the transitions between vegetation, bare soil and water. The Yancheng dataset is available from: https://github.com/SicongLiuRS/Hyperspectral-Change-Detection-Dataset-Wetland-Area

3.5. USA dataset

The USA dataset contains the 2004 and 2007 bi-temporal hyperspectral image pairs acquired from the Hyperion sensor attached to the EO-1 satellite (Liu et al. 2019). The research area is irrigated farmland in Benton County, Oregon, USA, and is 180 × 225 pixels in size. There are six change classes, and the main land cover changes are the transitions between different types of crops, soils and other land cover types. The USA dataset is available from: https://github.com/SicongLiuRS/Hyperspectral-Change-Detection-Dataset-Irrigated-Agricultural-Area

3.6. HRSCD dataset

The HRSCD dataset can be considered as the first large-scale dataset for MCD (Daudt et al. 2019). It consists of 291 co-registered RGB image pairs of 10,000 × 10,000 pixels, with five land cover classes and pixel-level change labels. And the image in this dataset all have the same resolution of 0.5 m. The HRSCD dataset is available from: https://ieee-dataport.org/open-access/hrscd-high-resolution-semantic-change-detection-dataset#files

3.7. xBD dataset

The xBD dataset is a large-scale dataset that can be used for multi-class disaster building CD (Gupta et al. 2019). The complete xBD dataset includes 19 different kinds of post-disaster satellite images with a total of 22,068 images which have 850,736 buildings. The images in this dataset were acquired by WorldView-3 with a spatial resolution of 0.3 m. Each image has a size of 1024 × 1024 pixels. It includes four different types of change (i.e. four different levels of damage) to address the lack of a scale for building damage assessment. The xBD dataset is available from: https://xview2.org/dataset

3.8. SECOND dataset

This benchmark dataset was created for MCD (Yang et al. 2021). The SECOND dataset includes images across various platforms and sensors, for a total of 4662 image pairs, and each image has a size of 512 × 512 pixels. The images in this dataset were collected mainly from Hangzhou, Chengdu, and Shanghai. This dataset has six major land cover classes and result in 30 common change classes, which are related to natural and anthropogenic geographical changes frequently. The SECOND dataset is available from: http://www.captain-whu.com/project/SCD/

3.9. Evaluation metrics

Accuracy evaluation is vital to evaluate and analyze the performance of MCD algorithms. The commonly used evaluation indexes include: Overall Error (OE), Overall Accuracy (OA), Average Accuracy (AA), Kappa Coefficient (KP) and F1-score (Wu et al. 2021). The OE is the sum of misclassified pixels. The OA is the percentage of properly classified pixels compared with all pixels. The AA is the mean of the sum of the recall of each class. The KP indicates the agreement among the final MCD results and ground truth. The F1-score is a comprehensive evaluation index, which can be expressed by the harmonic average value of precision and recall. The calculation formulas of these indexes are as follow:

(1) $\mathit{OE}=\mathit{FP}+\mathit{FN}$(1)

(2) $\mathit{OA}=\frac{\mathit{TP}+\mathit{TN}}{\mathit{TP}+\mathit{FP}+\mathit{TN}+\mathit{FN}}$(2)

(3) $\mathit{AA}=\mathit{sum}(\mathit{recall})/\mathit{n}$(3)

(4) $\mathit{KP}=\frac{\mathit{OA}-\mathit{PRE}}{1-\mathit{PRE}}$(4)

(5) $\mathit{PRE}=\frac{(\mathit{TP}+\mathit{FP})\times (\mathit{TP}+\mathit{FN})+(\mathit{TN}+\mathit{FN})\times (\mathit{TN}+\mathit{FP})}{{(\mathit{TP}+\mathit{TN}+\mathit{FP}+\mathit{FN})}^2}$(5)

(6) $\mathit{F}1=2\times \frac{\mathit{precision}\times \mathit{recall}}{\mathit{precision}+\mathit{recall}}$(6)

where the n represents the number of change classes; TP shows counts of positive samples in the correct classification; TN means counts of negative samples in the correct classification; FN denotes counts of positive samples in the incorrect classification; FP is the negative sample number of mis-classification; and the PRE is part of the Kappa coefficient calculation formula and indicates chance consistency.

4. Techniques of multi-class change detection

The MCD aims to detect and identify changes in various LULC classes, which are of great importance in remote sensing applications. Compared with BCD, MCD is more complex, because it requires more than detecting changes, it needs to distinguish different various classes of change. As shown in the Figure 4, they are the general processes of BCD and MCD tasks.

Figure 4. The general processes of BCD and MCD tasks.

In order to have a more comprehensive overview of different categories of MCD methods, this section divides MCD into three categories according to different categories of changes: 1) Ternary change detection, 2) Multiple change detection, 3) Semantic change detection. The details will be described below.

4.1. Ternary change detection

In traditional BCD, it is usually assumed that changes are located on one side of the histogram of log-ratio. However, changes may occur on both sides of the histogram. Therefore, the concept of ternary CD was proposed by Bazi, Bruzzone, and Melgani (2006), who also advanced a double-threshold method based on Kittler-Illingworth (KI) (Kittler and Illingworth 1986) for ternary CD based on the histogram of change images. Different from the traditional BCD which only detects the change area, the purpose of ternary CD is detecting changes and classifying them as positive and negative changes. In practice, the positive change class (C+) usually represents new artificial features (such as buildings rebuilt after a flood), while the negative change class (C-) represents the opposite changes. In general terms, the C+ is the change class that appears, and C- is the change class that disappears.

As of now, there has been a few articles about ternary CD, and the data on the use of these methods are primarily related to SAR images. Bovolo, Marchesi, and Bruzzone (2012) proposed compressed CVA (C²VA) that can detect multiple changes. In fact, ternary CD can be considered as a special case in C²VA technique, and the C²VA technique and this special case is shown in Figure 5(b). The details of C²VA are covered in the next section.

Figure 5. Illustrations of the C²VA technique and TCD in the polar coordinate system. (a) the C²VA technique in the two-dimensional polar domain. (b) the TCD where is the special case of C²VA.

Traditional BCD methods for SAR images have also been used to perform ternary CD (Bezdek 2013), for instance, KI and Fuzzy c-means (FCM). However, these methods have poor accuracy of results. The reason can be summarized into two aspects: 1) SAR images are susceptible to speckle noise, which cannot be handled well by traditional methods. 2) The information representation capability of SAR image pixels and their neighboring pixels is quite constrained. Recently, DL has received growing concerns. Due to its powerful feature learning capability, DL can convert SAR image into feature maps to seize the critical recognition information and restrain non-sense changes due to the natural environment. Therefore, some scholars have applied DL technology to ternary CD. For instance, in order to address the information imbalance image problem in ternary CD tasks, the Deep Learning and Mapping (DLM) framework was proposed by Su et al. (2017). Moreover, Gong, Yang, and Zhang (2017) proposed a ternary CD framework for detecting changes in bi-temporal SAR images, which combines the advantages of Sparse AutoEncoder (SAE), CNN and unsupervised clustering to learn more robust change feature for ternary CD.

In conclusion, although the research on ternary CD is still in its preliminary stage, it is expected to become a new trend in the development of CD due to its more valuable practical applications (such as the detection of changes occurring in disaster reconstruction work).

4.2. Multiple change detection

As mentioned above, ternary CD can be considered a special case of C²VA, and the theory of multiple CD is proposed accordingly with the technology of C²VA. According to previous literature review, multiple CD is considered as a CD technique that can detect different kinds of changes. In more detail, multiple CD specifically refers to CD technique that can only distinguish between different kinds of changes without semantic change information.

At present, there are many methods related to multiple CD, and these methods can be regarded as CVA extension methods (Bovolo, Marchesi, and Bruzzone 2012; Liu et al. 2015; Saha, Bovolo, and Bruzzone 2019). In this kind of method, the complex change information within the initial high-dimensional feature space is compressed and projected into the two-dimensional feature space, also known as the two-dimensional polar domain as shown in the Figure 5(a). The two-dimensional polar domain is composed of the change magnitude $\mathit{\rho }$ and the change direction $\mathit{\theta }$, which can be calculated as (7) and (8):

(7) $\mathit{\rho }=\sqrt{{\sum }_{b=1}^B{I}_{b,D}^2}$(7)

(8) $\mathit{\theta }=arccos\left(\frac{1}{\sqrt{\mathit{B}}}\times \frac{{\sum }_{b=1}^B{I}_{\mathit{b},\mathit{D}}}{\sqrt{{\sum }_{b=1}^B{I}_{b,D}^2}}\right)$(8)

where the $I_{\mathit{b},\mathit{D}}$ represents the $\mathit{b}$th ($\mathit{b}=1.,\mathit{B}$) band of the multispectral difference image $I_D=I_2-I_1$; $I_1$ and $I_2$ are multispectral images obtained at times $t_1$ and $t_2$ over the same geographical area.

In this two-dimensional polar domain, if only the binary change map is needed, a threshold value T can be set according to $\mathit{\rho }$ for threshold segmentation and the result can be obtained. If it is necessary to obtain multi-class change maps, firstly, the method of threshold segmentation is used to distinguish the change area and no-change area, and then the clustering algorithm is executed on $\mathit{\theta }$ according to different change directions to subdivide the change area into different change classes. Therefore, although the final CD results obtained by such methods have different change class information, they cannot know the “from-to” information from which category to which category, that is, lack of semantic change information.

Among such CVA extension methods, the C²VA method is the most classical. The C²VA technique is an enhanced model of the CVA presented in 2012 (Bovolo, Marchesi, and Bruzzone 2012). This technique improves on the CVA technique by using vectorial angle to compress the direction angle from $[0,2\mathit{\pi })$ to $[0,\mathit{\pi }]$. Moreover, C²VA does not involve complex Bayesian estimation in distinguishing various changes, and instead the facilitated K-means (KM) clustering method, which greatly reduces the complexity in time. Based on this, few other techniques were presented for complex image, for example, hyperspectral images and Very High Resolution (VHR) images.

Liu et al. (2014) proposed the Hierarchical Spectral Change Vector Analysis (HSCVA) method, which was able to analyze the spectral changes at different levels in a completely unsupervised way. Considering the complicated structure of hyperspectral images, the change in the hyperspectral image was defined by the authors in terms of spectral change vectors, in which major and minor changes were defined based on the significance of spectral changes. A detailed analysis of the hierarchical spectral changes from the roughing to the finishing level allows for better simulation of complex change structures. At each level, initialization was executed to facilitate the selection of an automated change model to detect amounts of multiple changes, then a clustering process was implemented to discriminate multi-class change information.

Moreover, Liu et al. (2015) proposed a semi-supervised sequential spectral change vector analysis (S²CVA) technique used to detect and distinguish multi-class changes within hyperspectral images. Based on the C²VA technique, the S²CVA iteratively analysis the change information through a structure from top to bottom. Consequently, the information complicated change in the initial high-dimensional feature space is repeatedly compressed and projected into the series of two-dimensional feature spaces. While each of them correlates with a certain part of the entire spectral change vector space.

Marinelli, Bovolo, and Bruzzone (2019) proposed an unsupervised multiple CD approach on the basis of the binary spectral change vector, which transforms the initial spectral change vector into binary code words with outstanding change information. Different kinds of changes can be distinguished using the binarization of the spectral change vector. Then, the binary spectral change vector is expressed as a tree-diagram to sort out the types of changes while considering the hierarchical organization of the change components. The binary presentation was effectively used for building the tree-diagram because of the easier and clearer way of representing change information than the actual value representation.

In addition, there are many other related CVA extension methods, such as Robust Change Vector Analysis (RCVA) (Thonfeld et al. 2016). Besides, Liu et al. (2017) investigated the performance of different multiple CD in the literature and acquired experimental results of different multiple CD methods on different datasets.

4.3. Semantic change detection

Different from multiple CD, semantic CD can not only detect different change classes, but also detect the information of the change classes, namely the “from-to” information. In this way, semantic CD plays an important role when more detailed information is needed, especially when it is possible to distinguish between different types of change associated with actual land cover change. Compared with BCD, semantic CD provides richer information and is more complex in practice.

The method of semantic CD can be regarded as the method of remote sensing image classification in essence, including the method of PCC and the method of DC (Zhou, Troy, and Grove 2008; Hao et al. 2016). The comparison between PCC method and DC method is shown in Figure 6. In PCC method, the images of two phases are classified by classification method respectively, and then the two processed classification images are compared per-pixel to get the final results. This method is straightforward and understandable, but the change detection results are often affected by the classification results, resulting in the accumulation of errors, resulting in poor detection effect. Compared with the PCC method, the DC method usually classifies the images superimposed by the dual-phase images directly, without the phenomenon of error accumulation. However, the development of direct classification method is limited because it always needs to use a great number of artificially labeled “from-to” change samples, which are always hardly available.

Figure 6. The comparison between PCC method and DC method.

Over the past decades, remote sensing technology advances dramatically, and various approaches for semantic CD using remotely sensed data have emerged (Pacifici et al. 2007; Daudt et al. 2019; Mou, Bruzzone, and Zhu 2018; Zhu et al. 2022). These approaches can be divided into two general classes: Pixel-Based Change Detection (PBCD) (Lu et al. 2004; Chen et al. 2010) and Object-Based Change Detection (OBCD) (Toure et al. 2016; Wu et al. 2020). The majority of early CD approaches are the PBCD approaches, for instance, image differencing, Principal Component Analysis (PCA), and CVA (Bruzzone and Prieto 2000; Deng et al. 2008; Chen et al. 2013). Nevertheless, as the spatial resolution of remote sensing increases, the rich and complex geometric information in the images can be used to detect more subtle changes. The PBCD methods without considering the spatial relationship between pixels tend to lead to a large amount of “salt-and-pepper” noise. Consequently, the consideration of spatial contextual information is essential for CD using HSR images. While the OBCD methods analysis unit is an object composed of similar pixels, which utilizes the spatial and spectral information in remote sensing images, thus receiving more attention (Zhou, Troy, and Grove 2008; Hao et al. 2016; Peng and Zhang 2017).

With the development of DL, DL-based methods are gradually applied to semantic CD (Lyu, Lu, and Mou 2016; Wang et al. 2018; Mou, Bruzzone, and Zhu 2018). Early DL methods were mainly based on the method of PCC, and usually different semantic segmentation DL networks were selected, such as Deeplab V3, PSPNet and FarSeg et al. (Chen et al. 2017; Zhao et al. 2017; Zheng et al. 2020b) were classified, and then multi-class of change results were obtained by comparison after classification. DL-based method for DC roughly similar to DL approach in the BCD, but in the final for CD results, the depth of the direct classification method of study is not merely by threshold segmentation to get change and not change, but will change the category by means of clustering and classification to subdivided into all sorts of small class.

5. Applications of multi-class change detection

Nowadays, MCD has been applied to a broad range of actual situations. This review summarizes the representative applications of MCD, including: 1) LULC monitoring, 2) Urban expansion, 3) Disaster assessment.

5.1. LULC monitoring

The evaluation of LULC change is critical to comprehending the humans-nature relationship. During the history of remote sensing, various CD methods have been utilized to detect changes in surface objects, and more recent methods are in progress. The remote sensing images are the principal sources providing access to information on LULC changes over recent decades, and it uses a wide range of different algorithms depending on research needs. Different from the traditional BCD method cannot discriminate the semantic changes, MCD method can reveal details of the changes. Specifically, MCD method can detect detailed urban changes, such as buildings, vegetation, and roads.

For example, Huang et al. (2017) utilized the Ziyuan-3 images to recognize changes from 2012 to 2013, and the experimental results can reflect the fine-scale change of urban land cover in one-year. Chini et al. (2014) used IKONOS and GeoEye-1 images to analyze detailed land cover classes, including but not limited rooftops, farmland, roads, etc. As for land use CD, Wu, Zhang, and Zhang (2016) interpreted the type of change through combining spectral and SIFT features, such as the transition between housing and factories. Huang et al. (2019) generated land use maps of Shenzhen during 2005 and 2017 using VHR satellite data for monitoring detailed land use classes as shown in Figure 7.

Figure 7. Land use maps in the Shenzhen in 2005 and 2017 (Huang et al. 2019).

5.2. Urban expansion

With the constant expansion of urban areas, urban CD can dynamically monitor the change areas caused by urban expansion and provide data support for urban analysis. Currently, urban CD has emerged as a popular study. In order to better analyze urbanization, a dataset adapted to urban CD is essential. For instance, Urban Atlas (Congedo et al. 2016) provides pan-European comparable LULC data for functional urban areas (FUA) as shown in Figure 8, where the change layer counts the changes in functional urban areas. However, the required urban CD dataset currently should have two conditions: HSR and abundant prior information. The abundant structural features in HSR images can help to discriminate different features and thus acquire distinct change boundaries. And abundant priori information enables to detect various change types.

Figure 8. The urban atlas (Congedo et al. 2016).

Therefore, Tian et al. (2020) proposed a large-scale semantic annotated ultra-high resolution urban CD dataset named Hi-UCD, which enables to detect and analyze urban changes. Compared with other urban CD datasets, the Hi-UCD dataset is more challenging. First, the ultra-high spatial resolution exacerbates problems such as shadows and occlusions. Then, the number of change classes of ground objects is much larger compared with semantic classes, increasing the complexity of the MCD task.

5.3. Disaster assessment

Sudden natural disasters affect the safety of human life and property. However, disasters usually occur at unpredictable times and types, such as earthquakes, fires, floods and hurricanes. Generally, we cannot visually identify the severity of the damage in the disaster area, and usually use remote sensing technology to assess the area of damage for disaster assistance and response, such as building damage assessment.

The building damage assessment can be seen as a pre- and post-disaster CD issue (Plank 2014). To capture the different levels of the damage of disaster-stricken buildings, Gupta et al. (2019) proposed the Joint Damage Scale, as shown in Figure 9. The Joint Damage Scale involves four levels: Non-Damage, Minor Damage, Major Damage and Destroyed. Therefore, the building damage assessment can also be regarded as an MCD problem. For example, Zheng et al. (2021) proposed a deep object-based MCD framework, for building damage assessment, called ChangeOS.

Figure 9. The Joint Damage Scale in building damage (Zheng et al. 2021).

6. Development and future research direction

Although tremendous research progress has been made recently in MCD, the major techniques remain limitations to satisfy the current application challenges. To summarize the development of MCD, we have searched and analyzed the relevant literature published since 2017 on the Web of Science database using the keywords of change detection, multiclass, land cover change, etc. Depending on the different analysis viewpoint desired in the paper (as shown in Figure 10), we have finally selected 116 relevant literatures as the basis for our analysis. In addition, we also considered the quality of the published journals when selecting papers, for example, Remote Sensing of Environment, ISPRS Journal of Photogrammetry and Remote Sensing, IEEE Transactions on Geoscience and Remote sensing were all considered. The results of the literature analysis are shown in Figure 10.

Figure 10. The statistics for MCD as the subject of articles published since 2017 according to the Web of Science database from different viewpoints: (a) type of data source (LR/MR: Low/Medium spatial resolution images, HR: High spatial resolution images, HS: Hyperspectral images), (b) temporal resolution of data, (c) change categories, (d) prior knowledge.

According to the emphasis of this paper, we analyze the above articles from the different viewpoints: type of data source, temporal resolution of data, change categories, prior knowledge. The results of the econometric analysis show that the mainstream data source for MCD is currently LR/MR images (38%), and the remaining data source types account for a relatively equal share, while the use of multi-source data (22%) for MCD appears to be a trend. Also, although most of the current articles still use bi-temporal images (66%), using multi-temporal time-series data (34%) becomes a popular issue. From the viewpoint of change categories, most of papers perform semantic CD (71%), followed by multiple CD (21%), and ternary CD (8%) the least. And since the semantic CD task requires large number of labeled samples, the supervised learning approach (62%) is still the dominant method. Unsupervised learning approaches (31%) also occupy a place due to the needs of different tasks. In recent years, semi-supervised learning methods (7%) requiring fewer training samples have also emerged. Depending on our analysis, future research is likely to concentrate on (but not limited to) the below fields.

6.1. MCD with multi-source data

Due to the speedy progress in sensor and satellite technology, remote sensing images have abundant types and resolutions. It is noteworthy that there are potential problems of missing images or inconsistent image resolution when conducting bi-temporal or even multi-temporal MCD tasks. For example, Adriano et al. (2021) constructed a global multi-modal and multi-temporal dataset for disaster-building change detection. Therefore, it is significant to research the MCD of remote sensing images with multi-modal and establish appropriate solutions for practical applications. Besides, the use of multi-source data (e.g. Digital Elevation Model, Digital Line Graph and point cloud data, etc.) to assist MCD tasks is also meaningful in some specific tasks (Cao et al. 2020).

6.2. Change trend analysis in long time-series images

In addition to developing MCD for low-resolution and medium-resolution images, it is necessary to monitor the change trend in specific ground targets, for instance, Piao et al. (2021) constructed and analyzed time-series LULC maps to identify LULC changes in North Korea over a long time period and to understand trends in forest change. The use of long term-series of VHR images observe these changes and capture their trends and patterns facilitates decision-making.

6.3. Weakly supervised MCD

The current mainstream MCD methods involve numerous samples with exact labeling for fully supervised learning. However, sample labeling for MCD is more complicated and time-consuming compared to other tasks. Meanwhile, weakly labeled samples are more available, for instance, Sakurada, Shibuya, and Wang (2020) propose a new semantic CD network which can be trained by weak supervision of existing datasets. Therefore, it is of great interest to perform MCD from weakly labeled samples using DL-based models.

7. Conclusions

MCD provides more effective monitoring and analysis of land cover change on the Earth’s surface. However, the most review literatures have only focused on BCD up to now, lacking a comprehensive review of MCD. This paper provides a systematic review of MCD from the aspects of data, methods and applications, and summarizes the existing problems and challenges in MCD, which fills the gap of such review. Moreover, it proposes some future directions for MCD based on literature statistical analysis. The suggestions for future work include: 1) using multi-source data to facilitate MCD; 2) using long time series remote sensing data to observing change trends; 3) performing MCD based on weakly labeled samples.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

This is a review paper, no data has been used and the open data has been presented and referenced in the paper.

Additional information

Funding

This work was supported by the National Natural Science Foundation of China [grant number 41901306], and the Key Lab of Spatial Data Mining & Information Sharing of Ministry of Education [grant number 2022LSDMIS09].

Notes on contributors

Qiqi Zhu

Qiqi Zhu received the Ph.D. degree in photogrammetry and remote sensing from Wuhan University, China, in 2018. She has been an associate professor of School of Geography and Information Engineering, China University of Geosciences since 2018. Her research interest includes high resolution remote sensing image understanding, geoscience interpretation for multi-source remote sensing data and applications. She has published more than 40 research papers, including peer-reviewed articles in international journals such as Remote Sensing of Environment, ISPRS Journal of Photogrammetry and Remote Sensing, and IEEE Transactions on Geoscience and Remote Sensing. She was also a Referee of more than 20 international journals.

Xi Guo

Xi Guo is a postgraduate at the School of Geography and Information Engineering at China University of Geosciences (Wuhan). She received the B.S. degree from the Southwest University of Science and Technology, Mianyang, China, in 2020. Her research interests include remote sensing image processing and change detection.

Ziqi Li

Ziqi Li is an undergraduate student at the School of Geography and Information Engineering at China. Her research interests include remote sensing image processing and change detection.

Deren Li

Deren Li received the Ph.D. degree in photogrammetry from the University of Stuttgart, Germany, in 1986. He is a scientist in surveying, mapping and remote sensing from Wuhan University, China. He enjoys dual memberships of both Chinese Academy of Sciences and Chinese Academy of Engineering. He is also the member of International Eurasia Academy of Sciences and International Academy of Astronautics. He received doctor degree from University of Stuttgart and honorary doctorate from ETH Zürich. International Society for Photogrammetry and Remote Sensing awarded him the Honorary Member and the Brock Gold Medal in recognition of outstanding contributions to photogrammetry. So far, his publications include 11 monographs, over 700 journal articles and over 300 colloquium papers, which have been cited for more than 27000 times.