A review of remote sensing image segmentation by deep learning methods

ABSTRACT

Remote sensing (RS) images enable high-resolution information collection from complex ground objects and are increasingly utilized in the earth observation research. Recently, RS technologies are continuously enhanced by various characterized platforms and sensors. Simultaneously, artificial intelligence vision algorithms are also developing vigorously and playing a significant role in RS image analysis. In particular, aiming to divide images into different ground elements with specific semantic labels, RS image segmentation could realize the visual acquisition and interpretation. As one of the pioneering methods with the advantages of deep feature extraction ability, deep learning (DL) algorithms have been exploited and proved to be highly beneficial for precise segmentation in recent years. In this paper, a comprehensive review is performed on remote sensing survey systems and various kinds of specially designed deep learning architectures. Meanwhile, DL-based segmentation methods applied on four domains are also illustrated, including geography, precision agriculture, hydrology, and environmental protection issues. In the end, the existing challenges and promising research directions in RS image segmentation are discussed. It is envisioned that this review is able to provide a comprehensive and technical reference, deployment and successful exploitation of DL empowered RS image segmentation approaches.

KEYWORDS:

• Remote sensing
• deep learning
• image segmentation

1. Introduction

As a practical and comprehensive technology for non-contact detection, remote sensing (RS) is useful with its characteristics of wide coverage, high spatial/spectral resolution, and variable data structures to obtain specific target information from distance (N. Zhang et al. 2020). In particular, from RGB and near infrared (NIR) to thermal infrared (TIR) and microwave bands, the wavelengths accessible depend on the remote sensing sensors (see Figure 1), including digital cameras, multispectral cameras, hyperspectral instruments, infrared sensors, and lidars. Such commonly used sensors are employed on aircrafts, unmanned aerial vehicles (UAVs), satellites, and various platforms (T. Zhang et al. 2024). Considering the diverse properties of remote sensing sensors and the platforms mounted on, different combinations are utilized to obtain helpful remote sensing data (e.g. RS images) for task-specific applications. The acquirement and applications are illustrated in Figure 2. In recent years, the significant advances of RS technology have led to the promotion of remote sensing images in the spatial/spectral/temporal resolution, providing robust datasets for RS downstream tasks. Therefore, RS images have been widely used in different fields and domains, which are indispensably crucial to the Earth observation surveys, land use monitoring, and environment protection (Yasir et al. 2023).

Figure 1. The spectral wavelengths/bands that different sensors contained.

Figure 2. The procedure of remote sensing image segmentation.

Given that the RS image coverage has gradually expanding with various types of land objects, it is increasingly difficult to conduct information extraction for RS images. Consequently, as a significant method for information interpretation, image segmentation method is able to process the RS images with different paradigms, hence analyzing the features extracted from the collected images. It is noteworthy that segmenting remote sensing images are especially different from natural images considering the characteristics of multichannel, abundant details, and few labeled samples (W. Liu et al. 2020). Traditional image segmentation methods (e.g. pixel-based and edge-detection-based methods) utilized for early interpretation of images mainly rely on shallow semantic information, such as image texture and color gradient, restricting the further capability to obtain high-level semantic information (J. Yuan, Wang, and Li 2014). In addition, for target extraction tasks in complex environments, traditional methods cannot meet the needs of RS image understanding (N. Zhang et al. 2020).

As a result, increasing interests have been drawn by scholars to explore the potential of deep neural networks applying on RS image segmentation since 2014 (Y. Chen et al. 2014). Unlike traditional image segmentation models, learning-based approaches approximate the complicated nonlinear relation via designed multi-layer networks, finding out the hidden information from the abundant spectral/spatial data. Besides, DL has shown great superiority in multi-scale feature extraction and combination, outperforming traditional models with substantial improvement in RS image processing and analyzing problems. Under this circumstance, continuous development of new architecture designs and other advanced DL methods are proposed to enhance segmentation performance and accuracy of RS images. For example, RS image segmentation is conventionally interpreted by classical Convolutional Neural Networks (CNNs) and Transformer Networks (e.g. Swin Transformer Z. Liu et al. 2021) for RS downstream tasks (Kampffmeyer, Salberg, and Jenssen 2016; Nogueira et al. 2019; L. Wang, Li, Duan et al. 2022; Z. Xu et al. 2021). Some novel networks are also designed for RS image segmentation, including LANet (Ding, Tang, and Bruzzone 2020), HRCNet (Z. Xu et al. 2020), and MANet (R. Li et al. 2021). Meanwhile, self-supervised learning (W. Li, Chen, and Shi 2021; H. Li et al. 2022), contrastive learning (Bai et al. 2022; J.-X. Wang et al. 2022), and transfer learning (Cui, Chen, and Lu 2020; Panboonyuen et al. 2019; Wurm et al. 2019) have also been introduced to deal with sensing segmentation problems and achieved promising performance.

Thanks to the strong capability of feature representation, DL has become a superior algorithm that widely applied on different sub-fields with complex scenarios, including geological, agricultural, hydrological, and environmental applications. For example, Kussul compared CNNs with multilayer perceptron (MLP) and random forest for land cover classification problem (Kussul et al. 2017). Zhang proposed a new deep convolutional neural network (DCNN)-based approach for automated crop disease detection using hyperspectral images captured by UAVs (X. Zhang et al. 2019). Yuan designed a novel model-multichannel network named MC-WBDN for robust water body detection (K. Yuan et al. 2021). Martins applied CNN on tree canopies segmentation in urban environments using aerial RGB imagery (Martins et al. 2021). As a consequence, it could be found that due to different sensing platform and unique requirement in each task, DL algorithms are often specifically designed for particular research purpose. It means that there are a considerable number of networks and image processing methods designed for RS images with different resolutions and task requirements respectively.

Therefore, to obtain a comprehensive understanding regarding the algorithms and applications of remote sensing image segmentation using DL methods, a systematic review is necessarily required, providing beneficial information for further progressed researches. Comparing with the current existing reviews about RS image segmentation, our review contains comprehensive introductions of extensive researches, multiple types of application fields, and overall reviews about both classical models and advanced techniques.

• Our review provided complete introductions of various works about typical research and applications with clear categorizations for each section. For previous reviews, Li introduced three DL image classification models, including convolutional neural networks, stacked autoencoders and deep belief networks, with their performance on publicly available datasets and imagery, but did not introduce further applications on downstream tasks (Y. Li et al. 2018). Huang provided a comprehensive review of RS image segmentation with discussions about advanced methods and main challenges, but also took little account in the field of corresponding applications (L. Huang et al. 2023). On the contrary, our review devised a multi-perspective categorization of DL models for segmenting RS images from three dimensions (namely various architectures of RS image segmentation methods, label-efficient approaches, and the popular domain applications), so that readers could quickly grasp the overall development of RS image segmentation.

• Our review covered multiple types of application fields concerning image segmentation approaches. For previous reviews, Neupane reviewed and performed a meta-analysis about deep learning-based semantic segmentation on urban scenes in satellite images (Neupane, Horanont, and Aryal 2021). Zang introduced algorithm development, approaches summarizations and quantitative experiments focusing on land-use mapping task (Zang et al. 2021). On the contrary, we introduced and itemized the algorithm implementation among four popular downstream domains in detail, including geological applications, precision agriculture, hydrological applications, and environmental protection, making the review more comprehensive and inspiring for stakeholders of specific domain.

• Our review analyzed classical models and advanced methods (e.g. multi-modal models, diffusion models, and foundation models) related to the algorithm development in the field of natural image segmentation. For previous reviews, Yuan reviewed the development of deep learning methods for RS image segmentation, mainly focusing on the methods for non-conventional data and the demands for accuracy improvement and the lack of samples (X. Yuan, Shi, and Gu 2021). Lv summarized the advantages and disadvantages of segmentation models on RS images with detailed introduction about each structure, but took little account in the field of label efficient methods and other advanced methods (Lv et al. 2023). On the contrary, we provided an overall review covering some emerging and developing techniques recently, summarizing innovations and challenges for future points and further developments.

In summary, compared to previous articles, our review is more comprehensive with summarized challenges and corresponding approaches plus with categorized classifications of applications. Since adopting DL methods in remote sensing image segmentation will lead to several challenges not appeared before, we also analyzed the relevant challenges and introduced the proposed methods responding to them. With regard to the advantages of different DL methods, we will conduct a detailed review in the following parts. The main contributions of this review are summarized as follows:

• The survey covers 160 DL algorithms and correlated methods for comprehensive review, providing an overview of deep learning-based methods for image segmentation in remote sensing field, including the respective analysis of different models and the applicability of DL models responding to existing challenges.

• We provide an introduction of DL approaches applying for remote sensing image segmentation for popular and promising downstream tasks, which could be categorized into four types in our review (e.g. geological applications, precision agriculture, hydrological applications, and environmental protection issues).

• Some directions by using novel DL strategies are introduced, which can be applied in remote sensing image analysis in the future.

2. Remote sensing observation platforms and sensors

The quality of remote sensing images is determined by the spatial resolution, spectral resolution, and temporal resolution of data measuring system respectively. A high quality image is generally determined by high resolutions. Poor spatial resolution displays large pixels with poor land cover characteristics. Spectral resolution indicates the number of spectral bands for reflected radiance collection that determined by the number of bands and bands position in the electromagnetic spectrum. The temporal resolution defines revisiting frequency for a certain location. Regarding RS images for specific tasks, two remote sensing systems are popularly used, namely aviation remote sensing and space remote sensing. They are important parts of Earth observation system, complementing each other in various applications.

Aviation platforms mainly include manned aircraft, helicopter, unmanned aerial vehicle (UAV), balloon, etc. Since these low altitude platforms collect data at the highest flying-height from 100 to 18,000 m (Everaerts 2008), they could make up for the spatial and real-time monitoring limitations of traditional facilities (e.g. tripods, vehicles, and fixed rail systems) (N. Zhang et al. 2020). Therefore, with high efficiency and flexible operations, aviation remote sensing has become important platform to obtain high-resolution sensing images. Among these platforms, UAV-based ones are low cost and flexible to provide high spatial resolution data due to the low flight altitude. Unlike manned aircraft requiring high costs and maintenance price, UAV could fill the resolution gap between manned aircraft (0.2–2 m) and traditional ground-based platforms ($<1$ cm) with a typical spatial resolution of 1–20 cm (Wójtowicz, Wójtowicz, and Piekarczyk 2016). In addition, UAV is popular for its small volume, light weight, and simple operations, thus being widely applied in the fields of agriculture and forestry monitoring applications recently.

As depicted in Figure 3, UAV sensing system is mainly composed of UAV platforms, sensing units, and ground control systems. According to different ways to provide lift, UAV platforms could be divided into three categories, namely rotor-wing, fixed-wing, and hybrid UAVs. Rotor-wing UAVs rely on spinning rotors to provide aerodynamic lift while fixed-wing UAVs using rigid wings like an aeroplane. Besides, hybrid UAVs take advantage of fixed-wing and rotor-wing UAVs concurrently to possess a long flight duration. Various sensors have respective advantages in survey process. The optical sensing sensors with different spectral wavelengths are illustrated in Table 1, including RGB camera, color infrared camera, multispectral camera, hyperspectral camera, LiDAR, and thermal camera. RGB cameras generally attain red, green, and blue band information with the advantages of low cost and simple operation but showing limiting capabilities to obtain inherent target objects band details when they are similar (T. Zhang et al. 2024). Thermal sensors could obtain the data used for identifying ground objects and inversion surface parameters such as temperature and humidity. Therefore, thermal systems are capable of capturing objects changes, providing useful information for specific tasks such as natural disasters detection and plant disease early warning (Su et al. 2023). However, thermal sensors are not suitable for large-scale analysis due to that infrared thermal imaging is of blurred edges and low signal-to-noise ratio (N. Zhang et al. 2020).

Figure 3. UAV sensing systems are comprised of UAV platforms, sensing units, and ground control systems.

Table 1. Optical sensing sensors with different spectral wavelengths.

Optical sensing sensor	Band numbers	Information
RGB camera	3	Blue (450–515 nm), Green (525–600 nm), Red (630–680 nm)
Color infrared camera	3	Visible light (390–780 nm) and Near-infrared (780–1300 nm)
Multispectral camera	3–20	From visible light (390–780 nm) to Near-infrared (780–1300 nm)
Hyperspectral camera	Hundreds/thousands of bands	Visible light (390–780 nm), Near-infrared (780–1300 nm), Short-Wave Infrared (1300–2500 nm), Mid-Infrared (2500–5000 nm), Long-Wave Infrared (8000–14,000 nm)
LiDAR	–	e.g. 905 nm, 1550 nm and 1064 nm
Thermal camera	3–20	Infrared (760 nm–10 μm)

Considering that the spectral data reflected by land objects are various, the introduction of hyperspectral and multispectral cameras can increase the data diversity, thus improving remote sensing observation performance. The spectral resolution of hyperspectral image is on the order of 0.01 with hundreds of bands in the visible and near-infrared regions, while the one of multispectral image is on the order of 0.1 with only several bands. Compared with multispectral cameras, hyperspectral ones could obtain much more spectral information for accurate images analysis, paving the way for sophisticated tasks such as crop disease prediction and geological hazards assessment. However, hyperspectral cameras are expensive, hence only being applied for scientific researches. As a result, multispectral cameras are considered as a relative precise and cost-effective sensing unit recently (Jang et al. 2020; T. Zhang et al. 2024).

For space remote sensing, both satellites, space stations and rockets have been explored as platforms in the region scale sensing image assessment. Since the first Earth Resources Technology Satellites (ERTS) launched in 1972, satellites have been gradually utilized and achieved a general application on collecting remote sensing data, indicating the new era of remote sensing. With the lowest flight height of 150 km, satellites could obtain the electromagnetic wave information reflected by target objects in a long distance, resulting in the sensing images and spectral data required for follow-up processing. The band information details about some typical satellite sensors are shown in Table 2. Landsat-1 was first launched in 1972 by NASA with four bands information (green, red, two infrared) at a resolution of 80 m and a revisit time of 18 days (Su et al. 2023). Later, in 1984, Landsat-5 was launched equipped with Thematic Mapper sensor at a spatial resolution of 30 m in blue, green, red, near-infrared, and three infrared (including thermal) bands (Masek et al. 2006). It has been widely used in remote sensing area several decades later. The launch of Sentinel-2A is a key part of Global Monitoring for Environment and Security program (GMES) supported by the European Space Agency (ESA) and European Commission (EC) towards the needs of conducting agriculture, forest monitoring, and natural disaster management (Drusch et al. 2012). The most remarkable characteristic of satellite remote sensing image is the rich semantics information provided by its high spatial/spectral resolution in region scale, satisfying the general need in land use, Earth mapping, disaster detection, etc. It could be said that satellite sensing is the best choice for regional scale mapping research (N. Zhang et al. 2020).

Table 2. Band information details of typical satellite sensors.

Satellite name	Spectral bands (maximum resolution)	Launched year
Landsat-1	G, R, two IR (80 m)	1972
Landsat-5 TM	B, G,R, NIR, MIR,TIR (30 m)	1984
SPOT-1	G,R,NIR (20 m)	1986
WorldView-2	Panchromatic, B, G, Y, R, red edge, NIR (0.5 m)	2009
Landsat-8	11bands (30 m)	2013
Gaofen-1	9 bands (8 m)	2013
SPOT-7	Panchromatic, B, G, R, NIR(1.5 m)	2014
WorldView-4	Panchromatic, B, G, R, NIR (0.31 m)	2016
Sentinel-2	13 bands (10 m)	2015/2017
Gaofen-5	Hyperspectral (30 m)	2017
Gaofen-6	Hyperspectral (8 m)	2018
Gaofen-7	Hyperspectral, Lidar (3.2 m)	2019
Landsat-9	9 bands (30 m)	2021
GOES-18	16 bands (500 m)	2022
Beijing-3B	4 bands (0.5 m)	2022

Comparing drawbacks of the two popular platforms mentioned above (e.g. UAV-based and satellite-based platforms), it could be found that UAV-based platforms are much more likely to be disturbed by the harsh environment and could not equip large sensors. On the contrary, although the single image coverage collected by satellites is undoubtedly wide, it lead to lower spatial resolutions than remote sensing images from UAV-based platforms. In addition, the huge imaging data brings high cost at the duration of data transmission between satellites and earth. Currently, the interpolation, reconstruction, and other technologies are used to solve this problem (Keshk and Yin 2017).

Based on the aforementioned platforms and sensors, various remote sensing systems are deployed for specific tasks. In fact, the data collected in remote sensing area is generally customized RS image dataset for the segmentation target studied. Specifically, various datasets with different spatial and spectral resolution are integrated with specially designed artificial intelligence algorithms for industry applications. Moreover, in recent years, related institutions have conducted publicly available RS image segmentation datasets, pushing the promotion of DL model via high-quality training. We briefly show some common public datasets in Figure 4. For example, ISPRS Vaihingen and Postdam images are aerial datasets collected over the area of city. Vaihingen contains three band information (near-infrared, red, and green) while Potsdam is captured in four channels (near-infrared, red, green, and blue). GID is a large-scale dataset conducted for land use and land cover classification. It contains 150 high-quality images collected by Gaofen-2 from over 60 different cities in China, providing satellite images in blue, green, red, and near-infrared bands.

Figure 4. Commonly utilized RS images for segmentation: (a) ISPRS Potsdam images (UAV optical images), (b) ISPRS Vaihingen images (UAV optical images), and (c) Gaofen Image Dataset (GID) (Satellite multispectral images).

Considering the information limitation of sensors that capture images in RGB bands, multi-spectral/hyper-spectral sensors are mostly utilized for the expectation of attaining images with abundant spectral information, hence becoming popular components of RS systems. We choose two systems as examples in the following.

First, unmanned aerial spectral system adopts UAV equipped with multi-spectral/hyper-spectral sensors to obtain images with high spectral resolution. In addition, since being collected at a relative low altitude, multiple objects are successfully obtained in agriculture and urban area, providing rich details for objects detection and classification problems. Nevertheless, with chosen target scenarios, UAVs are limited in the observation view and landmark elements. In the agriculture sensing technology, Osco utilized a multispectral camera with four bands in green, red, red-edge, and near-infrared on UAV to estimate the number and location of citrus trees (Osco et al. 2020). For urban area, the evolution of UAV based remote sensing is efficient to solve urban management and ensure the sustainability simultaneously (Noor, Abdullah, and Hashim 2018). Hartling developed a fusion technique to integrate multi-sensor data, demonstrating its potential for object classification under the complex urban environment (Hartling, Sagan, and Maimaitijiang 2021). UAV multispectral images were also applied in water quality monitoring (Y. Xiao et al. 2022) and the detection of asphalt pavement potholes and cracks (Y. Pan et al. 2018).

Another frequently used system is spectral satellite system. With large coverage, huge annotation types and wide spectral resolution, the obtained images have abundant semantic information for analysis, thus being widely utilized in environmental observation such as hydrological and geological applications. However, one critical challenge should be concerned for satellite data when applying on deep learning training models: a large amount of labels on images with large coverage lead to high costs of manual annotations and rather low utilization of satellite images. Hence, researchers resort to diverse label-efficient methods for satellite image segmentation, which would be introduced in the following. For downstream applications, several methods are adopted to improve land use and land cover classification performance using multi-spectral satellite imagery (Kim et al. 2018; Kupidura 2019). In addition, spectral satellite images are applied in marine environmental monitoring, marine pollution detection, ship detection, and so on.

3. Deep learning methods in remote sensing image segmentation

3.1. Challenges of deep learning applied on remote sensing image segmentation

Significant differences exist between remote sensing images and natural images from the aspects of data acquisition, detail information, and application fields. As mentioned above, RS systems generally contain diverse sensing platforms (aviation platforms, space sensing platforms, etc.) and sensors (optical cameras, hyper/multi-spectral cameras, LiDAR, etc.). These sensing systems could collect and record electromagnetic wave information radiated or reflected by objects in the target region. After that, the obtained information could be transformed into visual RS images via electromagnetic wave conversion and recognition. Hence, unlike natural images that collected by optical cameras, RS images usually embody plentiful spectral information from surface, presenting the fundamental material composition data for agriculture, geological exploration, and other applications in the realm of Earth observation. Besides, different from the natural images that generally contain objects in daily life, multiple types of land features (e.g. water bodies, vegetation, buildings) under various environments are reflected in RS images. Consequently, the pre-processing and annotations of RS images are more expensive than natural images with necessary demands for expertise, making the available large-scale labeled datasets of RS scenes much less than that of natural scenes.

Considering the aforementioned difference and unique characteristics of RS images compared with natural images, semantic segmentation methods for respective field are different in the aspects of data processing, feature extraction, and training paradigm. For example, novel modules and structures are designed in RS image segmentation models to make an effective integration of RS data from multiple spectral bands, which are not necessary in the natural image segmentation tasks. Therefore, despite the mature development of image segmentation algorithms in the field of natural images, such advanced DL models could not be directly applied on RS images segmentation tasks and typically require fresh design.

In summary, applying deep learning on remote sensing image segmentation are facing following difficulties and challenges. First, RS images are generally of high resolution with abundant details, including very high-resolution images and the wide spectrum of multi-spectral/hyper-spectral data. In addition, concerning the performance bottleneck of single-modality methods in complex RS scenes, multiple modalities data resources have also been investigated to resolve the limitation of information diversity. Making full use of the increasing multi-modal Earth observation data, multi-modal methods are able to attain further performance increase in semantic segmentation task. As a consequence, exploring the effective fusion methods and data integration to make full use of these data is one of the significant challenges to improve the segmentation performance. Second, different from simple object classification of natural images, the annotation of RS images demands much expertise. With limited labeled samples given in remote sensing image datasets, the performance of supervised learning is restricted to some extent, resulting in a strong impediment on promoting deep learning in conventional remote sensing tasks.

Considering the aforementioned challenges, various architecture designs and label-efficient approaches are proposed for performance development in semantic segmentation task, which will be introduced in detail in the following sections.

3.2. The various architecture design of deep networks

Supervised networks typically take RS images as inputs and perform a series of operations (e.g. convolution, pooling, and local normalization) to transform them into output semantic labels. As a key underpinning of DL-based segmentation algorithms, CNN was initially proposed and gradually promoted with the innovation of shared weights and back-propagation training. As a typical example, LeNet-5 made up of CNN structure was introduced for digits recognition as shown in Figure 5. In general, CNNs contain three types of layers, namely convolutional layers, nonlinear layers, and pooling layers. With all the receptive fields in a layer sharing weights, convolutional layers significantly decrease the parameters comparing with fully-connected neural networks. Inspired by the fully convolutional networks (FCN) (Long, Shelhamer, and Darrell 2015) that first developed for pixel-level segmentation, numerous deep learning-based segmentation methods are proposed with different underlying architectural contributions, leading to general applications on the field of remote sensing. These specially designed architectures are correlated to the model performance and specifically address the challenges of huge details and wide spectrum when segmenting RS images using DL algorithms. Considering the architectures of DL-based segmentation works, we summarize them into three types, namely CNN-based, Transformer-based, and hybrid structure-based models.

Figure 5. Architecture of convolutional neural networks (LeCun et al. 1998).

3.2.1. CNN-based models

One of the popular family of CNN-based models adopts the convolutional encoder–decoder architecture. Encoder–decoder mode contains a two-stage network capturing the deep semantic information of the input by the encoder and predicting the output from the latent-space representation via the decoder. SegNet was a classical encoder–decoder structure with corresponding encoder and decoder having the same spatial size and channel numbers (Badrinarayanan, Kendall, and Cipolla 2017). In addition, the typical convolutional networks were based on the basic idea of spatially sharing weights to reduce the parameters so that improving training performance, generally consisting of convolutional layers, max-pooling layers, and fully-connected (FC) layers. Since Deep Residual Network (ResNet) (K. He et al. 2016) was proposed as a significant event in the evaluation of CNN, the training difficulty of deep CNN models has been successfully resolved, enabling very deep CNN networks to comprehensively exploit the image semantic information.

Ever since then, the innovation of image segmentation networks has mainly focused on the optimization of the encoder and decoder structure for better performance and efficiency. In recent years, deep CNN-based models have achieved great success and wide range applications in the image segmentation tasks of different fields. Some classic CNN-based image segmentation approaches and advanced innovations are summarized in Table 3. Besides, convolution blocks are also adopted to develop sub-networks of multi-modal models. Hong proposed a general multi-modal deep learning framework based on extraction network and five types of fusion networks to integrate the features from hyper-spectral image(HSI), LiDAR image, and SAR image, realizing the utilization of diverse information for better RS image segmentation performance (Hong et al. 2020). Deeplab series (e.g. DeeplabV1 L.-C. Chen et al. 2014, DeeplabV2 L.-C. Chen, Papandreou, Kokkinos et al. 2017, DeeplabV3 L.-C. Chen, Papandreou, Schroff et al. 2017, and DeeplabV3+ L.-C. Chen et al. 2018) are continuous evaluation and extension based on the technique of dilated convolutions, which introduced dilation rate parameter to convolutional operations so that addressing the decreasing resolution problem caused by max-pooling and striding. Atrous spatial pyramid pooling (ASPP) was adopted in DeeplabV2 to attain multi-scale information for robust segmentation. The best DeeplabV3+ model utilized DeepLabv3 framework as encoder and added effective decoder module to optimize segmentation results. The ablation studies of DeeplabV3+ shown that adopting the proposed decoder module could improve the mIOU performance from 77.21% to 78.85%.

Table 3. Classic CNN-based models for image segmentation task.

Categories	Models	Backbones${}^{\ast }$	Data source	References
Classic Encoder–Decoder models	FCN	VGG-16	Urban scene	Long, Shelhamer, and Darrell (2015)
	SegNet	VGG-16	Road/indoor scene	Badrinarayanan, Kendall, and Cipolla (2017)
	EncNet	ResNet-101	Urban scene	H. Zhang et al. (2018)
	HRNet	HRNet-W48	Urban scene	Y. Yuan, Chen, and Wang (2020)
Deeplab series	Deeplab-v1	VGG-16	Urban scene	L.-C. Chen et al. (2014)
	Deeplab-v2	ResNet-101	Urban scene	L.-C. Chen, Papandreou, Kokkinos et al. (2017)
	Deeplab-v3	ResNet-101	Urban scene	L.-C. Chen, Papandreou, Schroff et al. (2017)
	Deeplab-v3+	Xception-71	Urban scene	L.-C. Chen et al. (2018)
UNet series	U-Net	VGG-16	Medical scene	Ronneberger, Fischer, and Brox (2015)
	Dilated Stacked U-Nets	–	Remote sensing scene	Z. Zhang, Liu, and Wang (2018)
	CT-UNet	ResNet-34	Remote sensing scene	Ye et al. (2021)
Multi-scale models	FPN	ResNet-101	Urban scene	Lin et al. (2017)
	PSPNet	ResNet-50	Urban scene	H. Zhao et al. (2017)
	MF-CNN	–	Remote sensing scene	Shao et al. (2019)
	Multiscale progressive networks	–	Remote sensing scene	Hang et al. (2022)
	HighDAN	HRNet-W48	Remote sensing scene	Hong, Zhang, Li, Li, Yao et al. (2023)
Dilated convolutional models	DilatedNet	VGG-16	Urban scene	F. Yu and Koltun (2015)
	LFE module	–	Remote sensing scene	Hamaguchi et al. (2018)
Attention-based models	Non-local Neural Networks	ResNet-50	Urban scene	X. Wang et al. (2018)
	DANet	ResNet-101	Urban scene	Fu et al. (2019)
	CCNet	ResNet-101	Urban scene	Z. Huang et al. (2019)
	AGLN	ResNet-50	Urban scene	J. Li, Zha et al. (2022)
	LANet	ResNet-50	Remote sensing scene	Ding, Tang, and Bruzzone (2020)
	AWNet	ResNet-101	Remote sensing scene	Y. Liu et al. (2021)

${}^{\ast }$Only the main backbone is listed here if various backbones are used in one work.

U-Net (Ronneberger, Fischer, and Brox 2015) and its variants are another popular family in image segmentation methods. The original U-Net network used a symmetric path and skip-connection to capture contexts in a U shape (e.g. first down-sampling when extracting features with convolutions and then up-sampling using deconvolutions), which is proved to be especially suitable for medical image segmentation as the structure simultaneously combines original information of images from different scales. Considering the complicated spatial/spectral information in RS images, U-Net has also inspired the segmentation works in RS and been applied to cope with the abundant data problems in remote sensing analysis. For example, Diakogiannis newly introduced residual connections, atrous convolutions, and pyramid scene parsing pooling, enabling ResUNet to extract semantics of high-resolution aerial images and perform well on the semantic segmentation (Diakogiannis et al. 2020). Other multi-scale model such as HR-Net was also adopted as backbone architecture in RS image segmentation task. Hong proposed a high-resolution domain adaptation network, namely HighDAN for cross-city semantic segmentation (Hong, Zhang, Li, Li, Yao et al. 2023). By using multi-modal high-resolution sub-network, HighDAN could learn the preliminary representations for different RS modalities.

Attention mechanism is brought to computer vision area imitating the human visual system in the form of dynamic weight adjustment process. Specifically, attention-based models could weight features according to the importance of inputs via attention blocks adaptively. SENet pioneered channel attention-based models with the core squeeze-and-excitation (SE) block to capture channel-wise relationships so that improving representation ability (Hu, Shen, and Sun 2018). DANet was a dual attention network designed for scene segmentation (Fu et al. 2019). In particular, the two types of attention modules appended on top of dilated FCN were utilized to model the semantic interdependencies in spatial and channel dimensions respectively. The ablation experiments shown that these attention modules improved performance significantly, exceeding the mIOU baseline by 3.3%. With the development of self-attention in computer vision, it is typically utilized as a spatial attention mechanism to capture global information. Except for CCNet (Z. Huang et al. 2019) and Non-local Block (X. Wang et al. 2018) applying self-attention in CNN-based networks for segmentation tasks, pure attention-based networks have been proved to achieve better results than convolutional neural networks especially on large datasets. Generally, such networks are summarized as transformer-based architectures that will be introduced in the following. In addition, other popular CNN based architectures are also listed in Table 3 for a more comprehensive illustration of image segmentation.

In conclusion, based on the operation of convolution, CNNs could capture the local features of images to deal with the detail information from satellite images. With the applications of advanced convolution modules and pyramid structure, CNN-based models are able to expand the receptive field in an adaptive way, hence realizing multi-scale feature fusion for RS image representation.

3.2.2. Transformer-based models

The local property of CNNs limits its capture of global context and long-range dependencies in the images. In addition, the convolutional filter weights are stationary for all inputs regardless of their properties. Inspired by the breakthrough of Transformer (Vaswani et al. 2017) from natural language processing (NLP), Transformer-based models have become hot topics in the field of computer vision, showing impressive performance on many vision-related tasks, particularly in image classification and segmentation. As a significant success of Transformer-based models applying to image classification, vision transformers (ViTs) (Dosovitskiy et al. 2020) have demonstrated its great potentials in global information modeling.

As shown in Figure 6, instead of using convolutional structures as CNN-based methods did, ViTs adopted self-attention mechanism to effectively capture the global relationships between patches. According to recent works (Naseer et al. 2021; Park and Kim 2022), ViTs were demonstrated with the capability of flexibly adjusting receptive fields and learning effective feature representations (Aleissaee et al. 2023). As a result, ViTs have been widely utilized in different fields with a variety of variants. As two key components of Transformer-based models, self-attention mechanism and multi-head attention are utilized to learn self-alignment and different interactions between embeddings respectively. Considering that inductive biases are also important for image analysis but not included in ViT architecture, Swin (Z. Liu et al. 2021) was proposed to produce hierarchical feature representations. Another popular Transformer backbone for dense prediction tasks is pyramid vision transformer (PVT) architecture (W. Wang et al. 2021). It utilized a progressively shrinking pyramid structure and a spatial-reduction attention layer to make it a unified backbone of both CNN and Transformer for producing multi-scale feature maps. Being expected to deal with the representation challenge of long-range dependencies among various bands, Transformer-based algorithms are adopted in the remote sensing image segmentation especially in the field of hyper-spectral/multi-spectral images (W. Wang et al. 2022). In particular, these methods are suitable for images with abundant channel data and multi-modal data due to their powerful capabilities of capturing temporal and long-distance information. For example, Yao extended conventional ViT with minimal modifications and proposed a novel Transformer-based multi-modal deep learning framework to make full use of various kinds of data (e.g. hyper-spectral data, LiDAR data, and SAR data). Multi-modal attention and token fusion method are specially designed to promote the information exchange across modalities (J. Yao et al. 2023).

Figure 6. The overview of vision transformer's architecture (Dosovitskiy et al. 2020).

A variety of models based on Swin Transformer have been utilized on public datasets and specific scene targets. For example, Efficient Transformer based on Swin transformer backbone was proposed to improve segmentation efficiency and the edge enhancement methods were meanwhile adopted to cope with the inaccurate edge problems (Z. Xu et al. 2021). Wang introduced the Swin Transformer as backbone and proposed a densely connected feature aggregation module (DCFAM) to conduct the relationships among multi-scale semantic features for precise segmentation (L. Wang, Li, Duan et al. 2022). The ablation study on the Potsdam and the Vaihingen dataset shown that the introduction of DCFAM yielded a 4.05% increase and a 3% increase respectively. Besides, a novel decoder was designed to densely connect and aggregate features, hence restoring the resolution for segmentation results prediction. It is notable that when RS image provides limited semantics due to the constraint of its spatial resolution, image super-resolution methods, such as DCNet (Hong, Yao et al. 2023), could be adopted for data quality improvement. Considering the approaches based on Transformer structures for RGB image segmentation and hyper-spectral image segmentation, a performance comparison is conducted in Table 4.

Table 4. Performance comparison of different semantic segmentation methods based on Transformer structures on RS image datasets in terms of overall accuracy (OA).

Methods	Datasets				Times
	Potsdam	Vaihingen	Pavia University	Indian Pines
Efficient-T (Z. Xu et al. 2021)	90.08%	88.41%	–	–	2021
STransFuse (Gao et al. 2021)	86.71%	86.07%	–	–	2021
DC-Swin (L. Wang, Li, Duan et al. 2022)	92.00%	91.63%	–	–	2022
Trans-CNN (C. Zhang et al. 2022)	91.00%	90.40%	–	–	2022
UNetformer (L. Wang, Li, Zhang et al. 2022)	92.00%	91.60%	–	–	2022
HSI-BERT (J. He et al. 2019)	–	–	99.75%	99.56%	2019
SpectralFormer (Hong et al. 2021)	–	–	91.07%	81.76%	2021
SSTN (Zhong et al. 2021)	–	–	98.02%	94.39%	2021
DSS-TRM (B. Liu et al. 2022)	–	–	98.50%	99.43%	2022
HiT (Yang et al. 2022)	–	–	92.00%	87.54%	2022
SSFTT (L. Sun et al. 2022)	–	–	99.21%	97.47%	2022

3.2.3. Hybrid structure-based models

Many research have utilized the merits of CNNs and Transformers as a hybrid structure to improve the segmentation performance, taking advantages of the strong capabilities for information extraction by CNN-based models and the powerful global information modeling by Transformer-based methods. The general frameworks could be categorized into two architectures, including integrating CNN and Transformer by hybrid strategy in a single branch and in dual flow branches. For example, comparing the model structures shown in Figure 7, hyperspectral image transformer (HiT) classification network (see (a)) integrated convolution operations with Transformer structure to capture the spectral discrepancies and spatial contexts in a single branch (Yang et al. 2022). However, it is considered that the abundant band information of HSI could not be sufficiently interpreted by the structure in a single branch. Specifically, local semantics and global interactions captured by one branch are not complementary for achieving accurate classes discrimination. Therefore, on the other hand, two branches structure named Hyper-LGNet (see (b)) utilizing CNN and Transformer models in a dual flow framework was proposed to obtain HSI spatial/spectral feature representation for classification problems (T. Zhang et al. 2022).

Figure 7. Two types of hybrid structure-based models using CNNs and transformers: (a) the framework of HiT integrating CNN and Transformer by hybrid strategy in a single branch (Yang et al. 2022) and (b) the framework of Hyper-LGNet integrating CNN and Transformer in a dual flow framework (T. Zhang et al. 2022).

In addition, STransFuse employed a staged model to extract different level feature representations at various semantic scales, and then fused the information from different stages by an adaptive fusion module (AFM) employing the self-attentive mechanism (Gao et al. 2021). The ablation studies demonstrated that employing AFM for feature map fusion performed 0.24% OA better in the Potsdam dataset than the baseline model. CCTNet combined the local details (e.g. edge and texture) and global context by the novel architecture coupled CNN and Transformer networks (H. Wang et al. 2022). UnetFormer is a scene segmentation model constructed for real-time urban scenarios, which remains top performance on the datasets of Potsdam and Vaihingen (L. Wang, Li, Zhang et al. 2022). By selecting the lightweight ResNet18 as encoder and developing an efficient global–local attention mechanism as Transformer-based decoder, UnetFormer could model both global and local information at the same time. On the contrary, another hybrid network adopted Swin transformer backbone to capture long-range dependencies by selecting a U-shaped decoder that employed ASPP block along with SE block to preserve local details of images (C. Zhang et al. 2022).

3.2.4. Computational complexity in different models

Given that the computational complexity measures the computational resources required for training and inference processes, it is one of the important relevant factors for model structure design. Generally related to the number of learnable parameters, the size of input data and the calculation operations of different layers, the computational complexity varied for different RS image segmentation models and significantly impacting their applicability on downstream tasks. For computational complexity measurement, FLOPs (Floating Point Operations) are a common index, calculating the number of floating-point operations performed during the forward propagation process. Higher FLOPs represent more computational resources required for models. The computational complexity of some typical methods are compared in Table 5 via FLOPs index. Therefore, since different blocks and modules (e.g. CNN blocks and multi-head attention blocks) generally possess different FLOPs, it is of great necessity for a special design to make the design of DL-based RS image segmentation structure perform well with the balance between high precision and lightweight model.

Table 5. Computational complexity of some RS image segmentation models.

Architectures	Methods	FLOPs	Refs.
CNN-based structures	HBCNet	24.15G	Y. Xu and Jiang (2022)
Transformer-based structures	Efficient-T	16.78G	Z. Xu et al. (2021)
Hybrid structure-based structures	EMRT	47.63G	T. Xiao et al. (2023)
	SSNet	10.73G	M. Yao et al. (2024)

3.3. Label-efficient methods for segmentation model

Except for the aforementioned supervised approaches and model innovations, selecting optimal paradigm and training techniques also lead to a crucial impact on feature extraction and subsequent classification accuracy. In fact, though being generally adopted with good performance, the performance of prevailing methods utilizing CNN-based or Transformer-based architectures is sensitive to the quantity and composition of datasets. However, a large amount of per-pixel annotations for segmentation algorithms are expensive and labor-consuming especially for satellite images. Under this circumstance, DL algorithms have been conducted via label-efficient learning process to alleviate the annotation costs in recent years. By exploring the potential of unlabeled and weak-labeled data, such methods are promising techniques to handle label shortage challenge in sensing image segmentation. According to the type of supervision and the usage of annotations, label-efficient methods could be categorized into three common perspectives, namely transfer learning, semi-supervised learning, and self-supervised learning (Bandara, Nair, and Patel 2022; Kolbeinsson and Mikolajczyk 2022; Vincenzi et al. 2021; D. Zhang, Liu, and Shi 2020).

3.3.1. Transfer learning

With a popular paradigm to execute pre-training on standard datasets with massive data and then fine-tune for downstream tasks without abundant annotations, transfer learning could utilize feature representations from existing models that have already been trained. Such a method is capable of decreasing the training time and optimizing the learning efficiency of segmentation models. In the early study of CNNs utilized for remote sensing image classification, Castelluccio has compared the performance among three learning approaches, namely using pre-trained CNNs as feature extractors, fine tuning, and training from scratch (Castelluccio et al. 2015). The results demonstrated that fine-tuning better performs full training strategy when the dataset scale is small. Three common styles of fine-tuning including training all parameters, training only last few layers and training additional fully connected layers of model after loading pre-trained weights.

Therefore, by introducing transfer learning method in remote sensing field, Zhang employed EfficientNet pre-trained on ImageNet as backbone at initial stage, and fine tuned the fully connected layer of the lowest layer at the second stage (D. Zhang, Liu, and Shi 2020). Specifically, to fit in the dimensions of RS image datasets, the original softmax classification layer in EfficientNet was replaced by X-dimensional softmax classification layer. The idea of domain-specific transfer learning was also introduced by Panboonyuen to reuse the weights, performing knowledge transfer between RS datasets with different resolution (Panboonyuen et al. 2019). However, the domain difference (e.g. data properties and image pre-processing) between source and target data is supposed to be considered when adopting transfer learning method, which will have great effects on the performance of supervised training process. Generally, the pre-trained models perform better on similar datasets rather than the different ones.

3.3.2. Semi-supervised learning

Aiming at discussing an efficient way to make full use of a large amount of unlabeled data for model improvement, semi-supervised learning is especially suitable for datasets with limited labels. Following previous works (Y. Chen et al. 2024; Jiao et al. 2022), we divided the most representative semi-supervised image segmentation into two categories, namely self-training and consistency learning. Self-training methods train the segmentation models with labeled data first and then are used for generating pseudo labels of unlabeled data. By incorporating the pseudo labels into dataset for next training process, the self-training is iterative and could improve model continuously. With the framework that learning classifiers on different views of data sources, co-training provides another mechanism of self-training method. On the other hand, despite variations of prevalent methods, the basic principle of consistency learning methods focuses on enforcing models to have consistent outputs for the same sample after different transformations. Classic practices of consistency learning methods include mean teacher (Tarvainen and Valpola 2017), PseudoSeg (Y. Zou et al. 2020), and graph-based regularization (Kipf and Welling 2016).

For remote sensing image segmentation task, conducting dense annotation for each image into various categories is time consuming and costly. By contrast, the acquisition of original images is relatively simple with convenient platforms and sensors, resulting in the low utilization of collected data especially on satellite datasets. Therefore, remote sensing image segmentation models have been further explored combining with the semi-supervised learning paradigm. For example, consistency regularization and average update of pseudo-label were integrated to generate pseudo labels, hence a large amount of pseudo labels and limited strong labels could be sent into RS semantic segmentation models for training (J. Wang et al. 2020). Wang combined the semi-supervised method with an iterative contrastive network, realizing the potential information learning of pseudo RS labels (J.-X. Wang et al. 2022).

3.3.3. Self-supervised learning

Recently, by learning representative features from unlabeled data, self-supervised learning could achieve comparative performance with supervised pre-training on many tasks with less supervision (e.g. manual annotations). The general paradigm of self-supervised learning is illustrated in Figure 8. Training with specially designed self-supervised pretexts, models are expected to capture high-level representations of unlabeled samples for supervised downstream tasks. Hence, such paradigm has become helpful for model training under the condition of unlabeled samples. The most commonly used pretexts mainly include three types, namely reconstructing via generative models, predicting a self-produced label from data augmentation and contrastive learning methods (Y. Wang et al. 2022).

Figure 8. The general paradigm of self-supervised learning and supervised learning: (a) Supervised Pipeline and (b) Self-Supervised Pipeline.

Generative pre-training aims at learning representations of input images via reconstruction or generation procedure, so that enhancing the feature extraction capability of semantic segmentation models. Two prominent generative models are generative adversarial networks (GAN) and autoencoder. Except for these models, recent works have demonstrated the potential of a cutting-edge category of generative models applying to image segmentation pre-training, namely Denoising diffusion probabilistic models (DDPM) (Baranchuk et al. 2021; Graikos et al. 2022). Moreover, the feasibility of directly generating high-quality segmentation maps of input images via DDPM has been demonstrated in different fields of segmentation tasks, including medical images (Wolleb et al. 2022; Wu et al. 2022, 2023) and remote sensing images (Bandara, Nair, and Patel 2022; Kolbeinsson and Mikolajczyk 2022). Different from generative models, predictive self-supervised methods generally rely on the labels produced in different pretext tasks. There are a variety of pretext tasks designed to acquire context information of the input data (e.g. relative position prediction Doersch, Gupta, and Efros 2015, Jigsaw puzzles Noroozi and Favaro 2016, missing patches reconstruction K. He, Chen et al. 2022). However, the quality of pretext tasks in predictive self-supervised learning has a direct relation with the learning performance. To make a more general design, contrastive learning methods are proposed as an effective paradigm to learn high-level representations from unlabeled samples. As a significant milestone, the SimCLR proposed by Chen was a simple framework applying contrastive learning on visual representations (T. Chen et al. 2020). After that, enumerate architectures were proposed to achieve better self-supervised learning performance (Caron et al. 2020, 2021; T. Chen et al. 2020; Grill et al. 2020; K. He et al. 2020).

For remote sensing tasks, autoencoders have been widely adopted to learn representation from various remote sensing data, such as multi-spectral images, hyper-spectral images, and SAR images, while GAN-based methods are few for self-supervised pre-training strategy (Y. Wang et al. 2022). For example, a fully Conv–Deconv network was proposed to learn spectral–spatial features from hyperspectral images via the initial data reproduction (Mou, Ghamisi, and Xiang Zhu 2017). As powerful generative models, DDPMs have been proved to learn useful features from RS images containing high-level semantic information for downstream tasks (N. Chen et al. 2023) and also utilized to generate segmentation mask directly (Kolbeinsson and Mikolajczyk 2022). With the integration of predictive self-supervised methods utilized in remote sensing area, Zhao proposed a multi-task learning framework combining the tasks of rotation recognition and scene classification to enhance the generalization capability of CNN models (Z. Zhao et al. 2020). It is notable that for RS images with abundant spectral data, pretext tasks require specific design so as to relate the spectral bands. For example, Vincenzi exploited the potential of colorization pretext task on satellite imagery. By leveraging the high-dimensional spectral bands to reconstruct the visible colors, meaningful representations could be exploited (Vincenzi et al. 2021). At the same time, contrastive loss is also widely used in remote sensing image segmentation. Hou utilized a two-stage hyper-spectral imagery classification training strategy based on contrastive learning to make the most of unlabeled samples, aiming at alleviating the problem of insufficient label information in hyper-spectral data (Hou et al. 2021).

3.3.4. Foundation models

Vision foundation models have attracted increasing interests in recent years. Such general deep learning models are usually trained on large volumes of data and computing resources. Besides, the great innovations of training paradigms (e.g. the aforementioned self-supervised methods) underscore the great potential of extracting deep semantics from large-scale unlabeled datasets, so that obtaining a powerful foundation model with rich feature representation. It means that, by learning enormous number of images, visual foundation models are expected to capture general patterns and prior knowledge, demonstrating strong generalization ability in multiple tasks and scenarios. Therefore, such foundation models enable customized fine-tuning on the based structures and achieve promising generalization and sustainable improvement on a wide range of downstream applications.

Some notable frameworks have been released in recent years in the realm of natural images segmentation. As a remarkable milestone in visual representation learning, SAM achieved impressive zero-shot performance and also inspired numerous researches to exploit the potential of SAM applying on a variety of downstream tasks (Kirillov et al. 2023). Its design of promptable segmentation tasks has endowed itself with the ability to respond appropriately to any prompt at inference time, leading to unprecedented success of foundation model pre-training mechanism. In addition, SEEM was proposed as a promptable and interactive model for segmenting everything everywhere all at once in an image, enabling diverse prompting tasks and rivaling the limited interaction types of SAM. With a unified encoder that encodes all visual and linguistic hints into a joint representation space, SEEM supported generic segmentation tasks (X. Zou et al. 2023).

For remote sensing images, there are also a variety of foundation models that built on the basis of abundant unlabeled RS data sources and innovative self-supervised paradigm, supporting effective pre-training for various types of segmentation applications by specific stakeholders. For example, considering the domain gap and the limited generalization capability of models between natural and RS scenes, Sun proposed an RS foundation model framework called RingMo trained with the mask image modeling method. Two classical visual Transformer architectures (ViT and Swin Transformer) were taken into account as the based structure for reconstruction tasks, and the learned encoder could be useful for various optical RS downstream tasks. A total of 2M images were utilized by RingMo to learn a more robust feature representation (X. Sun et al. 2023). Focusing on the research gap on spectral data, Hong created a general RS foundation model named SpectralGPT to handle spectral RS images (Hong, Zhang, Li, Li, Li et al. 2023). SpectralGPT was trained on a comprehensive dataset comprising over 1M spectral images from the Sentinel-2 satellite. With the combination of 3D generative pretrained transformer and multi-target reconstruction framework, SpectralGPT could capture spectrally sequential patterns and show its substantial potential in the downstream segmentation task.

4. Extensive applications by remote sensing image segmentation methods

Remote sensing image segmentation techniques have gained increasing attention in various applications recently. With similar algorithms adopted for image segmentation task, follow-up decision-makings are specially conducted in diverse domains for algorithm implementation in downstream tasks. In this section, we selected three main applications: geological applications, precision agriculture, hydrological applications, and one potential application: environmental protection for review. From a high-level perspective, two typical tasks in the geological applications are introduced firstly, namely land use/land cover classification and geological hazard assessment. One of the most interest regarding the applications is precision agriculture due to its wide use and promising performance. Precision agriculture using the combination of DL methods and remote sensing images is raising attention in recent years, with crop area classification and crop disease/pest detection being regarded as two popular downstream tasks. Furthermore, water body mapping and extraction is a crucial study in hydrology, while satellites or airborne systems are especially utilized for large-scale water assessment. Finally, as a significant task being emphasized recently, environmental protection especially carbon dioxide monitoring has been integrated with DL-based segmentation models to realize an efficient monitoring. The success of these four domains and other unmentioned applications has shown forwarding prospects of DL image segmentation methods applying on RS images and the follow-up introduction of these four applications is illustrated in Figure 9.

Figure 9. The tree diagram of the applications of DL-based image segmentation methods.

4.1. Geological applications

The remote sensing technologies and deep learning methods have enabled better understanding of the geological information, which is beneficial for reaching the goal of the Earth's environment protection and land management.

Land use and land cover classification (LULC) task aims at conducting quantitative assessment of the land element variations on the Earth surface, which generally brought by environment changes and the human activities (Digra, Dhir, and Sharma 2022). It has been considered as one of the most efficient ways to explore the land transformation. Since these changes could influence human's behaviors and may lead to unpredictable geological hazard, a timely and precise exploration is of paramount to reflect human society and environment changes (Z. Ma and Mei 2021). Therefore, many modeling approaches have been addressed to cope with this task, including machine learning and deep learning methods, where a diagram is given in Figure 10 to display the general evaluation of typical methods for LULC task.

Figure 10. Diagram displaying the main approaches and general evaluation for LULC task.

The overall framework of LULC task is presented in Figure 11. As the spatial/spectral resolution of remote sensing sensors advancing, various remote sensing platforms have been competent for LULC survey with different scale requirement. For instance, multi-spectral/hyper-spectral images collected by satellites are multi-temporal availability with large spatial and abundant spectral resolution, becoming popular options for large scale land cover mapping. After feature engineering phase that removing redundant information from the input data, different machine learning classifiers are trained with refined data to achieve precise land cover classification. Popular algorithms include support vector machine (SVM) (Jain et al. 2018), random forest (RF) (Shi and Yang 2016), K-means algorithm (F. Zhang et al. 2016), and so on. As researched and summarized previously, SVM and RF performed better on the LULC classification compared to other machine learning techniques (L. Ma et al. 2017; Talukdar et al. 2020). In the experiments conducted in the previous works, RF could obtain the most high-precision LULC map on satellite images among six examined algorithms (Talukdar et al. 2020).

Figure 11. The overall framework of LULC task.

Recently, the improvement of deep learning methods and the expansion of remote sensing datasets enable the well-performed deep learning models to achieve more precise LULC results compared with machine learning methods. Specifically, various works adopt deep learning models to realize pixel-based classification (e.g. road, grass, built-up area) of remote sensing images. Kussul conducted one of the first attempts to design a multilevel DL architecture for land cover classification using multi-temporal multi-source satellite imagery. By building a hierarchy of local features, CNNs were demonstrated with better performance than MLP and RF (Kussul et al. 2017). To extract local and global context information in remote sensing images, Xu proposed a high-resolution context extraction network (HRCNet) (Z. Xu et al. 2020). Considering the lack of labelled datasets in remote sensing images, semi-supervised approaches have also been adopted with a pseudo-label generation and model pre-training methods (L. Li et al. 2023).

At the same time, the great potential of hyper-spectral/multi-spectral images has been exploited by research combining with DL-based methods. These data contain abundant information with two spatial dimensions (the width and height of channels) and one spectral dimension (number of channels) (L. Zhang et al. 2012). In particular, the spatial domain attains the morphological information while the spectral one enable different material distinction corresponding to each pixel on the images. Chen first explored the potentials of deep learning on hyper-spectral feature extraction, using single-layer and stacked multi-layer autoencoders to learn shallow and deep features of hyper-spectral data respectively (Y. Chen et al. 2014). Considering the multi-band nature of hyper-spectral images, DL-based feature selection approaches are utilized. By selecting the relevant attributes of data after removing the redundancy of spectral channels, it is widely used in processing massive high-dimensional remote sensing data. Feng proposed a convolutional neural network based on band independent convolution and hard threshold (J. Feng et al. 2020). It is an end-to-end trainable network combining band selection, feature extraction, and classification. Based on the attention mechanism, He jointly extracted local/global information from both spatial and spectral aspects (K. He, Sun et al. 2022). Besides, a designed evaluation index was combined with information entropy and correlation coefficient to select the appropriate frequency band. However, there are still some problems in recent researches on feature selection for hyper-spectral images. Although the operation of feature selection mapped the original spectrum to a low dimensional space, interference and redundant spectra still affect the mapping process. At the same time, using complex DL algorithm to process will lead to the loss of original forms and patterns, making it difficult to seek physical explanations of images.

Geological hazards assessment is another significant application in geological domain. Geological hazards generally identified as the abnormal activity of geological environment that could result in serious disorder affecting the development of human society. Typical hazards include internal earthquakes, volcanic activity, and related geophysical processes, among which landslides are the most common geological hazards (Z. Ma and Mei 2021). Considering that the early landslide investigation is conducted through visual interpretation via manual expert knowledge and experimental surveys, it is undoubtedly time-consuming and costly. Therefore, automatic methods based on DL algorithms are also desirable, exploring the accurate identification of the landslide location by RS images for practical applications. For example, to extract potential active landslides in Synthetic Aperture Radar Interferometry (InSAR) imagery, Ma designed an end-to-end segmentation network called deep residual shrinkage U-Net (DRs-UNet) (Z. Ma and Mei 2021). The proposed network was also applied to detect landslides of test area located in Zhongxinrong County along Jinsha River. And the experimental results demonstrated the efficiency for its automatic recognition of potential landslide hazards. In addition, regarding the high cost of manual labeling using supervised segmentation methods, Zhou researched on a weakly supervised learning approach (Y. Zhou et al. 2022). With the combination of the class activation maps and cycle generative adversarial network, it could segment the RS landslide images in a precise manner. Furthermore, Jiang presented two methods, namely mask region-based convolutional neural networks and transfer learning Mask R-CNN, to detect and segment new and old landslides along the Sichuan-Tibet Transportation Corridor with the constructed datasets (Jiang et al. 2022). For other types of hazards assessment, Guerrero used visible high-resolution images combined with CNN based models for volcanic monitoring focusing on ash plumes segmentation, so that measuring the height of the plume to understand the magnitude of the explosion (Guerrero Tello et al. 2022). Moreover, based on UAV images, Xiong utilized CNN-based networks for building seismic damage assessment to improve the prediction accuracy for effective emergency after earthquake (Xiong, Li, and Lu 2020).

4.2. Precision agriculture

By optimizing the correct amount of agricultural inputs, precision agriculture (PA) is a critical component of sustainable agricultural systems aiming at improving productivity and increasing crop yields. In recent years, the development of artificial intelligence technologies enable agricultural producers to analyze and predict various factors, so as to make accurate management decision in agricultural production process. Combined with massive data, UAV devices, and remote sensing technology, DL-based models could collect diverse agricultural data, including meteorological data, soil data, and crop growth data. Taking advantage of rich data and powerful sensing capability, a comprehensive analysis could be realized to assist in crop field monitoring, thereby improving production efficiency. At the same time, DL-based models are also capable of crop disease/pest detection, so that agricultural stakeholders could have a timely understanding about the growth status of crops, soil conditions, and potential risks of pests and diseases. In conclusion, a series of applications are commonly conducted in PA, including crop area classification, crop vitality, water stress, and crop disease detection.

Crop area classification is first conducted to obtain a timely and accurate spatial distribution of crop area. Considering the large quantity of crop types with significant changes in agricultural crop lands, land resources monitoring is undoubtedly necessary. Recently, DL-based image semantic segmentation methods are combined with remote sensing images to identify different types of crop, so that achieving the management of land resources in designated areas to ensure the development of agricultural production. For example, the precise identification of crop area images from drone RS systems provides decision-making for drone seeding, crop fertilization, irrigation, and pesticide spraying operations, achieving the goals of reasonable planning for precision agriculture. Kussul proposed a multilevel DL architecture to classify crop types from satellite images, obtaining the accuracy of more than 85% on target crops including wheat, maize, sunflower, soybeans, and sugar beet (Kussul et al. 2017). Wang developed a segmentation approach based on the UNet++ architecture to classify ten categories on Sentinel-2A images with 10 m resolution from 2019 to 2021 (L. Wang, Wang et al. 2022). Transfer learning approach was introduced by Viskovic to alleviate the limited number of high-resolution hyperspectral annotations (Viskovic, Kosovic, and Mastelic 2019).

After realizing the classification of crop types, the indexes about crops that reflect crop vitality and water stress are expected to obtain for further understandings. Specifically, the recent development of PA has led to increasing demands for rapid recognition of crop vitality indexes, serving as powerful technical supports for yield prediction, disaster assessment, and other related works. Besides, an accurate and efficient recognition of crop water stress could maximize the efficiency of agricultural irrigation. By calculating the temperature difference between crop canopy and atmosphere, the Crop Water Stress Index (CWSI) indicates the water stress status of crops effectively. It is of great significance to construct CWSI-based model for a real-time monitoring of crop water conditions so that providing precise guidance of crop irrigation. Combining UAV remote sensing platforms with thermal infrared sensors are popular approaches to obtain regional surface temperature, hence determining regional water conditions and crop water stress.

With a relatively comprehensive understanding about crop area and crop types, the obtained information could support further PA applications. Crop disease and pest damage is one of the most severe challenges for global food security, thus requiring accurate detection to reduce their spread for precision agriculture management. However, great difficulties inevitably appear when realizing plant diseases and pests detection in complex natural environment. For instance, the differences between lesion area and the background are generally tiny. Meanwhile, the scales and characteristics vary in different types of lesion areas. Besides, inevitable noise appears in RS images (J. Liu and Wang 2021), leading to difficult procedures to describe the symptoms of diseases by computers. Therefore, crop disease and pest damage tend to be detected by manual process rather than via mathematical calculation at the previous times. Recently, due to the strong capability of feature extraction, DL-based segmentation methods have been increasingly applied in the field of crop disease monitoring (T. Zhang et al. 2021). With effectively recognizing different characteristics of plant diseases and pests, DL algorithms could directly achieve efficient detection of various diseases on diverse scales as presented in Figure 12 (e.g. wheat stripe rust, wheat yellow rust, tomato disease, corn leaf disease, etc.). Besides, the related research have demonstrated that, by using image recognition technology, accurate disease/pest diagnosis results and advanced prevention could be provided. As an example, DL-based methods are proposed to detect wheat yellow rust disease. Utilizing UAV multi-spectral images, Zhang proposed a real-time yellow rust detection algorithm named Efficient Dual Flow UNet (DF-UNet) to detect different levels of yellow rust, meeting practical requirements for a fast and robust algorithm (T. Zhang et al. 2022). Su exploited aerial visual perception for yellow rust disease monitoring by integrating unmanned aerial vehicle sensing, multispectral imaging, and deep learning U-Net (Su et al. 2021). Abbas trained a DenseNet121 model with transfer learning method for tomato disease detection (Abbas et al. 2021). Based on PSPNet semantic segmentation model, Pan identified wheat yellow rust from unmanned aerial vehicle images and experimentally compared the predicted performance with other machine learning and deep learning methods (Q. Pan et al. 2021).

Figure 12. Some applications of crop diseases recognition in RS images.

4.3. Hydrological applications

Survey data showed that only about 2.5% of the total amount of water on the Earth constitutes the freshwater, and 1.5% of which could be accessed for various biophysical processes (Chawla, Karthikeyan, and Mishra 2020). Therefore, the intensive study and sustainable management of water resources are vitally important for ecologic health. Remote sensing techniques are also combined with advanced DL techniques and utilized for hydrological applications.

Water body mapping and extraction denotes automatically extracting water bodies from remote sensing images, which is a crucial study in hydro-geology for sustainable development of regional ecosystems. With the increasing need of large area monitoring, remote sensing satellites and airborne systems with multi-spectral and hyper-spectral sensors have become popular approaches for water body mapping and extraction due to their well properties. Over a long period of time, research have been conducted about exploiting the potential of remote sensing images utilized for researching the natural resources like water body segmentation. After collecting images with multiple bands, image processing methods are adopted, which could be divided into three categories, including threshold-based methods, index-based methods, and learning-based methods. A simple diagram of these three methods is illustrated in Figure 13.

Figure 13. Comparison diagram of algorithm structure for (a) threshold-based, (b) index-based, and (c) learning-based methods.

Threshold segmentation utilized a threshold in one or more bands to distinguish water body and other objects (Guo et al. 2020). For the band selection, the near infrared (NIR) region has been proved to be the most suitable band in threshold segmentation (Bijeesh and Narasimhamurthy 2020). For example, Bolanos conducted a threshold-based procedure to extract water region from SAR imagery (Bolanos et al. 2016). Comparing with the threshold-based methods, index-based methods developed the extraction technology by calculating the band ratios namely water body indexes instead of using the band pixel information directly. The mostly utilized water body indexes are normalized difference water index (NDWI), modified NDWI (MNDWI), and automated water extraction index (AWEI). Based on these indexes, various research were conducted and improved the classification performance to some extent. Ali detected the urban surface water bodies by using Landsat OLI TIRS and NDWI (Ali et al. 2019). Recently, NDWI was integrated with unsupervised deep learning methods by Li for accurately water extraction (J. Li, Meng et al. 2022).

However, there are some difficulties exist in threshold-based methods and index-based methods as the optimal thresholds vary across regions, restricting the increasing utilization of large-scale multi-spectral images. Therefore, learning-based algorithms are utilized to solve the water extraction problem with the help of remote sensing images. Previous study has demonstrated that, with samples increasing, the performance order of optimal machine learning models could be ranked as support vector machine, neural network, random forest, decision tree, and XGBoost (A. Li et al. 2021). Recently, deep learning methods have also become a research hot topic applied on water extraction and mapping. Popular networks, such as CNN, FCN, U-Net, and variations of CNN-based networks, were adopted to achieve water-body extraction (Basaeed, Bhaskar, and Al-Mualla 2016; W. Feng et al. 2018; L. Li et al. 2019). Long recommended a structure integrated with a CNN and logistic regression classifiers for water body extraction (L. Yu et al. 2017). Isikdogan proposed a fully convolutional neural network named Deep-WaterMap to segment water on Landsat imagery (Isikdogan, Bovik, and Passalacqua 2017). Feng presented an enhanced deep convolutional encoder–decoder network based on Deep U-Net to extract water resources from remote sensing images (W. Feng et al. 2018). In 2020, Wang proposed a novel CNN structure based on DenseNet (G. Huang et al. 2017) to identify water in the region of Poyang Lake (G. Wang et al. 2020). Recently, Xiang adopted a dense pyramid pooling module (DensePPM) to capture global contexts for segmenting water bodies from aerial images. The DensePPM achieved a 5.35% increase in the U-Net-based ablation experiment and reached the state-of-the-art performance compared with previous models (Xiang et al. 2023).

4.4. Environmental protection

Since environmental problem raising up attention around the world, a number of correlational researches (some examples are listed in Figure 14) have been conducted in recent times. Carbon dioxide (CO${}_2$) pollution, one of the critical parts, has caused a wide range of adverse effects on human health, environment conservancy, and increasing global warming trends (Brunekreef and Holgate 2002; Manisalidis et al. 2020; Ramanathan and Feng 2009). And it could lead to unpredictable destroy for the Earth, such as biodiversity reduction, sea level rise and climate change in the ecological environment. Therefore, the changes of atmospheric CO${}_2$ concentration are supposed to be identified and quantified for valid emission monitoring, including its distribution and diffusion mechanisms on spatial and temporal dimensions.

Figure 14. Environmental research using remote sensing images: (a) segmentation of weakly visible environmental microorganism images (Kulwa et al. 2023), (b) estimation of Ground-Level NO${}_2$ Pollution (Scheibenreif, Mommert, and Borth 2022), and (c) Hotspot for atmospheric CO${}_2$ concentrations retrieved from Greenhouse Gases Observing Satellite data (Crivelari-Costa et al. 2023).

Different from some real-time monitoring for environmental protection that mostly conducted on the ground, the monitoring of CO${}_2$ concentration data is difficult to retrieve in time. Consequently, satellite hyper-spectral sensing systems are selected as appropriate methods for carbon emission monitoring by providing a fast access to carbon emissions-related data. For sensors to detect atmospheric CO${}_2$, early research mainly depended on the thermal infrared band that is sensitive to the change of the mid high tropospheric CO${}_2$. On the contrary, given that shortwave infrared band is more sensitive to the changes near the ground, it could be utilized to achieve CO${}_2$ concentration retrieval at the bottom of the atmosphere. Since the greenhouse effect is mainly attributed to CO${}_2$ that contribute about 70% to global warming (S. Zhou et al. 2023), it has come to a conclusion that reducing carbon dioxide is one of the top priorities for environmental protection works.

To better integrate and analyze the remote sensing data, deep learning methods such as neural networks with different structures are adopted for CO${}_2$ monitoring. For example, based on hyper-spectral remote sensing images, Zhang applied a new classification network to combine spatial and spectral information for discriminating carbon dioxide in sensing images, solving the problem that many noises would exist in the results when only using spectral information for classification (L. Zhang, Wang, and An 2020). Traditional remote sensing image methods such as ISODATA and SVM were also utilized for performance comparison. Towards the estimation of another common anthropogenic air pollutant, namely nitrogen dioxide (NO${}_2$) at high resolution with global coverage, Scheibenreif introduced deep learning utilizing multi-band imagery for air pollution estimation (Scheibenreif, Mommert, and Borth 2022). Considering other environmental protection works, DL-based algorithms are also adopted for efficient monitoring. As an example, the identification of proper environmental microorganisms (EMs) and their corresponding physiological characteristics via automatic image processing techniques are proposed to enhance the segmentation of weakly visible (namely transparent and noisy with low contrast) EM images (Kulwa et al. 2023).

5. Conclusion

This paper conducts a comprehensive review on remote sensing image segmentation by deep learning models, mainly concerning three aspects: the remote sensing systems integrated with different platforms, the algorithms proposed with architecture innovation and techniques advancement, and the wide applications in downstream tasks. In particular, observation platforms and sensors plus popular RS image datasets are first presented, with detailed analysis on the UAV-based aviation remote sensing and satellite-based space remote sensing systems. After that, challenges of deep learning applied on RS image segmentation are concluded. Then the classification of DL algorithms is performed according to the evolution order and model characteristics, mainly comprising of architecture designs (e.g. CNN, Transformer and hybrid structures) and advanced techniques except for supervised learning modes (e.g. transfer learning, self-supervised learning, and semi-supervised learning). The analysis and comparison are itemized for each category, leading to a general conclusion that DL algorithms could make full use of both spectral and spatial information of RS images, thus presenting superiority over traditional methods. Besides, by extracting appropriate features for classification or segmentation tasks, DL-based models have turned to be reliable and achieved outstanding accuracy on wide range of applications. Especially in four domains mentioned previously (geological, agricultural, hydrological, and environmental applications). With a remarkable development of remote sensing imaging and AI technologies, great improvement of segmentation performance has been achieved in a precise and efficient manner.

However, significant challenges needed to be addressed for further researches and developments:

(1)	As summarized in our review, the limited number of labeled samples is one of the most bottlenecks in RS image analysis. Therefore, advanced techniques such as transfer learning, generative adversarial networks, and semi-supervised learning could contribute to solving the problem brought by limited labeled training samples. In addition, considering the abundant information lying in RS image data, including multiple bands of multi-spectral/hyper-spectral images and very high resolution images, research and innovations are expected to better interpret and represent high-level semantics in remote sensing images for downstream classification and segmentation tasks.
(2)	Multi-modal remote sensing image segmentation methods utilized multiple data sources obtained from different sensors (e.g. optical images, radar data, infrared images) could be further improved for generalization enhancement and precision improvement. Consequently, to make a more comprehensive utilization of these information, well-designed frameworks and novel fusion approaches are expected for the integration of deep semantics of multiple modalities. Besides, lightweight optimization is also a significant part responding to the computational requirements in practical applications.
(3)	Since increasing interests have been focused on foundation models in the field of visual representation learning, the remote sensing visual foundation models are also expected to have various kinds of applications on different sub-fields of RS scenes. Model structures, training paradigm of large-scale unlabeled data and the ever-growing training resources could be evolved to increase the flexibility of RS image segmentation foundation models for different input data and task requirements.
(4)	For all kinds of RS images, inaccurate edge is a main problem to tackle with, which mainly includes two aspects. On the one hand, regarding the cropping operation in the pre-process of large-scale images, the RS image patches used for model training are added with edges inevitably. The adverse impact of edges is supposed to be reduced for global interaction between different patches and achieve precise segmentation on the overall image. On the other hand, since expanding receptive fields of DL models could increase contextual information for segmentation but decrease detail features at the same time, this contradiction led to inaccurate recognition of edge details. Consequently, researchers could pay more attention to edge enhancement methods for detailed edge segmentation.
(5)	Prior knowledge, generally referring to the domain expertise, previous research and logical rules, is extremely helpful for the learning and optimization of neural networks. Therefore, exploring an effective way to introduce the prior knowledge into DL learning models is promising in RS image segmentation domain.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

No new data were created or analyzed in this study. Data sharing is not applicable to this article.

Additional information

Funding

This work was supported by the Natural Science Foundation of China under Grant 42201386, in part by the Fundamental Research Funds for the Central Universities and the Youth Teacher International Exchange and Growth Program of USTB (QNXM20220033), and Interdisciplinary Research Project for Young Teachers of USTB (Fundamental Research Funds for the Central Universities: FRF-IDRY-22-018).