FMPNet: a fuzzy-embedded multi-scale prototype network for sea-land segmentation of remote sensing images

ABSTRACT

Sea-Land Segmentation (SLS) of remote sensing images is a meaningful task in the remote sensing and computer vision community. Some tricky situations, such as intraclass heterogeneity due to imaging constraints, inherent interclass similarity of sea-land regions and uncertain sea-land boundaries, still are and continues to be the significant challenges in SLS. In this paper, a fuzzy-embedded multi-scale prototype network, named FMPNet, is proposed to target the above challenges of SLS task. We design a dual-branch joint attention feature extraction module (DAFM) for effective feature extraction. Memory bank (MB) is designed to collect multi-scale prototypes, aiming to obtain discriminative feature representations and guide feature selection. In addition, fuzzy connection (FC) unit is embedded in the network structure to mitigate the uncertain sea-land boundaries through 2D Gaussian fuzzy method. Extensive experimental results on a publicly SLS dataset and real region images captured by the Gaofen-1 satellite demonstrate the superior performance of the proposed FMPNet over the other state-of-the-art methods.

KEYWORDS:

• Fuzzy method
• multi-scale prototype
• remote sensing images
• sea land segmentation

Introduction

Sea-land segmentation (SLS) refers to the pixel-level classification that distinguishes between sea and land areas in remote sensing imagery. It holds great significance and finds wide applications in marine resource management (Fonner et al., 2020), natural environment monitoring (X. Li et al., 2018), and coastal change analysis (Sun et al., 2023). With the rapid advancement of aerospace technology and the commercialization of satellites, remote sensing platforms are now capable of capturing clearer images (Kotaridis & Lazaridou, 2021). The complex spectral and spatial textures present in remote sensing images enable more precise feature representation, allowing for a clearer expression of textural features and spatial structural properties of landcovers (K. Hu et al., 2023; X. Li et al., 2023). Remote sensing has become an effective and economical technique for large-scale map imaging. However, the abundance of information in remote sensing images has also increased their complexity, posing a significant challenge in extracting valuable information from them.

Semantic segmentation has emerged as an effective technique for addressing this challenge, as it allows for fine-grained pixel-level classification of features (Hao et al., 2020; Minaee et al., 2021). Recently, convolutional neural network (CNN) has become state-of-the-art technique for image analysis (Alzubaidi et al., 2021; Tian et al., 2022). CNN can provide end-to-end solutions for various visual tasks including instance segmentation (Fang et al., 2023; Y. Liu et al., 2020), semantic segmentation (Garcia-Garcia et al., 2018; X. Liu et al., 2019), object detection (Y. Ji et al., 2021; Wu et al., 2020), and so on. It takes the original image as input and outputs predictions without the need for feature engineering. CNN is given powerful context extraction capabilities by convolution and pooling operations. In the field of remote sensing, CNN overcomes the limitations of traditional target-oriented pixel segmentation methods in terms of accuracy and greatly improves the robustness and automation level of remote sensing image interpretation (Hao et al., 2020; Minaee et al., 2021).

In recent years, researchers have explored various CNN-based networks for SLS of remote sensing images, achieving remarkable results. Initially, most CNN-based methods focused on improving network performance by increasing network depth and width. However, this solution may exacerbate feature differentiation, leading to potential gradient disappearance (K. He et al., 2016). To overcome these challenges, attention mechanism (Oktay et al., 2018) and residual connection (Chaurasia & Culurciello, 2017) have been commonly employed to optimize networks, enhance information dissemination, and model intricate dependencies between input and output feature maps. J. Li et al. (2022) proposed a UNet optimization structure for SLS that combines atlas spatial pyramid pooling (ASPP) and flexible rectified linear unit (FReLU), along with an attention mechanism to improve the perception capability of the network. Aghdami-Nia et al. (2022) refined the UNet architecture by using residual operations in convolutional blocks and Jaccard loss function to improve the accuracy of SLS. Shamsolmoali et al. (2019) combined up-down sampling paths with several densely connected residual blocks aggregating multi-scale contextual information for SLS of remote sensing images. Despite the rapid development of SLS based on previous research, most current methodologies are developed utilizing the principles commonly employed in semantic segmentation approaches, and their pertinence to SLS requires further improvement. We aim to consider the key problems in SLS and design a network to alleviate these problems point-to-point. Specifically, through researches and experiments, we found the following two main challenges in SLS of remote sensing images:

1. The phenomena of intra-class heterogeneity and inter-class similarity are prevalent in SLS. Complex geomorphic information, the variable physical feature of coastlines (as shown in Figure 1(a)), and the instability of remote sensing imaging (e.g. clouds and their shadows obscuring or blurring the remote sensing images in Figure 1(b)) make it possible for a single category to contain variety features, and there is often a similarity of spectral features between different categories, which greatly increases the difficulty of SLS.

   Figure 1. (a) The land building (two yellow boxes on the lower right) and the harbor (yellow box in the middle) have similar spectral features and shapes. (b) Remote sensing images are disturbed by clouds and their shadows. (c) The frequent changes in the shape of the shoal caused by the tide make the shoal a mixed area of sea and land, which exacerbates the uncertainty of the sea-land boundary.

   

2. Uncertainty of sea-land boundaries. From the geological viewpoint, sea-land boundary can be defined as the transition zone between sea and land, usually the coastline where the sea meets the land (Barceló et al., 2023). However, the fine segmentation of the boundary is not easy due to the complexity and diversity of coastlines. The sea-land boundary exhibits unique geographic, morphological, and dynamic characteristics, which further contribute to the uncertainties in sea-land segmentation (SLS) tasks. As shown in Figure 1(c), due to tidal phenomena, the shoal becomes a mixed sea-land area with irregular boundaries, making it difficult to locate the boundary accurately. Moreover, remote sensing images are affected by imaging conditions, resulting in mixed pixels at the sea-land boundary. This phenomenon poses a significant challenge for accurate segmentation of the sea-land boundary.

In this paper, FMPNet is proposed in order to cope with the above challenges in SLS of remote sensing images. In response to the first challenge, we designed DAFM, a dual-branch joint attention feature extraction module. Meanwhile, the MB was designed to store multi-scale prototype features, guiding feature selection based on their discriminative power. This approach enhances the differences between different classes and alleviates the issues caused by intra-class heterogeneity and inter-class similarity. Furthermore, we recognize a new perspective on the SLS task: not all areas of the scene require the same level of attention. In many scenes, it is crucial to focus on weak information, such as boundaries, which are difficult to discriminate but hold significant importance. To improve the network’s perception of boundaries, we employ FC to establish a fuzzy relationship between neighboring pixels of sea-land boundaries. To validate the effectiveness of FMPNet for SLS, we compare it with state-of-the-art methods using publicly available SLSD dataset (Chu et al., 2019) and field remote sensing images from Gaofen-1 satellite. The results demonstrate the superior performance of FMPNet. We hope that the insights presented in this paper will inspire and contribute to future research in the field of SLS.

The main contributions of this paper are summarized as follows:

1. We summarize two intractable difficulties in SLS of remotely sensed imagery: (a) the abundance of intra-class heterogeneity and inter-class similarity in SLS scenes. (b) Uncertainty of sea-land boundaries. To address these difficulties, we propose a fuzzy-embedded multi-scale prototype network called FMPNet.

2. DAFM and MB are designed to guide feature selection through multi-scale prototypes for efficient feature extraction to alleviate segmentation difficulties in SLS due to intra-class heterogeneity and inter-class similarity. FC is designed to establish the fuzzy relation between the pixels at the land and sea boundaries to alleviate the boundary uncertainty problem.

3. Extensive experiments were performed on the SLSD dataset. In addition, the model trained on the SLSD dataset was tested in Gaofen-1 real scenarios outside of the SLSD dataset. Quantitative and qualitative results show that our model outperforms other state-of-the-art methods.

The remaining sections of this paper are organized as follows. Section 2 provides a brief review of related work. Section 3 describes in detail the FMPNet proposed in this paper. Section 4 demonstrates the performance of our method through comparison experiments on two datasets, and proves the effectiveness of each module through ablation experiments. Finally, Section 5 summarizes this paper and presents an outlook for future research.

Related works

Sea-land segmentation of remote sensing image

SLS of remote sensing images is a significant and widely studied field. In the early stages of research, the mainstream method for automatic segmentation was inspired by threshold-based segmentation, where each pixel is compared with a preset threshold. Otsu (1979) presented a pioneering threshold-based work using gray-scale histograms to effectively implement SLS. X. Chen et al. (2014) made enhancements to the segmentation effect and response speed by incorporating both coarse and precise threshold operations. However, these methods exhibit high sensitivity to spectral variations and lack the capability to learn from neighboring range pixels. As a result, they can only yield satisfactory outcomes in scenarios where there are pronounced spectral differences. When there are complex texture distributions and scene variations in remote sensing images, it is difficult to find the optimal threshold to adapt to all segmentation scenes, which leads to misclassification becoming common. Consequently, the drawbacks of threshold-based methods significantly hinder their effectiveness in practical applications.

With the blowout development of CNNs and their exceptional feature extraction capabilities and generalization performance, general-purpose semantic segmentation networks have proven to be effective in various specific tasks, including SLS. Currently, CNN-based methods dominate the state-of-the-art in segmentation. Fully convolutional network (FCN) (Long et al., 2015) is considered to be the first milestone of end-to-end CNN that produces satisfactory segmentation results by combining appearance and semantic information. With the success of FCN in the field of semantic segmentation, there has been a proliferation of innovative approaches based on FCN Badrinarayanan et al. (2017) proposed SegNet, which uses an encode-decoder structure to achieve effective pixel-wise semantic segmentation. UNet (Ronneberger et al., 2015) employs a similar structure where the encoding structure extracts image features and the decoding structure recovers detailed features lost in downsampling. deeplabv3+ (L.-C. Chen et al., 2017) uses atrous convolution (L.-C. Chen et al., 2018) with different dilatation rates to extract features with different receptive fields. SENet (J. Hu et al., 2018) improves network representation by modelling interdependencies between convolutional feature channels. HRNet (Wang et al., 2020) combining adaptive spatial pooling and fine-grained feature-aware operations to enhance network representation of multi-scale features.

Numerous CNN-based methods have also been proposed for SLS. Specifically, H. Lin et al. (2017) proposed a multi-scale FCN for maritime semantic annotation. They categorized the pixels of maritime images into three categories: sea, land, and ship. (R. Li et al. (2018) proposed a convolutional neural network DeepUnet improved over the classical coding and decoding structure to deepen the network depth for SLS, which leads to the problem of information loss due to the structural characteristics of up-down sampling, which affects the segmentation of image details and edges. Similar to Deepunet (Tseng & Sun, 2022), mitigate the interference of climate change on SLS by training a deep network to learn the relationship between neighboring pixels (Cheng et al., 2016). proposed SeNet, which utilizes a local smoothing regularization method to achieve improved spatial consistency results in SLS tasks, along with a multi-task loss. Cui et al. (2020) proposed a scale-adaptive network (SANet), which is designed with adaptive multi-scale feature learning modules instead of traditional convolutional operations for large scene SLS. X. Ji et al. (2023) proposed a two-branch integrated network DBENet for SLS tasks, which proposed a highly efficient integrated attention learning strategy to enhance the correlation between the two branches, thereby facilitating feature fusion and information transfer Heidler et al. (2021) employed a multi-task approach combining semantic segmentation and edge detection to achieve segmentation of the Antarctic coast.

While the aforementioned methods have achieved good results in SLS, the design of these networks is not well-targeted for the SLS task, and there is room for improvement in terms of generalizing to different scenes. The segmentation of sea-land boundaries in complex scenes performs poorly, and even large misclassification within classes occurs.

Fuzzy method for semantic segmentation

Deep learning is a deterministic feature learning model, but it provides limited help in reducing data uncertainty. Remote sensing image data inevitably contains a large amount of noise and unexpected uncertainties. In order to better characterize and control the uncertainty in remote sensing image data, the research of introducing fuzzy methods into remote sensing image segmentation task has attracted the attention of more and more scholars, but it is still at a preliminary stage of exploration. Singh et al. (2018) designed a new adaptive type II fuzzy filter for removing pretzel noise from images, and designed two type II fuzzy principal affiliation functions to determine the thresholds of pixel goodness. H. He et al. (2019) proposed a new fuzzy uncertainty modeling algorithm to characterize the land cover pattern, and proposed an adaptive interval-value based interval type II fuzzy clustering method Lei et al. (2018) proposed a fast fuzzy c-mean clustering algorithm designed to obtain contour-accurate hyper-pixel images for color image segmentation by designing a multiscale morphological gradient reconstruction operation that takes less time while providing better segmentation results (Zhao et al., 2021). embedded fuzzy logic units into CNN to make the segmentation results finer and more reliable Chong et al. (2022) combined fuzzy pattern recognition method and CNN for urban remote sensing image segmentation. Q. Lin et al. (2022) proposed a novel fuzzy based quality control algorithm to quantitatively predict the quality of segmentation results by generating fuzzy sets to describe the captured uncertainty. Ma et al. (2023) combined fuzzy deep learning with conditional random fields for fine segmentation of high-resolution remote sensing images. Qu et al. (2023) introduced fuzzy neighborhood and multi-attention gate modules for remote sensing image segmentation.

Compared with traditional deterministic representations, fuzzy methods provide a new way of modeling that encodes uncertain conceptual semantics in remote sensing image analysis problems to reduce the effects of remote sensing image uncertainty. For the segmentation of sea-land boundaries, fuzzy methods help to obtain feature representations that are more perceptible and comprehensible.

Method

The network architecture of the FMPNet is given in Figure 2. FMPNet utilizes an encoder-decoder structure, with the encoder and encoder marked by blue dashed lines and orange dashed lines, respectively, in Figure 2. Firstly, the input image enters the DAFM, which consists of a dual-branch feature extraction module and a joint attention module in series to achieve efficient feature extraction with multiple receptive fields. Meanwhile, the MB stores multiscale prototypes at the encoding stage, and prototypes with strong discrimination are used to guide feature extraction. Finally, FC constructs fuzzy relationships between neighbors for the features in the encoder, and then concatenates these features with those of the same scale in the decoder to enhance the ability of the network to classify pixels at the category boundaries.

Figure 2. Overview of FMPNet.

Dual-branch joint attention feature extraction module

Considering the specificity of SLS scene and the task characteristics, we propose DAFM inspired by atrous convolution and attention mechanism (X. Li et al., 2019). DAFM adopts a dual-branch structure that contains a dual-branch feature extraction module to extract and fuse features under different receptive fields and a joint attention module to provide the ability to focus on important features in different dimensions. The flow is shown in Figure 3.

Figure 3. Illustration of DAFM, where (a) denotes the dual-branch feature extraction module and (b) denotes the joint attention module.

In the dual-branch feature extraction module, the first branch utilizes two consecutive generalized convolutions to extract fine features, while the other branch employs three convolutions with varying dilation rates to capture features from different receptive fields. To prevent exponential growth in computation, we use 3 × 3 sized convolutional kernels. Additionally, to avoid the raster effect caused by atrous convolution (as shown in Figure 4(a)), we sequentially apply dilation convolutions with dilation rates of 1, 2, and 3 (as shown in Figure 4(b)). This approach expands the receptive field without neglecting any pixels. The dual-branch structure ensures the extraction of fine features as well as a certain degree of contextual information.

Figure 4. (a) Visualization of the range of atrous convolutional feature extraction using three successive atrous convolutions with dilation rate of two produces a raster effect, i.e. a large number of pixels in the receptive field are completely ignored. (b) The visualization range of atrous convolutional feature extraction when the three successive dilation rates are 1, 2, and 3, respectively, with the learning range spreading uniformly from the center to the perimeter.

Subsequently, the feature maps are concatenated at the feature fusion unit and fed into the joint attention module after recovering the dimensions by a 1 × 1 convolution operation. This module consists of two branches of spatial attention and channel attention serially. The feature map first enters the spatial attention branch, which emphasizes the significance of various spatial locations in the input feature maps, highlighting regions with significant influence. It begins by reducing the spatial dimension to a vector through global average pooling. Subsequently, convolution is used to learn the spatial weights for each channel, which are then applied to the original feature map. Channel attention focuses on the importance of different channels in the input feature map, distinguishing which channels are more important for the task at hand and improving the utilization of different channel features.

Multi-scale prototype extraction

Despite the excellent feature extraction capability of DAFM, the features are inevitably neglected or lost to varying degrees. This is mainly due to the intra-class heterogeneity caused by the limitations of the imaging conditions, the inter-class similarity between the land and sea areas, and the inherent limitations of the convolution and up-down sampling operations.

Our design is motivated by the memory mechanisms that arise from contrast learning and meta-learning (X. Liu et al., 2021). In previous memory mechanisms, one common approach is to retain as much information as possible or even store all feature maps, which can be relatively inefficient. On the other hand, another approach retains only a few features, resulting in weaker generalization performance of the feature extractor. Thus, it seems that features learned from different scenes need a filtering process to become a more stable concept.

Continuing the concept of memory mechanism, we designed a multi-scale prototype extraction module based on MB. The prototype features are extracted in the multiscale feature map for guiding feature selection and enter into the MB, which explicitly establishes the relationship between inter-class features and intra-class features. This design avoids a large memory footprint and time consumption while establishing an effective MB.

Overall process of MB is shown in Figure 5. Initially, the prototype features ${\left\{p_i\right\}}_{\mathit{i}=1}^4$ are extracted in the feature maps ${\left\{F_i\right\}}_{\mathit{i}=1}^4$, ${\left\{F_i\right\}}_{\mathit{i}=1}^4$ denote the multi-scale feature maps captured by DAFM during the encoding stage. In the process of extracting $p_i$, we adopt a masking mechanism. Specifically, the similarity between feature $F_i$ and $p_{\mathit{i}-1}$ is computed by einsum operation to generate the similarity map. Then, the positions in the similarity map that exceed the threshold are set to 1, while the remaining positions are set to 0, generating a similarity mask. Note that the threshold is initialized to 0.5 and is a trainable parameter. $F_i$ is multiplied with the similarity mask to generate $p_i$. Note that $F_1$ is directly taken as $p_1$ in this process, and the subsequent $F_i$ is always calculated with the $p_{\mathit{i}-1}$.

Figure 5. Illustration of Memory Bank.

$p_i$ enters into the MB, which maps the current $M_i$ to$\mathit{\hspace{0.17em}}{\left\{F_i\right\}}_{\mathit{i}=1}^4$, completing the first round of prototype extraction and mapping. Subsequently this process is repeated on different scales of feature maps, during which $p_i$ is continuously enters into $M_i$ for the momentum update process of $M_i$. When $\mathit{i}$ = 4, $M_4$ contains prototype features at four scales. This implementation enforces consistency and disparity constraints on class features, effectively alleviating the difficulties caused by inter-class similarity and intra-class heterogeneity.

Fuzzy connection unit

Considering another challenge for SLS, on the one hand, the tidal phenomenon brought about by the rotation of the moon around the earth will have an impact on the morphology and location of the shoal, which increases the difficulty of segmentation of the sea-land boundary. On the other hand, remote sensing imaging conditions introduce mixed pixels, where a single pixel contains multiple categories. This phenomenon hinders the accurate segmentation of sea-land boundaries. To address these issues, we propose FC to reduce the uncertainty associated with sea-land boundaries, as shown in Figure 6. Through the fuzzy method, we quantitatively represent the aforementioned issues and establish fuzzy relationships to enhance the flexibility. This concept is captured by Equation 1:

(1) ${\mathrm{\Omega }}_i=\{\left(\mathit{x},F_{{\mathrm{\Omega }}_i}\left(\mathit{x},\mathit{y}\right)\right)|\mathrm{\forall x}\in \mathcal{R}\}$(1)

Figure 6. Illustration of 2D Gaussian fuzzy method.

where $\mathit{\hspace{0.17em}}{\mathrm{\Omega }}_i\mathit{\hspace{0.17em}}$ represents the fuzzy set of category $\mathit{i}$ on the feature map $\mathcal{R}$, and $F_{{\mathrm{\Omega }}_i}\left(\mathit{x},\mathit{y}\right)$ is the affiliation of pixel $\left(\mathit{x},\mathit{y}\right)$ to the fuzzy set ${\mathrm{\Omega }}_i$. The fuzzy set ${\mathrm{\Omega }}_i$ is determined by the fuzzy function $F_{{\mathrm{\Omega }}_i}$, and the fuzzy set ${\mathrm{\Omega }}_i$ degenerates to an ordinary set when $F_{{\mathrm{\Omega }}_i}=\left\{0,1\right\}$. Inspired by the fuzzy set theory mentioned above, we adopt a 2D Gaussian fuzzy method to establish fuzzy relationships between neighborhood pixels of the sea-land boundary and quantify the uncertainty problem. Firstly, a Gaussian kernel of size 3 × 3 is set up and the relative affiliation of the pixels within the kernel to the center pixel $\left(\mathit{x},\mathit{y}\right)$ is calculated as shown in Equation 2:

(2) $\mathit{f}\left(x_p,y_p\right)=\frac{exp\left(-\frac{{\left(x_p-\mathit{x}\right)}^2+{\left(y_p-\mathit{y}\right)}^2}{2{\sigma }^2}\right)}{2\mathit{\pi }{\sigma }^2},$(2)

where $\left(x_p,y_p\right)$ is the relative position coordinate of pixel $\mathit{p}$ in the Gaussian kernel. $\mathit{\sigma }$ denotes the variance with an initial value of 1. The default is the standard 2D Gaussian distribution. The final feature representation $\mathit{F}$ of $\left(\mathit{x},\mathit{y}\right)$ is calculated by using the pixel itself as well as its neighboring 8 pixels, a total of 9 pixels combined, as in Equation 3:

(3) $\mathit{F}\left(\mathit{x},\mathit{y}\right)=\sum _{\mathit{p}=1}^9{\alpha }_p\mathit{N}\left(\mathit{f}\left(x_p,y_p\right)\right),$(3)

where ${\alpha }_p$ represents the pixel value of neighborhood pixel $\mathit{p}$. N represents the normalization operation that makes the sum of the nine weights equal to one. The affiliation degree $\mathit{f}\left(x_p,y_p\right)$ of $\mathit{p}$ is determined by the distribution position, with the center pixel itself having the largest weight, followed by the four pixels above, below, left and right, and the diagonal pixels having the lowest weight.

Experiments

In order to evaluate the capability of the proposed FMPNet, qualitative and quantitative analyses have been performed using a publicly available SLS dataset and Gaofen-1 satellite data from actual scenes in the Jiaodong Peninsula region of China.

Dataset

SLSD is a public SLS dataset published by (Chu et al., 2019). The data source is from Google Earth and contains 210 remote sensing images, all of which have three bands, are about 1000 × 1000 in size, and have a spatial resolution of 3 ~ 5 m. A training set containing 800 images of 512 × 512 and a test set of 320 images were composed by clipping. Examples of images and labels for the SLSD dataset are shown in Figure 7.

Figure 7. Original images and labels of the SLSD dataset.

The Gaofen-1 satellite dataset contains two remote sensing images of the Jiaodong Peninsula of China, each of which is about 4500 × 4500 in size and has a spatial resolution of 2 ~ 8 m. Our work department is located in the region, which facilitates our field visits and validation.

Experimental settings

In this work, we use a laboratory server for experiments and training. The main hardware and software configurations are shown in Table 1. Based on the trade-off between model performance and scale, the input image size of the model in this experiment is a 512 × 512 × 3 color image, and the output prediction is a single-channel binary map of 512 × 512 × 1. To show the segmentation results in a more graphic way, we refilled the output with colors, blue represents the sea and yellow represents the land.

Table 1. Hardware and software configuration of the experiments.

Hardware and Software	Requirements
CPU	Intel(R) Core(TM) i9-12900K
GPU	NVIDIA GeForce RTX 3090
RAM	64 G
CUDA	11.6
Operating System	Ubuntu 18.04
Language	Python 3.8
Framework	Pytorch 1.14.0

Evaluation metrics

SLS performs pixel-level segmentation of images, so pixel-level evaluation metrics are commonly used to evaluate the performance of SLS methods. We selected six evaluation metrics for evaluation. Overall accuracy (OA) denotes the ratio of correctly categorized pixels to the total number of pixels. Precision denotes the ratio of pixels correctly categorized as sea and land to those categorized as sea and land. Recall denotes the ratio of pixels correctly categorized as sea and land to the original water and land pixels. F1 Score is a composite evaluation that includes both precision and recall. intersection over union (IoU) indicates the intersection ratio of individual category predictions to true labels. Mean intersection over union (mIoU) represents the mean of the intersection ratio between the predicted values and the true labels of the sea and land categories. The evaluation metrics are expressed in the formulas shown in equations (4)-(9) (Long et al., 2015).

(4) $\mathit{OA}=\frac{\mathit{TP}+\mathit{TN}}{\mathit{TP}+\mathit{TN}+\mathit{FP}+\mathit{FN}},$(4)

(5) $\mathit{Precision}=\frac{\mathit{TP}}{\mathit{TP}+\mathit{FP}},$(5)

(6) $\mathit{Recall}=\frac{\mathit{TP}}{\mathit{TP}+\mathit{FN}},$(6)

(7) $F_1\mathit{Score}=\frac{2\cdot \mathit{Precision}\cdot \mathit{Recall}}{\mathit{Precision}+\mathit{Recall}},$(7)

(8) $\mathit{IoU}=\frac{\mathit{TP}}{\mathit{TP}+\mathit{FP}+\mathit{FN}},$(8)

(9) $\mathit{mIoU}=\frac{1}{k}\sum _{\mathit{i}=0}^k\frac{\mathit{TP}}{\mathit{TP}+\mathit{FP}+\mathit{FN}},$(9)

where TP is true positive, TN is true negative, FP is false positive, FN is false negative, k in the mIoU formula denotes the number of categories.

Comparative experiments

To demonstrate the effectiveness of FMPNet, we conducted experimental comparisons with other state-of-the-art methods on the SLSD dataset and the Chinese Jiaodong Peninsula data from Gaofen-1 satellite.

SLSD

In the same experimental setting, we compared FMPNet with 10 advanced segmentation methods. Table 2 presents the results of the quantitative evaluation of the segmentation effectiveness of the different methods on the SLSD dataset. As observed, FMPNet always achieves the best performance in most candidate metrics. In terms of quantitative evaluation, DeepUnet is a strong competitor to FMPNet, with Precision metrics ahead of ours. However, the visualized segmentation results in Figure 8 show that DeepUnet may produce coarser sea-land boundaries with some misclassifications, while the results of FMPNet are closer to the ground truth (GT).

Table 2. Performance comparison of the different methods on the SLSD dataset. All values are reported in percentage (%). Color convention: best, 2nd-best, and 3rd-best.

Method	OA	Precision	Recall	F1-Score	mIoU	IoU per class
						Sea	Land
FCN8s	98.58	98.57	98.88	98.72	97.18	96.86	97.48
UNet	98.22	98.25	98.54	98.40	96.47	96.07	96.58
SegNet	93.94	97.51	91.40	94.36	88.51	87.70	89.32
LinkNet	96.47	98.15	95.18	96.77	93.14	92.53	93.74
PSPNet	98.57	98.32	98.09	98.70	97.16	96.86	97.44
AttentionUNet	97.22	98.27	96.69	97.47	94.55	94.01	95.08
DeepLabv3+	98.65	98.21	99.35	98.79	97.31	96.99	97.61
DeepUNet	98.91	99.05	98.98	99.02	97.83	97.59	98.06
SENet	96.29	98.41	94.85	96.60	92.81	92.19	93.43
FNNN	97.86	97.97	98.17	98.07	95.77	95.31	96.22
MFDAN	95.25	99.16	92.21	95.57	90.89	90.27	91.51
FMPNet(our)	99.09	98.95	99.42	99.18	98.19	97.97	98.37

Figure 8. Visualization of segmentation results of FMPNet with other advanced methods in port and island. (a)Input image. (b)GT. (c)AttentionUNet. (d) FCN. (e)DeepUNet. (f)DeepLabv3+. (g)LinkNet. (h)PSPNet. (i)SegNet. (j)FMPNet(ours).

For more intuitive visual segmentation results, we selected four representative scenes from the SLSD dataset for the SLS task: port, island, city coastline, and seaway. To aid visual differentiation, we assigned the ocean a blue color and the land a yellow color in the visualization results.

Figure 8 showcases the visualization results for the port and island scenes. In the port scene, FMPNet achieves a complete segmentation of the long dike and accurately captures the intricate details within the port. In contrast, other methods for segmentation of port detail textures suffer from a lot of misclassifications. In the second scene, where the spectral representations of the island and nearby ocean are remarkably similar, FMPNet succeeds in achieving a relatively comprehensive segmentation of the island, successfully identifying small target objects like ships. This demonstrates the capability of FMPNet to meet the specific segmentation requirements of this task scene. Overall, FMPNet exhibits superior performance in terms of scene segmentation completeness, which stands out as its most prominent advantage over other methods.

For further testing the capability of FMPNet, we conducted experiments on two large-sized scenes. The segmentation results of the different methods for the city coastline and the seaway are shown in Figure 9. The coastline scene has a great number of detailed features, and the seaway scene contains complex city spectral features, as well as bridges and waterways, and FMPNet achieves satisfactory results in both scenes. From the visualization results, it is easy to find that FMPNet still maintains a leading position in the completeness of segmentation compared with other methods.

Figure 9. Visualization of segmentation results of FMPNet with other advanced methods in city coastline and seaway. (a)Input image. (b)GT. (c)AttentionUNet. (d) FCN. (e) DeepUNet.UNet. (f)DeepLabv3+. (g)LinkNet. (h)PSPNet. (i)SegNet. (j)FMPNet(ours).

Field remote sensing data from Gaofen-1 in Jiaodong, China

In this section, we conduct comparison experiments using remote sensing images of the Jiaodong Peninsula in China, which were captured by the Gaofen-1 satellite. Unlike the previous experiment, we directly test the model trained on the SLSD dataset on these remote sensing images. We specifically selected four challenging scenes from the images, including docks and islands, which are marked by the yellow boxes in Figure 10.

Figure 10. Remote sensing images of the Jiaodong Peninsula in China captured by the Gaofen-1 satellite, with the four areas we utilized for the evaluation marked with yellow boxes in the figure.

The four scenes are located in Qingdao and Yantai, which are very close to our work department. We conducted detailed fieldwork and evaluated the segmentation results of each method qualitatively and quantitatively using a random sample of 10,000 points. As shown in Table 3, FMPNet consistently achieves the best performance among all candidate metrics and achieves a significant advantage in terms of mIoU, which is 6.04% higher than the next best performance. It is worth noting that we achieved satisfactory results by directly applying the model trained solely on the SLSD dataset to this real scene.

Table 3. Performance comparison of the different methods on the Gaofen-1 data. All values are reported in percentage (%). Color convention: best, 2nd-best, and 3rd-best.

Method	OA	Precision	Recall	F1-Score	mIoU	IoU per class
						Sea	Land
FCN8s	93.59	93.05	92.92	92.99	87.88	86.89	88.85
UNet	91.10	84.35	94.55	89.14	83.50	81.88	85.11
SegNet	91.36	93.40	87.28	90.24	83.92	82.22	85.62
LinkNet	93.06	93.03	91.72	92.37	86.94	85.82	88.05
PSPNet	92.32	92.82	90.19	91.49	85.63	84.32	86.93
AttentionUNet	90.01	94.80	82.69	88.33	81.52	79.10	83.92
DeepLabv3+	94.39	92.04	94.05	93.03	89.34	88.68	90.00
DeepUNet	83.92	95.17	68.31	79.53	71.32	66.02	76.61
SENet	92.01	92.37	89.91	91.14	84.12	82.17	85.88
FNNN	93.04	92.36	92.90	92.63	86.95	87.62	86.28
MFDAN	94.22	94.18	88.75	93.74	89.02	88.22	89.81
FMPNet(our)	97.64	96.38	95.75	96.12	95.38	95.19	95.57

The visualization results of Gaofen-1 data for SLS are shown in Figure 11. Among them, Figure 11(g) displays the segmentation visualization results of FMPNet, which achieved satisfactory performance. AttentionUNet and SegNet performed poorly, with large incorrect segmentations. DeepLabv3+ and DeepUNet achieved good results due to their deeper networks and diverse receptive fields. However, they still exhibited some incorrect segmentations at the sea-land boundary. In contrast, FMPNet performed the best on the Gaofen-1 data, achieving complete and detailed segmentation of the sea-land boundaries, including the challenging breeding area. This demonstrates the excellent SLS performance of FMPNet, as well as its ability to adapt to different scenes.

Figure 11. Visualization of segmentation results of FMPNet with other advanced methods in Gaofen-1 data. (a) Input image. (b) AttentionUNet.UNet. (c) FCN. (d)SegNet. (e) DeepLabv3+.Labv3+. (f) DeepUNet.UNet. (g) FMPNet(ours).

Ablation experiments

To evaluate the performance improvement brought by each module in FMPNet, we conducted ablation experiments on the SLSD dataset. We use OA, F1 score and mIoU as evaluation metrics. The quantitative analysis is shown in Table 4. After adding DAFM, MB and FC modules respectively to the baseline AttentionUNet, all candidate metrics show large improvements. The most significant improvement in mIoU is 2.58%, 2.5%, and 2.18%, respectively, which proves the effectiveness of our proposed modules. FMPNet achieved the best results when all three modules were used, with OA, F1 score, and mIoU improving by 1.87%, 1.71%, and 3.64%, respectively, demonstrating the compatibility and robustness of the modules.

Table 4. Results of the ablation experiment.

Method	DAFM	MB	FC	OA(%)	F1-Score(%)	mIoU(%)
Baseline	-	-	-	97.22	97.47	94.55
FMPNet	✓	-	-	98.56	98.71	97.13
FMPNet	-	✓	-	98.53	98.68	97.05
FMPNet	-	-	✓	98.36	98.53	96.73
FMPNet	✓	✓	✓	99.09(+1.87)	99.18(+1.71)	98.19(+3.64)

There is a correlation between attention level and segmentation results. In the SLS task, the visualization of feature maps can help us to understand more clearly which regions the model focuses on and to visualize the contribution of each module. We visualized the impact of ablating each module using Gaofen-1 data, as shown in Figure 12. The brightness of a region indicates how much attention the modules pay to the region, the brighter the region is the more interested the module is in that region. Thus, we can notice how the modules refine different features.

Figure 12. Visualization of features from each module on the Gaofen-1 data.

Figure 12(a) shows the feature visualization of the baseline, where the regions of interest are scattered and the sea-land boundaries are not clear. The feature visualization after adding MB is shown in Figure 12(b), with more regions of interest, the network learns features more uniformly, and the boundaries are clearer than before. Figure 12(c) shows the feature visualization after adding FC, the sea-land boundaries are emphasized and the boundary contours have obviously become clearer, especially the strip boundaries in the harbor, which is crucial for SLS. The visualization after adding DAFM is shown in Figure 12(d), where large contiguous areas are focused on while the boundaries are completely delineated.

The feature visualization of FMPNet is the best, as shown in Figure 12(e), where the land and sea boundaries are clearly delineated and the colors of the land and sea areas are represented evenly respectively, indicating that FMPNet distinguishes them clearly. These feature visualizations in Figure 12 illustrate the role of each module for different regions, which shows that FMPNet can clearly distinguish between sea and land and can effectively complete the SLS task.

Conclusion

In this paper, we propose FMPNet, a fuzzy-embedded multi-scale prototype network, to address the challenges caused by inter-class similarity, intra-class heterogeneity, and uncertainty of sea-land boundaries in SLS. FMPNet combines fuzzy method and multi-scale prototype learning to alleviate these problems. The proposed DAFM effectively combines the advantages of atrous convolution and attention to enhance the feature learning ability of the network, and the MB is designed to collect multi-scale prototypes of each class to extract discriminative feature representations to alleviate the problems posed by inter-class similarity and intra-class heterogeneity. In addition, embedding the proposed FC into the network structure can effectively eliminate the mixed pixels and improve the accuracy of the network in classifying the pixels at the sea-land boundary. Effectively combining fuzzy methods with deep learning provides new insights for SLS. Extensive experiments on a public SLS dataset and real remote sensing data from Gaofen-1 validate the feasibility, effectiveness and generalization of the proposed FMPNet.

Acknowledgments

All authors would sincerely thank the reviewers and editors for their suggestions and opinions for improving this article.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The SLSD dataset is available at https://github.com/lllltdaf2/Sea-land-segmentation-data; the field data for the Jiaodong Peninsula from Gaofen-1 was provided by our work department, and some of them are publicly available at https://github.com/CVFishwgy/Gaofen1_sea_land_data.

Additional information

Funding

This research was funded by the National Natural Science Foundation of China under Grant 62072391 and the Graduate Science and Graduate Innovation Foundation of Yantai University, GIFYTU.