An unsupervised semantic segmentation method that combines the ImSE-Net model with SLICm superpixel optimization

ABSTRACT

In the field of remote sensing, using a large amount of labeled image data to supervise the training of fully convolutional networks for the semantic segmentation of images is expensive. However, using a small amount of labeled data can lead to reduced network performance. This paper proposes an unsupervised semantic segmentation method that combines the ImSE-Net model with SLICm superpixel optimization. First, the ImSE-Net model is used to extract semantic features from the image to obtain rough semantic segmentation results. Then, the SLICm superpixel segmentation algorithm is used to segment the input image into superpixel images. Finally, an unsupervised semantic segmentation model (UGLS) is used to combine high-level abstract semantic features with detailed information on superpixels to obtain edge-optimized semantic segmentation results. Experimental results show that compared with other semantic segmentation algorithms, our method more effectively handles unbalanced areas, such as object boundaries, and achieves better segmentation results, with higher semantic consistency.

KEYWORDS:

• High-spatial-resolution remote sensing images
• semantic segmentation
• ImSE-Net model
• SLICm superpixel optimization model
• unsupervised semantic segmentation model

1. Introduction

With the advancement of Earth observation satellite technology, high-spatial-resolution remote sensing images (HSRSIs) have found applications in various fields, including urban planning (Ahmadi et al. 2010), city environmental modeling (Liu et al. 2019a), ecological assessment (Su et al. 2023), land use change monitoring (Li, Huang, and Gong 2019), and digital city evolution (Huang, Cao, and Li 2020). HSRSIs are characterized by high spatial resolution, clarity, ready availability, and richness of information. They offer detailed insights into ground objects and the relationships between adjacent elements. Segmenting HSRSIs is a crucial step in image interpretation, as the quality of segmentation directly impacts the accuracy of subsequent analysis and processing. Therefore, research on HSRSI segmentation holds both theoretical and practical significance (Baatz and Schape 2000; Kotaridis and Lazaridou 2021; Li et al. 2021; Shao et al. 2022; Wang 2021a; Wang et al. 2022).

HSRSI semantic segmentation aims to assign category labels to individual pixels in an image, achieving pixel-level classification. This process is one of the fundamental and crucial steps in extracting and analyzing image information (Xu et al. 2021; Yuan, Shi, and Gu 2021). Gradually evolving into a pivotal research area within computer vision, HSRSI semantic segmentation has found extensive applications in autonomous driving, scene comprehension, and robot navigation. However, attaining high-accuracy semantic segmentation of high-resolution remote sensing images poses challenges, primarily stemming from the difficulty in extracting effective features owing to the presence of significant intra-class differences and subtle inter-class variations in these images. Additionally, conventional HSRSI semantic segmentation techniques heavily rely on classifiers integrated with manually extracted features, encompassing supervised, semi-supervised and unsupervised semantic segmentation methods.

The supervised and semi-supervised semantic segmentation algorithm necessitates an extensive training dataset that aligns consistently with the spectral texture characteristics of the targeted remote sensing image for classification. Furthermore, manual generation of high-quality training datasets presents challenges of low annotation efficiency and high costs. Hence, in scenarios in which suitable training data is lacking, opting for unsupervised depth classification becomes a reasonable choice. Compared with the supervised method, the unsupervised remote sensing image classification model does not require high-quality training data with real labels, consumes fewer computing resources and less time, and holds higher practical application value. In recent years, unsupervised semantic segmentation technology based on HSRSI deep learning has increasingly become a central research focus for scholars. This approach allows for the automatic extraction of information concerning low-level, intermediate, and high-level semantics of images through a network (Jiang et al. 2016). Subsequently, it incorporates classifier assistance for pixel classification using networks such as the fully convolutional network (FCN), U-Net, SegNet, and the DeepLab series (Chen et al. 2022; Wang et al. 2021b; Wei et al. 2022). The FCN accepts images of varying sizes and generates dense pixel-level predictions by eliminating fully connected layers (Sun and Wang 2018; Yang et al. 2020). Following this, various derivative models have been proposed and effectively applied to diverse semantic segmentation tasks (Alam et al. 2021; Jiang et al. 2020; Ma and Chang 2022).

Owing to limitations in computational resources, existing architectures often employ downsampling operations, such as convolution with a stride of two or more and max-pooling, to achieve a balance between accuracy and efficiency. These operations effectively reduce the size of feature maps while broadening the receptive field, thus reducing the computational complexity of the model and preserving comprehensive semantic information. However, the repeated stacking of pooling or downsampling layers can lead to the loss of detailed object information and the absence of local structures, resulting in subpar segmentation at the edges of objects (Du et al. 2021; Guo, Xu, and Liu 2021; Wang et al. 2022). To address this issue, numerous methods have been proposed to mitigate the loss of local information by replacing strides with dilated convolutions, but this requires adding extra layers (Yu and Koltun 2016). Establishing semantic relationships between adjacent features in HSRSIs is more challenging than in natural images with simpler backgrounds.

Superpixels significantly reduce computational complexity (Ren and Malik 2003). They capture semantic relationships between pixels, including texture and topological relationships between image objects, rather than focusing on individual pixel spectral information. This results in more accurate contour boundaries within local regions. Thus, the use of superpixels is a potential strategy for addressing the issue of inaccurate boundary segmentation and inconsistent prediction of sub-regions within objects, particularly in the context of high-resolution remote sensing imagery. Mainstream superpixel algorithms include simple linear iterative clustering (SLIC) (Achanta et al. 2012), linear spectral clustering (Ren, Zhai, and Sun 2023), and entropy rate superpixel (Wang et al. 2023). Researchers have integrated superpixel techniques into deep neural networks, leading to the development of various remarkable algorithms (Kwak, Hong, and Han 2017). Gadde et al. (2015) demonstrated that implementing superpixel-based downsampling without altering the initial propagation path of the architecture significantly improved model performance. Zhao et al. (2018) proposed combining superpixels with conditional random fields to optimize the semantic segmentation performance of the FCN deep learning model. Furthermore, superpixels are utilized as a post-processing technique. Merging the edges of the superpixels with the pre-segmentation outcome results in enhanced segmentation results (Lei et al. 2022). Utilizing the adjacency relationship between superpixels enhances the abstract degree of features and facilitates the transfer and update of features across multiple layers. This enables a better understanding of the semantic consistency between targets.

In conclusion, semantic image segmentation through supervised network training with a large volume of labeled image data results in high labeling costs, while relying on a limited amount of labeled data leads to reduced network performance. The present study introduces an unsupervised semantic segmentation model that combines ImSE-Net with the SLICm superpixel. The ImSE-Net extracts semantic features from the image and integrates them with the SLICm superpixel segmentation algorithm to obtain a coarse segmentation of the image. This unsupervised model, augmented with high-level abstract semantic features, enhances semantic consistency across patches, improves segmentation accuracy for irregular boundaries, and generates final semantic segmentation results.

The key innovations of this segmentation framework are as follows:

1. Through the integration of the ImSE-Net model with the SLICm superpixel optimization model and the replacement of strided convolution processing with superpixel-based downsampling, finer predicted images are obtained (i.e. improved prediction accuracy of the encoder–decoder architecture).

2. High-level convolutional features provide accurate semantic information, while superpixel edge features provide accurate spatial information; thus, the problems of inaccurate boundary segmentation and inconsistent predictions within objects in HSRSI semantic segmentation with deep learning are addressed, resulting in improved segmentation accuracy.

3. An innovative unsupervised semantic segmentation approach based on HSRSIs is introduced. The approach integrates the latent spatial relationships between global objects (GHPsP) and excavates fine-grained local edge features (LHPsP), aiming to effectively enhance semantic consistency within the global scope.

2. Methods

This paper proposes an unsupervised semantic segmentation model that combines the ImSE-Net model with the SLICm superpixel model (Figure 1). The upper half of the red box represents the ImSE-Net model extracting semantic features of the image to obtain rough clustering results. The lower half of the red box represents the superpixel optimization component, which conducts superpixel optimization processing on the input image, yielding preprocessing semantic segmentation results. The green box symbolizes the utilization of an unsupervised semantic segmentation model that integrates high-level abstract semantic features and detailed superpixel information. This integration aims to achieve high-precision segmentation results with enhanced regional semantic consistency.

Figure 1. Illustration of the workflow for the proposed model. The pre-segmentation phase integrates the ImSE-Net model with SLICm superpixel optimization, generating a preliminary semantic segmentation result denoted by the red dashed line above. The UGLS algorithm for further refinement classification results is indicated by the lower green dashed frame. (a) Original image; (b) segmentation result using ImSE-Net; (c) contour refinement based on SLICm; (d) segmentation result combining ImSE-Net and SLICm; (e) final segmentation result.

2.1. Implicit superpixel embedding network (ImSE-Net)

To effectively utilize the properties of superpixels and reduce the loss of ground information, the ImSE-Net method was adopted (Liu et al. 2019b). This technique employs superpixel-based downsampling and uses sampled pixels as seeds to group pixels. It decodes low-resolution images into high-resolution ones based on the generated superpixels. Consequently, superpixels are specifically utilized to restore lost resolution and preserve detailed ground information. The ImSE-Net method primarily involves data preprocessing and clustering (Figure 2).

Figure 2. Illustration of the workflow for the proposed ImSE-Net method. FCN-32 uses ResNet-101 as the backbone, in combination with the proposed ImSE-Net model from Conv2_x to Conv5_x. This method groups pixels from the downsampling layer to create a allocation matrix assigning predicted target values to each pixel group. Unlike existing neural network methods, this method does not explicitly employ superpixel segmentation for downsampling. Hence, the method can be integrated into existing architectures without altering their feedforward paths.

2.1.1. Preparatory

According to Figure 2, FCN consists of convolutional layers, ReLU activation layers, and downsampling layers, among which the max-pooling layer and the convolutional layer comprise at least two layers (Shelhamer, Long, and Darrell 2017). Assuming ${\mathrm{g}}^{\left(\mathrm{s}\right)}$represents the Conv+ReLU module corresponding to FCN-32 architecture, where s denotes an output stride, and ${\mathrm{d}}^{\left(\mathrm{s}\right)}$represents the downsampling layer. Then, let $d^{\left(s\right)}:{\mathcal{R}}^{{\displaystyle \frac{\mathrm{HW}}{S^2}\times N}}\to {\mathcal{R}}^{{\displaystyle \frac{\mathrm{HW}}{S^2}\times M}}$, with $s<s^{\mathrm{\prime }}$, indicating a reduction in the feature image resolution. The downsampling layer reduces the resolution from $\left({\displaystyle \frac{H}{s},{\displaystyle \frac{W}{s}}}\right)\mathrm{to}\left({\displaystyle \frac{H}{2s},{\displaystyle \frac{W}{2s}}}\right)$.

Therefore, the downsampling layer is defined as $d(\cdot )$, where the feature dimension of the downsampling feature image changes when strided convolution is used for downsampling. The prediction image generated by FCN-32 is (1) $y^{\left(32\right)}={\mathrm{\varnothing }}^{\circ }{g}^{\left(32\right)}\circ d^{\circ }{\dots }^{\circ }{g}^{\left(2\right)}\circ d^{\circ }{g}^{\left(1\right)}\left(X\right),$(1) where $X=\left[x_1,x_2,\dots ,x_n\right]\in {\mathcal{R}}^{H\times W}$denotes the input data, with each image having a height H and width W; $\mathrm{\varnothing }(\cdot )$represents the output feature value of the mapping function, which is the direct mapping from the input image $X$ to $y^{\left(32\right)}$.

2.1.2. Implicit clustering module

Grouping adjacent pixels in the downsampling layer and using all the pixels belonging to the same cluster as a common prediction can significantly reduce the model parameters and computational complexity while effectively preserving the spatial correlation of local pixels and improving the efficiency and accuracy of segmentation.

First, the representation of the j-th cluster with output stride s is given as (2) $c_j^{\left(s\right)}=\left\{i|\mathrm{\forall k},S\left(W^{\left(s\right)}{x}_i^{\left(s\right)},{\tilde{W}}^{\left(2s\right)}{s}_j^{\left(2s\right)}\right)\ge S\left(W^{\left(s\right)}{x}_i^{\left(s\right)},{\tilde{W}}^{\left(2s\right)}{s}_k^{\left(2s\right)}\right)\right\},$(2) where S is cosine similarity, and $s_j^{\left(2s\right)}$ represents the j-th pixel in the downsampled feature image; $s^{\left(2s\right)}=d\left(x^{\left(s\right)}\right)$. Therefore, $W^{\left(s\right)}\in {\mathcal{R}}^{K\times N}$and ${\tilde{W}}^{\left(2s\right)}\in {\mathcal{R}}^{K\times N}$are learnable weight matrices for $x^{\left(s\right)}\in {\mathcal{R}}^{{\displaystyle \frac{\mathrm{HW}}{s^2}\times N}}$ and $s^{\left(2s\right)}\in {\mathcal{R}}^{{\displaystyle \frac{\mathrm{HW}}{4s^2}\times M}}$, respectively, where feature vectors are mapped to a k-dimensional space; k refers to 64 in the experiment . Then, a set of pixels $\left\{C^{\left(s\right)}\right\}$is generated at each resolution.

To effectively insert the clustering module into the downsampling layer, bilinear upsampling is used to decode the prediction back to the original resolution. The cluster to which the i-th pixel belongs can be obtained using $\mathrm{argma}{\mathrm{x}}_{k}S\left(W^{\left(s\right)}{x}_i^{\left(s\right)},{\tilde{W}}^{\left(2s\right)}{s}_k^{\left(2s\right)}\right)$. To enable end-to-end training of the model, a temperature softmax function is applied. Compared with traditional clustering, this model allows for gradual optimization during the training process, making the probability of assigning each sample to different clusters more uniform, thereby preventing overfitting. The formula is as follows: (3) $A^{\left(s\right)\ast }=\arg \underset{A^{\left(s\right)}\in {\left\{0,1\right\}}^{U\times V}}{max}\sum _{\mathrm{ij}}{A}_{\mathrm{ij}}^{\left(s\right)}{S}_{\mathrm{ij}}^{\left(s\right)},s.t.,\sum _j{A}_{\mathrm{ij}}^{\left(s\right)}=1.$(3) Here, $A^{\left(s\right)\ast }$and $S^{\left(s\right)}\in {\mathcal{R}}^{U\times V}$ represent the allocation matrix and the similarity matrix, respectively, and $S_{\mathrm{ij}}^{\left(s\right)}=S\left(W^{\left(s\right)}{x}_i^{\left(s\right)},{\tilde{W}}^{\left(2s\right)}{s}_j^{\left(2s\right)}\right)$ can be obtained.

U and V represent the number of data points in $x^{\left(s\right)}$ and $S^{\left(2s\right)}$, respectively. Then, utilizing the higher-level features learned during the clustering process allows for reconstructing low-resolution downsampled feature images into fine-resolution feature images, as shown below: (4) ${\tilde{x}}^{\left(s\right)}=A^{\left(s\right)\ast }{x}^{\left(2s\right)}.$(4) Therefore, the model predicts the target value$y^{\left(32\right)}=\mathrm{\varnothing }\left(x^{\left(32\right)}\right)$ from the low-resolution feature image, where $y^{\left(32\right)}$ represents the predicted target value. By recursively decoding the low-resolution prediction results, the original high-resolution feature image is obtained as shown below: (5) $y^{\left(1\right)}=\prod _s^{\mathrm{\prime }}=\left\{16,8,4,2,1\right\}{A}^{\left(s^{\hat{\mathrm{\prime }}}\right)\ast }{y}^{\left(32\right)}.$(5) To further improve matrix $A^{\left(s\right)\ast }$ to a distribution matrix ${\tilde{A}}^{\left(s\right)}\in (0,1{)}^{U\times V}$, achieve a differentiable clustering process, and complete end-to-end training, each sampled clustering seed point is defined with a soft allocation matrix ${\tilde{A}}^{\left(s\right)}$ in the clustering process. ${\tilde{A}}^{\left(s\right)}$ contains the degree to which each input feature $x_i^{\left(s\right)}$is assigned to each sampled clustering seed point $s_j^{\left(2s\right)}$, as shown below: (6) ${\tilde{A}}_{\mathrm{ij}}^{\left(s\right)}={\displaystyle \frac{\exp \left(\frac{S\left(W^{\left(s\right)}{x}_i^{\left(s\right)},{\tilde{W}}^{\left(2s\right)}{s}_j^{\left(2s\right)}\right)}{\tau }\right)}{{\sum }_k\exp \left(\frac{S\left(W^{\left(s\right)}{x}_i^{\left(s\right)},{\tilde{W}}^{\left(2s\right)}{s}_k^{\left(2s\right)}\right)}{\tau }\right)},}$(6) where $\tau$ is the temperature parameter that controls the scale of the clustering matrix ${\tilde{A}}^{\left(s\right)}$. When $\tau$ is high, the allocation is close to uniform, which prevents the model from falling into a local optima. When $\tau$ is low, the model assigns higher values to the nearest cluster center, leading to stable model convergence. When $\tau$ tends to 0, ${\tilde{A}}^{\left(s\right)}$ is equal to $A^{\left(s\right)\ast }$. A is set to 0.07 for our experiment, and the specific steps are shown in Table 1 below.

Table 1. Detailed process for the ImSE-Net.

Algorithm 1. ImSE-Net segmentation model
Step 1: Input an image $X=\left[x_1,x_2,\dots ,x_n\right]$and generate a prediction map $y^{\left(32\right)}=\mathrm{\varnothing }\left(x^{\left(32\right)}\right)$;
Step 2: Downsampling based on superpixels in the FCN-32 architecture: Pixels are grouped as superpixel seeds to generate multiple clusters $\left\{C^{\left(s\right)}\right\}$; Determining the cluster $A^{\left(s\right)\ast }=\arg \underset{A^{\left(s\right)}\in {\left\{0,1\right\}}^{U\times V}}{max}\sum _{\mathrm{ij}}{A}_{\mathrm{ij}}^{\left(s\right)}{S}_{\mathrm{ij}}^{\left(s\right)}$, $s.t.,\sum _j{A}_{\mathrm{ij}}^{\left(s\right)}=1$, to which each pixel i belongs;
Step 3: Decoding low-resolution feature images to high-resolution ${\tilde{x}}^{\left(s\right)}=A^{\left(s\right)\ast }{x}^{\left(2s\right)}$;
Step 4: Recursive decoding with bilinear upsampling $y^{\left(1\right)}=\prod _{s^{\widehat{{}^{\mathrm{\prime }}}}}=\left\{16,8,4,2,1\right\}{A}^{\left(s^{\widehat{{}^{\mathrm{\prime }}}}\right)}\ast y^{\left(32\right)}$;
Step 5: Output the ImSE-Net segmentation result Y.

2.2. SLICm and superpixel optimization

Although ImSE-Net restores some detail by adding a decoding module, it still relies on two bilinear interpolation upsamplings to enhance the feature resolution, leading to less-than-ideal semantic segmentation along object edges. Considering that superpixels can protect object edges, this study optimizes semantic segmentation by combining high-level semantic features with superpixel object edge information.

Superpixels typically represent clusters of pixels sharing similar characteristics such as location, color, and texture. In this research, the Simple Linear Iterative Clustering (SLIC) algorithm (Achanta et al. 2012) is employed for superpixel segmentation of the input image. First, it converts the color image to the CIE-LAB color space, where each pixel is linked to a five-dimensional vector composed of color values (L, a, b) and coordinates (x, y). Subsequently, a distance metric is constructed for these vectors, and then iterative local clustering is applied to the image pixels. The specific steps and formula for the calculations are outlined below:

1. Initialization of seed points. It is assumed that there are N pixels in the image, and the SLIC algorithm divides the image into K superpixels with the same size. The size of each superpixel is represented by N/K. The distance between seed points can be expressed as$S=\sqrt{N/K}$. To avoid affecting the subsequent clustering results, the seed points are moved within a $3\times 3$ neighborhood window, while the gradient values of all pixels are calculated. The seed points are moved to the position with the smallest gradient value to prevent the seed points from being assigned to the edge or noise pixels of the image. A label is assigned to each seed point.

2. Similarity measurement. The method involves searching for each pixel, calculating the similarity degree between the pixel and the seed point (including color distance and spatial distance), and iterating continuously until convergence: (7) $d_{\mathrm{lab}}=\sqrt{{\left(l_k-l_i\right)}^2+{\left(a_k-a_i\right)}^2+{\left(b_k-b_i\right)}^2},$(7) (8) $d_{\mathrm{xy}}=\sqrt{{\left(x_k-x_i\right)}^2+{\left(y_k-y_i\right)}^2,}$(8) (9) $D=\sqrt{{\left({\displaystyle \frac{d_{\mathrm{lab}}}{m}}\right)}^2+{\left({\displaystyle \frac{d_{\mathrm{xy}}}{S}}\right)}^2.}$(9)

Here, $\left[l_k,a_k,b_k,x_k,y_k\right]$represents the five-dimensional feature vector of the seed point; $\left[l_i,a_i,b_i,x_i,y_i\right]$ represents the feature vector of the pixel point to be judged; k denotes the seed point; i refers to the pixel point to be searched in the image; $d_{\mathrm{lab}}$ refers to the color similarity between pixel points and $d_{\mathrm{xy}}$ is the spatial distance between pixel points in the neighborhood of the image; m is used to measure the relative weight of color information and spatial information, taking the fixed constant in the experiment; D represents the similarity magnitude between two pixel points: the higher the similarity, the larger the value of D. To speed up the convergence of the algorithm, SLIC searches within the range of 2S×2S.

According to the obtained SLIC superpixel results, the SLICm optimization algorithm is formulated by combining high-level semantic features and superpixel object edge information to enhance the semantic segmentation results. First, the preliminary semantic segmentation result from the ImSE-Net output is obtained. Subsequently, the count of pixels occupied by each semantic category within each superpixel is determined. Finally, the semantic category with the highest total pixel count is selected and allocated to the respective superpixel. Detailed steps are provided in Table 2 below.

Table 2. Detailed process for superpixel refinement.

Algorithm 2. SLICm Superpixel refinement model
Step 1: Input image $I$and ImSE-Net segmentation result Y;
Step 2: Define the N kinds of semantic classifications $C=\left\{C_1,C_2,\dots ,C_N\right\}$, and the image I Segment result based on SLIC $S=\left\{S_1,S_2,\dots ,S_n\right\}$, and there are $m_i$ pixels in the superpixels $S_i$;
Step 3: Find the semantic subclass of $p_i$ in C from Y, $1\le i\le n,1\le j\le m_i,p_i$ denotes the j-th pixels in $m_i.$ If $p_i$refers to $C_k,$let $M_k=$ $M_k+1,$where $M_k$ means the total number of pixels referring to $C_k.$ Until find the $C_{\max }$ which have the most corresponding pixels in the superpixels $S_i$ and assign $C_{\max }$ to this superpixel.
Step 4: Obtain the final superpixel refinement semantic segmentation result $\tilde{Y}$.

2.3. Unsupervised semantic segmentation model UGLS

The unsupervised semantic segmentation model integrates the over-segmented images generated by the model in Section 2.2. The global hidden pseudo-positive samples (GHPsP) aim to diminish the similarity matrix and alleviate data redundancy by considering diverse global semantic information. The local hidden pseudo-positive samples (LHPsP) achieve fine-tuning of semantic similarity in locally unbalanced areas, complete the overall optimization of local and global consistency, and obtain the final image segmentation (Figure 3).

Figure 3. Illustration of the global hidden pseudo-positives and local hidden pseudo-positive selection process. GHPsp: unlabeled image samples $\underset{i}{\overset{\mathrm{\prime }}{x}}$; feature extractor F; task-agnostic reference pool $Q^{\mathrm{ag}}$; anchor features $f_i$; the segmentation head S produces corresponding segmentation features $s_i=S\left(f_i\right)$; an index set of $P_i^{\mathrm{ag}}$ for each i-th anchor feature $f_i$; the momentum segmentation head $S^{\mathrm{\prime }}$ produces corresponding segmentation features$s_i^{\mathrm{\prime }}=S^{\mathrm{\prime }}\left(f_i\right)$; task-specific reference pool $Q^{\mathrm{sg}}$; an index set of $P_i^{\mathrm{sp}}$ by comparing $s_i^{\mathrm{\prime }}$ and $Q^{\mathrm{sg}}$; the projection head Z produces a projection anchor vector $z_i$; GHPsP contrastive loss for i-th patch in unsupervised semantic segmentation with multiple positives $L_{\mathrm{ag}}^{\mathrm{cont}}$ and $L_{\mathrm{sp}}^{\mathrm{cont}}$; LHPsp: an index set $M_i^{\mathrm{nei}}$ that contains the i-th anchor and its neighboring patches; it uses the average attention score value of ${\tilde{T}}_i$ as the threshold for selecting LHPsP $M_i^{\mathrm{loc}}$ among$M_i^{\mathrm{nei}}$ and then takes the above-average portion as the final experiment selection; the calculated gradient (G) for mixed feature $s_i^{\mathrm{mix}}$combines the neighboring positive features $G_i$ with the corresponding attention scores $T_i$ proportionally; the LHPsP objective functions $Z\left(s_i^{\mathrm{mix}}\right)$produces a projected mixed vector $z_i^{\mathrm{mix}}$; the global cost is Lc, and the local loss is L_r.

2.3.1. Global hidden pseudo-positives GHPsP

To uncover hidden positive pseudo-samples, enhance the semantic consistency of pixels, and assist in optimizing the contrastive loss function of unsupervised semantic segmentation, this paper proposes a GHPsP module that mainly includes task-agnostic GHPsP $P_i^{\mathrm{ag}}$ and task-specific GHPsP $P_i^{\mathrm{sp}}$.

Given a mini-batch $\underset{i}{\overset{\hat{\mathrm{\prime }}}{X}}={\left\{{\stackrel{\hat{\mathrm{\prime }}}{x}}_i\right\}}_{i=1}^E$ including E unlabeled image samples, $\underset{i}{\overset{\hat{\mathrm{\prime }}}{x}}$ obtains anchor features $f_i\in {\mathrm{\Re }}^{H\times W\times C}$ by processing them through a feature extractor F. For each image, a random feature $f_i$ is sampled to construct a task-agnostic reference pool $Q^{\mathrm{ag}}=\{q_d{\}}_{d=1}^D$, ensuring the universality of the reference pool and increasing the comparability of the model.

We can calculate the similarity $b_i$ of each pixel feature $f_i$ to $Q^{\mathrm{ag}}$ inside and collect hidden positive examples, using the following formula: (10) $b_i=\underset{q_d\in Q^{\mathrm{ag}}}{max}\mathrm{sim}\left(q_d,f_i\right),$(10)

where D represents D random features from F, and $\mathrm{sim}\left(\cdot ,\cdot \right)$ represents cosine similarity between two vectors. If the similarity between $f_i$ and $f_j$ is greater than $b_i$, then other features in the mini-batch$f_j$ are considered positive samples.

To ensure consistency in training and avoid ambiguity in the relationship between two patch features, a reference pool with distribution awareness is implemented to enable the discovery of globally similar features by considering each anchor point. (11) $P_i^{\mathrm{ag}}=\left\{j|\mathrm{sim}\left(f_i,f_j\right)>b_i\vee \mathrm{sim}(f_i,f_j)>b_j\right\},$(11) where j indicates the index for different patch features in the mini-batch processing.

The randomness of the reference pool $Q^{\mathrm{ag}}$ accounts for the diversity of global semantic information but lacks task specificity. Therefore, we propose sampling the features of the momentum segmentation head $S^{\mathrm{\prime }}$ and constructing additional GHPsP $P_i^{\mathrm{sp}}$ to obtain a more comprehensive and representative set of positive samples, thus enhancing the robustness of the network. According to Eq. (11), (12) $P_i^{\mathrm{sp}}=\left\{j|\mathrm{sim}\left(s_i^{\mathrm{\prime }},s_j^{\mathrm{\prime }}\right)>b_i^{\mathrm{\prime }}\vee \mathrm{sim}(s_i^{\mathrm{\prime }},s_j^{\mathrm{\prime }})>b_j^{\mathrm{\prime }}\right\},$(12) where$s^{\mathrm{\prime }}=S^{\mathrm{\prime }}\left(f\right)$ and task-specific reference pool $Q^{\mathrm{sg}}$ comprises s’. Unlike the fixed samples generated by the pretrained backbone network $Q^{\mathrm{ag}}$, $Q^{\mathrm{sg}}$ is updated synchronously with S’ during the training period.

β% negative sample set for each i-th anchor $H_i$ is randomly selected from the remaining patches in our experiment. Therefore, $H_i^{\mathrm{ag}}$ and $H_i^{\mathrm{sg}}$ correspond to $P_i^{\mathrm{ag}}$ and $P_i^{\mathrm{sg}}$, and the loss function can be defined as follows: (13) $L^{\mathrm{cont}}\left(z_i,P,H\right)=-{\displaystyle \frac{1}{\left|P\right|}\sum _{p\in P}\log {\displaystyle \frac{\exp \left(\mathrm{sim}\right(z_i,z_p\left)/\tau \right)}{{\sum }_{h\in \left(H\cup P\right)}\exp \left(\mathrm{sim}\right(z_i,z_h\left)/\tau \right)},}}$(13) where $z_i=Z\left(S\left(f_i\right)\right)$ represents the projected anchor vector, P represents the positive index set, and H represents the negative index set.

2.3.2. Local hidden pseudo-positives LHPsP

While adjacent pixels might share the same semantic label, the segmentation boundary often exhibits substantial variability in semantic information, posing challenges for achieving semantic consistency within the local area. Inspired by the capacity of vision transformers (ViTs) (Dosovitskiy et al. 2020) in modeling long-range dependencies and global context, we propose leveraging the backbone F of ViT to construct the local matching set LHPsP by selecting regions characterized by high attention scores. The feature vector is used as a weight to propagate the gradient correctly to the LHPsP and achieve semantic similarity fine-tuning of locally imbalanced regions. This approach facilitates comprehensive optimization of both local and global consistency, significantly enhancing the model's segmentation performance.

Given an index set $M_i^{\mathrm{nei}}$ that contains the i-th anchor and its neighboring patches, and the spatial attention score ${\tilde{T}}_i\in {\mathcal{R}}^{\mathrm{HW}}$ obtained from the output of the last self-attention layer of backbone F, our method uses the average value of ${\tilde{T}}_i$ as the threshold for selecting LHPsP $M_i^{\mathrm{loc}}$ among$M_i^{\mathrm{nei}}$: (14) $M_i^{\mathrm{loc}}=\left\{j|j\in M_i^{\mathrm{nei}}\wedge t_j>\mathrm{Avg}\left({\tilde{T}}_i\right)\right\},$(14) where $t_j$ represents the j-th element of ${\tilde{T}}_i$, and $\mathrm{Avg}(\cdot )$ represents the mean function. The neighboring positive features $G_i$ and their corresponding attention scores $T_i$ are obtained using Eq. (14): (15) $G_i=\left\{s_j|j\in M_i^{\mathrm{loc}}\right\},$(15) (16) $T_i=\left\{t_j|j\in M_i^{\mathrm{loc}}\right\},$(16) where $s_j$ is the feature of the j-th patch extracted by the momentum segmentation head S. Mixing the patch features from $G_i$ with the corresponding attention scores from $T_i$ in proportion, we obtain: (17) $s_i^{\mathrm{mix}}={\displaystyle \frac{{\sum }_{j\in M_i^{\mathrm{loc}}}\sigma g_j{t}_j}{\left|M_i^{\mathrm{loc}}\right|},z_i^{\mathrm{mix}}=Z\left(s_i^{\mathrm{mix}}\right),}$(17) where $s_i^{\mathrm{mix}}$ is a mixed patch; $z_i^{\mathrm{mix}}$ is a projected mixed vector; $\sigma$ is a scalar value that measures the attention score and scales the attention scores to an appropriate range to ensure the stability and correctness of gradient propagation; $g_j$ indicates the j-th element of $G_i$.

2.3.3. Loss function

The cost function of this article consists of two parts: global cost and local cost. The global cost is represented by Eq. (18). (18) $L_c=\sum _i\sum _k{\tilde{\mathcal{Q}}}_{\mathrm{ik}}\log {\tilde{\mathcal{Q}}}_{\mathrm{ik}}-{\tilde{\mathcal{Q}}}_{\mathrm{ik}}\log {\tilde{P}}_{\mathrm{ik}}+l\left(P\right),$(18) where $l(\cdot )$ is the regularization term, $\tilde{\mathrm{P}}$ is the probability that the pixels are assigned to seed points, $i$ represents a pixel point, and k represents a seed point. The local loss ${\mathrm{L}}_{\mathrm{r}}$ is a key means for adjusting the weight parameters according to the gradient reconstruction. It encompasses the reconstruction of the color and spatial features of the input image and is defined as (19) $L_r=L_{r_c}+\mathrm{\varnothing }\ast L_{r_s},$(19) where ${\mathrm{L}}_{\mathrm{rc}}$ is the reconstruction loss of color features, ${\mathrm{L}}_{\mathrm{rs}}$ is the local loss of spatial features, and $\mathrm{\varnothing }$ controls the weights of ${\mathrm{L}}_{\mathrm{rc}}^{\mathrm{i}}$and ${\mathrm{L}}_{\mathrm{rs}}^{\mathrm{i}}$. However, as the gradient is reconstructed as a bidirectional gradient, ${\mathrm{L}}_{\mathrm{r}}$ can be written as (20) $L_r=\sum _{i\notin V_b}\left(L_{r_c}^i+\mathrm{\varnothing }\ast L_{r_s}^i\right)+\sum _{i\in V_b}\left(L_{r_c}^i+B_i\ast \mathrm{\varnothing }\ast L_{r_c}^i\right),$(20) where ${\mathrm{V}}_{\mathrm{b}}=\left\{\mathrm{n}|{\mathrm{B}}_{\mathrm{n}}>ϵ\right\}$ is the number of pixel sets, and $\sum _{\mathrm{i}\in {\mathrm{V}}_{\mathrm{b}}}\left({\mathrm{B}}_{\mathrm{i}}\ast \mathrm{\varnothing }{\ast \mathrm{L}}_{{\mathrm{r}}_{\mathrm{s}}}^{\mathrm{i}}\right)$ is a regularization term to avoid over-reliance on the spatial characteristics of pixels in ${\mathrm{V}}_{\mathrm{b}}$ during the clustering process. Finally, the loss functions in the proposed method are (21) $L=L_c+\text{β}\ast L_r,$(21) where $\text{β}$ balances the two loss functions.

The steps of the unsupervised semantic segmentation model are presented in Table 3 below.

Table 3. Detailed process for unsupervised semantic segmentation UGLS.

Algorithm 3. Unsupervised semantic segmentation model
Step 1: Input a small sample set $\stackrel{\widehat{{}^{\mathrm{\prime }}}}{X}={\left\{\underset{i}{\overset{\widehat{{}^{\mathrm{\prime }}}}{X}}\right\}}_{i=1}^E$ containing E unlabeled images;
Step 2: Build a global matching set GHPsP; Collect task-agnostic reference pools ${\mathrm{Q}}^{\mathrm{ag}}$ and ${\mathrm{Q}}^{\mathrm{sg}}$ according to Eq. (11) and Eq. (12), respectively; During training, ensure that $Q^{\mathrm{ag}}$ is frozen, while $Q^{\mathrm{sg}}$ and the momentum segmentation head S’ is synchronously updated;
Step 3: Construct the local matching set LHPsP; Calculate the threshold for local hidden positives $M_i^{\mathrm{loc}}$ among$M_i^{\mathrm{nei}}$ according to Eq. (14); Mix neighboring positive features $G_i$ and corresponding attention scores $T_i$ in proportion to obtain the projected mixed vector$z_i^{\mathrm{mix}}$;
Step 4: Compute the loss function and complete all clustering;
Step 5: Output the final segmentation result.

3. Experiments and discussion

3.1. Datasets

To assess the effectiveness of the proposed method, we selected three widely used HSRSIs categorized into three groups, totaling 12 scenes. These images primarily encompassed the COCO-stuff dataset (Caesar, Uijlings, and Ferrari 2018), Jilin-1 dataset (JL-1), and Beijing-2 dataset (BJ-2). A detailed breakdown of the images is provided in Table 4. The dataset consisted of 12,000 images designated for semantic segmentation. For the experiment, 80% of the data was allocated to the training set, and the remaining 20% served as the validation set. Ground truths corresponding to the original remote sensing images were color-assigned for each cluster, facilitating visual comprehension through color and name matches.

Table 4. Grouping information about datasets.

Grouping	Serial number	Source	Size (pixels)	Resolution (m)
First	I1	COCO-stuff	481 × 321	Unknown
	I2	COCO-stuff	480 × 640	Unknown
	I3	COCO-stuff	480 × 640	Unknown
	I4	COCO-stuff	632 × 491	Unknown
Second	I5	JL-1	513 × 513	0.75
	I6	BJ-2	513 × 513	0.80
	I7	JL-1	513 × 513	0.75
	I8	JL-1	513 × 513	0.75
Third	I9	JL-1	513 × 513	0.75
	I10	BJ-2	513 × 513	0.80
	I11	JL-1	513 × 513	0.75
	I12	JL-1	513 × 513	0.75

The COCO-stuff dataset is widely employed in experimental tasks such as object detection and image semantic segmentation owing to its extensive categories and finely labeled data. We selected nine sub-categories from this dataset, excluding the background category (Table 5).

Table 5. Test set task categories.

Test set	Categories
COCO-stuff	Person, tree, shadow, building, animal, sky, chair, ground, low veg.
JL-1	Low veg., ground, building, road
BJ-2	Water, tree, low veg., ground, building, greenhouse, road

JL-1 originates from the China Center for Resource Satellite Data and Applications (CRESDA). Regarded as China’s inaugural commercial satellite, JL-1 features a high spatial resolution and sensitivity, enabling precise ground imaging. The dataset includes nine sub-categories, excluding the background category (Table 5).

The BJ-2 constellation system originated from China’s national ‘15th Five-Year Plan’ and the ‘863’ program, and it is focused on addressing significant scientific and technological challenges. It is China’s first remote sensing satellite constellation project established through a market-based approach. The project promotes institutional and technological innovation within China’s aerospace industry, facilitating market resource allocation. The dataset comprises nine sub-categories, excluding the background category (Table 5).

3.2. Evaluation accuracy

Qualitative and quantitative methods are used to evaluate the performances of the proposed model and the baseline algorithms. Boundary adhesion, shape heterogeneity, and the segmentation accuracy of different feature categories are compared in the qualitative analysis. In the quantitative analysis, the algorithms are compared in terms of the F1-score, OA, MIoU, ACC, and NMI indicators (Table 6).

Table 6. Quantitative evaluation indicators.

Evaluation index
$\mathrm{Precision}={\displaystyle \frac{p_{\mathrm{ii}}}{{\sum }_{j=1}^k{p}_{\mathrm{ji}}},\mathrm{Recall}={\displaystyle \frac{p_{\mathrm{ii}}}{{\sum }_{i=1}^k{p}_{\mathrm{ij}}},{\begin{array}{c}\\ F\end{array}}_1\mathrm{\_socre}=2\times {\displaystyle \frac{\mathrm{Precision}\ast \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}},}}}$	(22)
$\mathrm{OA}={\displaystyle \frac{{\sum }_{i=1}^k{p}_{\mathrm{ii}}}{{\sum }_{i=1}^k{\sum }_{j=1}^k{p}_{\mathrm{ij}}},}$	(23)
$\mathrm{MIoU}={\displaystyle \frac{1}{k}\sum _{i=1}^k{\displaystyle \frac{p_{\mathrm{ii}}}{{\sum }_{j=1}^k{p}_{\mathrm{ij}}+{\sum }_{j=1}^k{p}_{\mathrm{ji}-}{p}_{\mathrm{ii}}},}}$	(24)
$\mathrm{ACC}={\displaystyle \frac{{\sum }_{i=1}^{N}1\left\{y_i=\max \left(k_i\right)\right\}}{N},}$	(25)
$\mathrm{NMI}={\displaystyle \frac{H\left(C\right)+H\left(Y\right)}{H\left(C,Y\right)}}$	(26)

Here, we compare the model results with the actual data to obtain the confusion matrix $P=\left\{p_{\mathrm{ij}}\right\}\in {\mathbb{N}}^{k\times k}$ of the result;$p_{\mathrm{ii}}$ denotes the number of pixels of category i predicted as category i; $p_{\mathrm{ji}}$ represents the number of pixels belonging to category j predicted as category i; $p_{\mathrm{ij}}$ indicates the number of pixels from class i predicted as class j; k denotes the number of categories. Furthemore, $\mathrm{Pr}$, Re and $k_i$ represent the segmentation accuracy, the boundary recall rate and the clustering assignment of data, respectively (Wang et al. 2021c).

F₁-score is a common evaluation metric applied to semantic segmentation (Hou et al. 2020); OA is the proportion of the number of correctly classified pixels to the total number of pixels (Zhao et al. 2022); MIou refers to the average intersection over union of each category (Zhao et al. 2022); ACC is the accuracy; and NMI is the normalized mutual information, where K is the cluster distribution, Y is the ground truth, and H() is the entropy. When distribution K is similar to Y, the NMI value approaches 1; otherwise, the NMI value is close to 0 (Zhao et al. 2018).

3.3. Analysis of results

Comparison with Baseline Algorithms: K-means+ (Pedregosa et al. 2011), SLIC based on K-means classification (SLIC*) (Achanta et al. 2012), automatic fuzzy clustering framework based on K-means classification (AFCF*) (Lei et al. 2020), sobel adaptive morphological reconstruction and watershed algorithm based on K-means classification (SOAWS*) (Jia et al. 2020), unsupervised semantic segmentation using self-learning superpixel network (SLSP-Net*) (Yang et al. 2022), ImSE-Net+SLICm, unsupervised semantic segmentation using invariance and equivariance in clustering (PiCIE) (Cho et al. 2021), unsupervised semantic segmentation by feature correspondences (STEGO) (Hamilton et al. 2022). Detailed information is provided in Table 7 below, where the first row denotes the method, and the second row lists the author and publication date.

Table 7. Description of the unsupervised semantic segmentation algorithms.

K-means+	SLIC*	AFCF*	SOAWS*	SLSP-Net*	ImSENet + SLICm	PiCIE	STEGO	Ours
Pedregosa et al. 2011	Achanta et al. 2012	Lei et al. 2020	Jia et al. 2020	Yang et al. 2022		Cho et al. 2021	Hamilton et al. 2022

Legend: +Method based on K-means. *Method does not directly learn a classification function and requires further application of the K-means classifier for unsupervised image semantic segmentation.

The experiments were conducted on a Windows 10 platform featuring an Intel(R) Core(TM) i7-10700 2.9 GHz CPU and 16 GB RAM. All algorithms and evaluations were implemented using PyTorch 1.1, Python 3.8, CUDA 11.3, and Matlab 2020a. We initialize the unsupervised semantic segmentation using the ViT models pretrained by ImageNet for fine tuning and employ extended convolution to extract dense image semantic features. Stochastic gradient descent was used for model training, with an initial learning rate set to 0.0005 and a weight decay of 0.1. Additionally, following (Liu, Rabinovich, and Berg 2015), we adopted the ‘poly’ learning rate policy for decay, in which the initial learning rate is multiplied by $1-{\left({\displaystyle \frac{\mathrm{iter}}{\max \mathrm{\_}\mathrm{iter}}}\right)}^{0.9}$ (Chen et al. 2018). Data augmentation involved random scaling of input images with a ratio of 0.5–2 and flipping during training.

3.3.1. Segmentation test results

The recommended methods were employed to perform segmentation on the COCO-stuff test set and our two original image test tasks. Experimental results are grouped and visualized in Figures 4–6, while corresponding quantified results are presented in Tables 8–12.

Figure 4. Example results with the COCO-stuff test set.

Figure 5. Example results on the build test set.

Figure 6. Example results on the mixed region test set.

Table 8. Per-class result on the COCO-stuff test set.

Method	F1-score(%)
	Person	Tree	Low veg.	Ground	Shadow	Building	Animal	Sky	Chair
K-means+	53.23	54.36	53.21	65.32	57.36	57.45	55.36	55.36	57.26
SLIC*	51.36	50.36	51.36	65.36	56.56	58.32	54.36	56.36	60.36
AFCF*	62.35	61.36	60.26	73.21	67.36	59.36	58.36	61.36	66.36
SOAWS*	65.58	63.56	61.36	74.36	69.36	66.34	63.21	67.36	68.53
SLSP-Net*	73.26	72.53	68.63	72.36	72.35	65.36	65.45	67.24	69.36
ImSE-Net+SLICm	73.23	73.26	68.75	71.36	70.88	69.35	65.22	68.02	70.29
PiCIE	70.32	70.38	64.32	70.63	68.36	65.78	65.86	65.36	64.23
STEGO	76.72	76.53	68.36	71.25	69.36	66.36	68.36	65.36	70.32
Ours	78.76	79.36	69.36	75.36	76.59	70.56	69.54	70.16	74.33

Table 9. Evaluation scores (%) of different baseline methods based on the COCO-stuff test set.

Method	OA	MIoU	ACC	NMI
K-means+	60.35	20.59	63.69	65.89
SLIC*	60.39	21.77	68.56	67.36
AFCF*	65.32	23.23	74.56	73.26
SOAWS*	70.36	25.12	73.86	74.36
SLSP-Net*	75.53	28.33	76.36	76.31
ImSE-Net+SLICm	75.36	32.56	75.21	75.16
PiCIE	74.18	28.42	68.36	72.25
STEGO	75.36	33.53	75.36	76.36
Ours	77.59	37.23	77.89	78.21

Table 10. Comparison between baseline and state-of-the-art methods for experiment 2. The best values are highlighted in bold.

Method	F1-score				OA	MIoU	ACC	NMI
	Low veg.	Ground	Building	Road
K-means+	58.36	60.36	58.35	57.68	59.87	19.82	59.36	60.36
SLIC*	59.32	61.25	59.36	59.89	57.36	20.56	57.36	66.47
AFCF*	63.36	62.58	67.36	64.21	64.36	22.32	69.56	71.56
SOAWS*	63.83	64.32	65.12	65.39	65.36	24.31	71.69	69.88
SLSP-Net*	71.36	70.58	70.32	70.36	71.56	27.83	72.59	74.36
ImSE-Net+SLICm	71.86	69.89	70.36	71.33	70.98	31.42	74.35	73.26
PiCIE	59.23	61.23	60.12	65.54	66.65	28.33	67.58	68.36
STEGO	72.36	71.36	71.25	68.98	74.83	30.23	74.36	71.36
Ours	75.85	71.36	72.36	72.65	75.38	35.26	75.47	75.58

Table 11. Per-class result for experiment 3.

Method	F1-score
	Water	Tree	Low veg.	Ground	Building	Green-house	Road
K-means+	58.83	57.36	57.32	55.36	53.25	51.99	52.65
SLIC*	61.36	58.36	59.12	60.32	57.56	58.69	60.36
AFCF*	62.53	65.89	67.36	65.32	59.88	73.32	65.99
SOAWS*	65.36	66.36	67.56	73.32	62.56	69.84	63.84
SLSP-Net*	67.59	65.32	68.36	70.36	63.32	71.36	64.25
ImSE-Net+SLICm	68.56	67.23	67.53	69.66	63.22	75.36	63.25
PiCIE	58.36	58.36	60.32	61.24	58.69	58.84	60.36
STEGO	70.18	65.69	68.34	71.36	66.59	68.34	67.68
Ours	73.32	68.36	70.35	72.26	68.26	71.32	70.55

Table 12. Comparison between baseline and state-of-the-art methods for experiment 3.

Method	OA	MIoU	ACC	NMI
K-means+	55.98	19.32	56.99	58.63
SLIC*	55.36	20.19	64.86	59.63
AFCF*	61.36	20.23	65.32	67.36
SOAWS*	64.21	22.98	68.87	65.89
SLSP-Net*	71.36	23.23	67.36	66.79
ImSE-Net+SLICm	70.36	26.63	69.11	66.89
PiCIE	59.32	23.36	65.89	65.36
STEGO	71.21	28.23	69.86	68.32
Ours	71.63	32.53	71.89	70.38

3.3.1.1. Test results for experiment 1

1. Qualitative Evaluation

Figure 4 displays qualitative segmentation examples of the first task set achieved by our method, comparing it with state-of-the-art competitors. The images show representative scenes from natural landscapes based on COCO-stuff, featuring simple object types but exhibiting diverse and rich content. The clear color and texture of the terrain make it susceptible to confusion with surrounding objects.

Considering the segmentation results for Figure 4. I1 and Figure 4. I2, K-means+ and SLIC* tend to overly emphasize intricate details and noise from the original images, resulting in inadequate boundary adhesion, as illustrated in Figure 4. Conversely, SOAWS*, AFCF*, SLSP-Net*, ImSE-Net+SLICm, PiCIE, and STEGO exhibit superior performance in terms of the shape and integrity of segmentation. However, these methods encounter challenges in addressing over-segmentation, leading to further fragmentation in certain regions. Our proposed method shows superior performance in preserving object boundary information, particularly in separating small blocks, such as persons, chairs, and grass. It consistently achieves accurate segmentation of larger objects such as buildings.

According to Figure 4. I3 and Figure 4. I4, K-means+, SLIC*, SOAWS*, and AFCF* methods generate segmented regions with varied sizes and shapes. Although these methods align the segmentation boundaries well with the main buildings, the overall segmentation accuracy remains comparatively inadequate. Additionally, SOAWS*, AFCF*, SLSP-Net*, ImSE-Net+SLICm, PiCIE, and STEGO outperform other methods in terms of time efficiency. However, they suffer from severe under-segmentation, irregular rectangular shapes, and, in some instances, incorrectly merged fuzzy boundaries. Our proposed method significantly enhances the model’s ability to make explicit predictions in object classification and performs exceptionally well in capturing boundary details. For instance, in Figure 4. I3, the segmentation distinctly separates the sky and ocean, displaying high completeness and evident boundaries between categories, effectively overcoming the effects of spectral similarity. Although there are instances of misclassification, such as trees being mistakenly grouped with buildings, our method provides the closest segmentation to the actual results overall.

(2)	Quantitative Evaluation

The image segmentation results from experiment 1 were accurately evaluated using metrics such as F1-score, OA, MIoU, ACC, and NMI (Tables 8 and 9). The best values are highlighted in bold.

According to Tables 8 and 9, K-means+ and SLIC* result in severe over-segmentation and exhibit relatively low evaluation metrics. SOAWS* and AFCF* incorrectly merge building and vegetative regions, particularly with minimal improvement in excessive segmentation of vegetation. SLSP-Net*, ImSE-Net+SLICm, PiCIE, and STEGO present numerous noisy points in building segmentation, with noticeable deviations at terrain segmentation boundaries. The proposed method outperforms the other algorithms in terms of the experimental metrics and demonstrates superiority in terrain segmentation.

3.3.1.2. Test results for experiment 2

1. Qualitative Evaluation

Figure 5 displays sample images, corresponding label maps, and qualitative classification results of the second task set obtained by our method, juxtaposed with state-of-the-art competitors. This dataset, sourced from Jl-1 and BJ-1, represents a densely built-up area with low intra-class variance. The rich roof color and texture structure pose challenges in differentiating them from the surrounding terrain, thereby increasing segmentation difficulty.

According to the images in Figure 5. I5 and Figure 5. I6, a large number of noisy points are generated when K-means+, SLIC*, and SOAWS* are used to segment vegetation and buildings, resulting in a deterioration of the overall segmentation effect. AFCF*, SLSP-Net*, PiCIE, ImSE-Net +SLICm and STEGO fragment the vegetation and road regions, displaying irregular rectangles in segmentation, with some regions showing broken segments. Moreover, some dense areas are incorrectly merged. Our method demonstrates improvements in addressing over-segmentation, resulting in more consistent sizes and shapes of segmented objects with the reference image. However, some mis-segmentation persists, notably in certain areas such as roads and vegetation. Considering the segmentation results for Figure 5. I7 and Figure 5. I8, K-means+ and SLIC* exhibit strong adherence to terrain but suffer from severe over-segmentation. Owing to spectral similarities between building boundaries and roads, SOAWS*, AFCF*, SLSP-Net*, and ImSE-Net+SLICm show evident mis-segmentation, leading to reduced accuracy. PiCIE and STEGO display significant under-segmentation and excessive merging of superpixel regions. Although the proposed method encounters errors in merging, it improves segmentation fragmentation and aligns more consistently with the visual segmentation effect.

(2)	Quantitative Evaluation

The best values are in bold. The results of image segmentation in experiment 2 were evaluated in terms of F1-score, OA, MIoU, ACC, and NMI, as shown in Table 10.

Compared with other algorithms, the proposed algorithm exhibits superior segmentation performance for detailed areas such as object edges owing to its integration of superpixel information (Table 10). Overall, our method exhibits superior performance across multiple indicators (OA = 75.38, MIoU = 35.26, ACC = 75.47, NMI = 75.58), indicating that the improvement was brought by the proposed hidden positive pseudo-samples based on the unsupervised semantic segmentation model.

3.3.1.3. Test results for experiment 3

1. Qualitative Evaluation

As shown in Figure 6, the third dataset is a mixed land area, with more chaotic spatial relationships and complex adjacent relationships among objects than the datasets used in experiments 1 and 2. The sizes and scales of objects are also diverse, making segmentation more difficult.

According to the images in Figure 6. I9 and Figure 6. I10, the K-means+, SLIC*, SOAWS*, AFCF*, SLSP-Net*, ImSE-Net+SLICm, PiCIE, and STEGO algorithms successfully segment water and tree areas, but the segmentation results appear fragmented and irregular, with some areas being notably more fragmented than others. Additionally, there are instances of incorrect merging in dense areas. Our method addresses over-segmentation, with sizes and shapes of segmented objects that are more consistent with the reference image. However, some errors in object segmentation persist in locations such as water and plants.

Figure 6. I11 and Figure 6. I12 present diverse mixed land areas encompassing water, low vegetation, farmland, bare ground, roads, and buildings, forming a more complex scene. Our method outperforms K-means+, SLIC*, SOAWS*, AFCF*, PiCIE, SLSP-Net*, ImSE-Net+SLICm, and STEGO algorithms in separating buildings from adjacent water fields and preserving the integrity of greenhouse areas. However, our method exhibits sensitivity to spectral features of plants, resulting in a few misplaced small segments and misclassification of farmland as vegetation.

(2)	Quantitative Evaluation

Comparison of five indicators (F1-score, OA, MIoU, ACC, and NMI) for experiment 3. The best values are in bold, as shown in Tables 11 and 12.

As observed in Tables 11 and 12, when there is a simultaneous increase in map width, type of ground object, spatial distribution, image resolution, and contour complexity, both the K-means+ and SLIC* segmentation methods show relatively low segmentation accuracy. The segmentation results of SOAWS*, AFCF*, SLSP-Net*, ImSE-Net+SLICm, PiCIE, and STEGO indicate that while the accuracy of these algorithms is generally impacted, the convergence and stability differ considerably owing to the distinct principles underlying each algorithm. Notably, the deep fuzzy clustering models such as ImSE-Net+SLICm, STEGO, and our model outperform the shallow models, highlighting the advantage of leveraging deep learning techniques for acquiring clustering-friendly representations. Moreover, our proposed model shows improvements in OA, MIoU, ACC, and NMI values by 0.42%, 4.30%, 2.03%, and 2.06%, respectively, compared with STEGO. These results emphasize that uncovering hidden pseudo-samples enhances the semantic consistency of pixels, thereby enhancing the clustering performance of deep fuzzy models.

In summary, the evaluation indicators across the three sets of experimental image collections decrease as the difficulty of high-resolution remote sensing image segmentation increases. Nevertheless, the proposed method consistently demonstrates a higher evaluation value than the other algorithms. According to this analysis, the visual effect of the segmented image is consistent with the intuitive conclusions drawn from the experimental indicators, and the proposed method achieves the highest segmentation accuracy among the tested methods.

We have demonstrated the capability of our approach to effectively manage unbalanced areas, specifically object boundaries with significant variability in semantic information, yielding more accurate contours and superior segmentation outcomes marked by heightened semantic consistency. However, several aspects within our proposed framework remain unexplored and underdeveloped. Although our work enhances the prediction accuracy of encoder–decoder architectures by integrating the proposed new approach and replacing downsampling in the encoder with our superpixel-based downsampling to group pixels, we still face challenges in eliminating errors in local semantic information. These errors become more pronounced when the source and target domains have different data imaging modes. Additionally, while our method performs segmentation on the COCO-stuff test set and our two test set tasks with original images, the effectiveness of our approach on other datasets remains a question. In the follow-up research, we aim to expand the data sources and scale to further support the investigation of semantic segmentation model structures for remote sensing images. Furthermore, we plan to incorporate data into the model coding layer, leveraging prior knowledge to enhance the model’s feature learning ability and ultimately improve recognition accuracy.

3.3.1.4. Ablation study

In the ablation study, inspired by the accomplishments of the data decomposition model (Feng et al. 2020), we considered three test aspects to evaluate our approach:

1. Effectiveness of embedding clustering model: We compared the effectiveness of clustering model connections among ResNet-18, ResNet-34, ResNet-50, and ResNet-101 backbones of FCN-32 to determine the optimal combination for ImSE-Net (Table 13).

2. Sensitivity of hierarchical levels: We compared the clustering accuracy of the backbone network when combined with ImSE-Net at different hierarchical levels, including Conv2_x, Conv3_x, Conv4_x, and Conv5_x, to identify the sensitive region within the hierarchy (Table 14).

3. Influence of hidden pseudo-samples model: We evaluated the contribution of different components of UGLS with various combinations to examine the improvement brought by the proposed global–local hidden pseudo-samples model (Table 15).

4. Precision analysis of different stages: We compared the effectiveness of the model links in stage 1 (ImSE-Net+SLICm), stage 2 (UGLS), and the proposed method in different stages to determine the optimality of the model stage links in the proposed method (Table 16).

Table 13. Experimental results with various backbone network combinations.

Metric	Im + ResNet-18	Im +ResNet-34	Im +ResNet-50	Im +ResNet-101
OA	62.32	63.22	63.65	63.98
MIoU	27.83	28.19	28.98	29.36
ACC	64.86	65.32	65.77	65.86
NMI	61.63	62.32	62.53	63.01

Table 14. Evaluation results of the hierarchical levels.

	Backbone	Stride convolution layer				OA	MIoU	ACC	NMI
		Conv_2	Conv3_x	Conv4_x	Conv5_x
(a)	ResNet-18				√	58.28	24.22	59.44	57.67
(b)	ResNet-18			√	√	62.32	27.83	64.86	61.63
(c)	ResNet-18		√	√	√	63.55	29.28	65.01	62.88
(d)	ResNet-18	√	√	√	√	60.12	26.36	62.14	60.31
(e)	ResNet-101				√	59.30	25.18	61.33	58.65
(f)	ResNet-101			√	√	63.98	29.36	65.86	63.01
(g)	ResNet-101		√	√	√	62.15	27.58	64.52	61.24
(h)	ResNet-101	√	√	√	√	65.74	31.89	66.93	63.47

Table 15. Influence of main components on semantic segmentation results.

	Method	GHPsP		CA	LHPsP	OA	MIoU	ACC	NMI
		TAs	TSs
(a)	IS+ TAs+ CT	√		√		70.14	32.36	70.44	70.61
(b)	IS+ TAs+ CT+ LHPsP	√		√	√	72.25	33.59	72.66	72.83
(c)	IS+ GHPsP+ CT	√	√	√		73.50	34.22	73.54	73.75
(d)	IS+ GHsP+ LHPsP	√	√		√	71.23	32.63	71.93	71.52
(e)	IS+UGLS (Ours)	√	√	√	√	75.36	35.49	75.44	75.76

Table 16. Influence of different stages on semantic segmentation results.

	Different stages			OA	MIoU	ACC	NMI	Time / (s)
	stage 1	stage 2	Ours
(a)	√			65.74	31.89	66.93	63.47	4526
(b)		√		73.23	33.26	73.86	72.36	5079
(c)			√	75.36	35.49	75.44	75.76	9605

Here, ‘Im’ refers to the ImSE-Net model, which replaces the striding of Conv4_x and Conv5_x in the backbone of FCN-32. In this ablation study, ImSE-Net achieves its best merging capability when ResNet-101 is used as the backbone network, but the test results for ResNet-34 and ResNet-50 are close. This illustrates that as the number of convolutional layers increases, the network becomes more adept at handling complex features and larger datasets, resulting in higher accuracy and improved generalization ability. Furthermore, our proposed ImSE-Net model demonstrates excellent robustness, enabling seamless integration into existing architectures without altering their feedforward paths.

We only present the performance of ResNet-101 and ResNet-18 because the performance of ResNet-34 and ResNet-50 falls between those of these two models, as confirmed by Table 13. Considering the clustering effectiveness of the backbone shown in Table 14, the performance consistently improves as the hierarchical levels increase.

Comparisons of (b) and (a) as well as (f) and (e) indicate that the most significant performance enhancements occur when ImSE-Net is incorporated at hierarchical levels, progressing from 1 (Conv5_x) to 2 (Conv4_x and Conv5_x). When the hierarchical levels are further expanded to 3 (Conv3_x, Conv4_x, and Conv5_x), as in (g) and (c), the algorithm presents varying degrees of fluctuations, resulting in a slight increase (b) and a significant decrease (f). However, embedding the model with four levels results in a sharp drop for ResNet-18 (h) compared with (c), while ResNet-101 (h) experiences a significant boost, reaching its peak compared with (g) and outperforming all other algorithms, which demonstrates that the improvement is brought by the proposed implicit clustering module.

Here, IS, GHPsP, LHPsP, TAs, TSs, and CT denote ImSE-Net+SLICm, global hidden pseudo-positive, local hidden pseudo-positive, task-agnostic training, task-specific training, and consistency assignment, respectively.

According to Table 15, compared with (a) in which both the task-specific GHPsP and LHPsP are not used, employing each of them leads to improvements in OA (+2.11%), MIoU (+1.23%), ACC (+2.22%), and NMI (+2.22%) for (b), and OA (+3.36%), MIoU (+1.86%), ACC (+3.10%), and NMI (+3.14%) for (c). These improvements demonstrate that the introduction of both task-specific GHPsP and LHPsP enhances the semantic consistency of pixels and contributes to the optimization of the classification accuracy of unsupervised semantic segmentation. Additionally, a comparison of (d) and (e) demonstrates the significance of maintaining consistent assignments in selecting GHPsP. The comparison shows advancements in OA (+4.13%), MIoU (+2.86%), ACC (+3.51%), and NMI (+4.24%). Overall, the combination of GHPsP and LHPsP offers a significant advantage over the baseline and some popular deep clustering methods (Tables 8–12) in accurately segmenting ground objects and different regions with various objects.

Table 16 show that stage 1 and stage 2 are independent and significantly different from each other, and both have a beneficial effect on the performance of the algorithm in this paper.

The stage 2 further improves the clustering accuracy of stage 1 on OA (+7.49%), MIoU (+1.37%), ACC (+6.93%), NMI (+5.89%) and Time (+533s), which demonstrates that although excavating hidden positives to learn rich semantic relationships and ensure semantic consistency in local regions is beneficial for segmentation tasks, it takes more time to implement. Note that the overall segmentation accuracy improves progressively with the gradual addition of stages. The performance of Ours is better than that of stage 2 (OA: +2.13%; MIoU: +2.23%; ACC: +1.58%; NMI: +3.40%; Time: +4526s), showing the importance of the proposed stage 1. In summary, our proposed method achieves the best segmentation capacity, as it combines both algorithms and consistently. By testing them all (and more) in Tables 13–16, the results justify our choices.

4. Conclusion

To address inaccuracies in boundary segmentation and inconsistent sub-area predictions for ground objects in the deep learning-based semantic segmentation of HSRSIs, the approach integrates the ImSE-Net model with the SLICm superpixel model, merging high-level abstract semantic features with object edge details from superpixels to generate semantic segmentation results. This method aids in recovering image details lost during the pooling or downsampling processes, particularly in capturing object edges. Moreover, the method leverages an unsupervised semantic segmentation model and incorporates both global and local implicit regularization methods to learn rich semantic information with local consistency and effectively merge over-segmented regions to obtain final segmentation results. However, while our method shows improvements, the increase in accuracy is moderate. This network optimization algorithm only adjusts certain function modules to ensure that the models are available for task-specific segmentation. Therefore, establishing a robust migration model for different data sets is a challenge. Additionally, the underlying scene structure extracted by relationship and interaction modeling is an important research direction to promote the accuracy of image semantic segmentation in the future. Although researchers have been trying to go deep to the relational modeling, this research is still in the preliminary stage.

Author contributions

Acquisition of the financial support for the project leading to this publication, H.N. and H.L. Application of statistical, mathematical, computational, or other formal techniques to analyze or synthesize study data, Z.Y., X.W. and K.Y. Preparation, creation, and/or presentation of the published work by those from the original research group, specifically critical review, commentary, or revision, including pre- or post-publication stages, H.N. and Z.Y.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The code used in this study are available by contacting the corresponding author.

Additional information

Funding

This research was funded by Scientific and Technological Innovation Team of Universities in Henan Province, grant number 22IRTSTHN008.