Few-shot segmentation based on multi-level and cross-scale clustering

Abstract

The problem of image segmentation with few-shot learning is addressed in this paper, which is a challenging task due to the lack of sufficient high-precision annotated data. A novel method that consists of two modules is proposed: a multi-level fuzzy clustering guidance module and a cross-scale feature fusion module. The former module can extract image features in a class-independent feature space and fuse them with different scale information, while the latter module can reduce the information loss caused by cross-scale transmission. The feature association map between the support image and the query image can be learned by the proposed method, and the inconsistency of target object categories can be overcome. The proposed method is evaluated on Pascal and COCO datasets, and it is shown that it outperforms the state-of-the-art algorithms in both one-shot and k-shot segmentation scenarios.

KEYWORDS:

• Few-shot segmentation
• fuzzy clustering
• multi-level
• cross-scale

Introduction

Using deep learning methods to solve image processing problems is one of the current research hotspots (Liu, Liu, Cao, et al., 2022; Zhang et al., 2021). However, deep learning techniques are highly data-dependent. When the high-precision labelled training samples are insufficient, the algorithm model of deep learning cannot achieve convincing results (Eltouny et al., 2023). Labelling data is not only a heavy workload but also requires markers with relevant expertise (Deng et al., 2009; Sung et al., 2018; Zhang et al., 2019).

To solve the above problems, some researchers have proposed a new algorithm, called few-shot learning, which can learn the universal laws of things from a limited number of examples and apply them to solve new problems. In few-shot learning, applying the learned prior knowledge to the unknown domains is the key to the success of the algorithm (Dong & Xing, 2018; Lang et al., 2022, 2023; Liu, Liu, Yao, et al., 2022; Zhao et al., 2017).

The mainstream framework for few-shot learning is shown in Figure 1. First, the feature extraction network extracts support image features $F_s$ and query image features $F_q$ with shared weights. Then $F_s$ and $F_q$ are input into the association mapping learning module, in which the correlation map between $F_s$ and $F_q$ is established. In the final stage, new query image features are generated under the association mapping guidance module and input to the target network for segmentation. In few-shot image segmentation, the key problem is to learn the relationship between support images and query images (Ding et al., 2023).

Figure 1. The mainstream framework of few-shot learning.

Currently, there are two ways to obtain guidance information in the association mapping guidance module: kinship learning and prototype feature learning (Deng et al., 2009; Zhang et al., 2019; Zhao et al., 2017). Specifically, kinship learning can directly match the pixel relationship between the support image and the query image. And use the pixel relationship as prior knowledge to improve the performance of image segmentation (Lu et al., 2023; Sung et al., 2018). However, kinship learning has poor generalisation ability. Prototype learning compresses the mask information of the supporting image into a support prototype. Compared with kinship learning, prototype learning is more robust. But it is easy to ignore the critical information in the image (Zhang et al., 2019). In addition (Liu, Wang et al., 2020), if the target object in the support image is significantly different from the target object in the query image, neither kinship learning nor prototype feature learning can establish an accurate association mapping guidance module (Chen et al., 2020; Gao, Fang, et al., 2022; Gao, Xiao, et al., 2022). To address these issues, some recent studies have proposed some new few-shot semantic segmentation methods, which all utilise the information in the support set to improve the segmentation performance of the target categories in the query image (Boudiaf et al., 2021; Lu et al., 2021).

In the process of establishing the association mapping guidance module, more accurate prior knowledge of the support image is needed to provide more valuable prior knowledge for the query image segmentation (Zhao et al., 2017). However, the prior knowledge in association mapping learning is too specific, which reduces the generalisation ability of the model.

In this paper, a multi-level fuzzy clustering guidance module is designed innovatively to build association mapping guidance. Instead of using a single scale image, the module retrieves multi-level information of the query image in four different sizes. And, the idea of clustering is introduced to obtain pixel blocks with similar attributes in the image, which are used as prototypes for subsequent operations. In addition, in order to fully utilise and integrate features of different scales, a cross-scale feature fusion module is proposed. This module can mine the correlation between query image features and support image features at different levels, resulting in reduced the loss of the image information caused by cross-scale transmission.

Experiments on Pascal (Vinyals et al., 2016) dataset and COCO dataset (Lin et al., 2014) illustrate that the proposed network not only performs well in one-shot segmentation, but also can be easily extended to k-shot segmentation experiments. It is worth mentioning that the transition to k-shot does not incur any additional computation cost.

The main contributions of this paper can be summarised in three aspects.

1. The idea of fuzzy clustering is introduced to extract the image features from a category-independent feature space, which provides prior knowledge for query image segmentation.

2. A multi-level fuzzy clustering guidance module is proposed to grasp the support prototype from four spatial dimensions. It can reduce the negative impact caused by the inconsistency between the target objects in the support image and the query image.

3. Cross-scale feature fusion module is designed to fully use and integrate feature information of different scales and reduce the loss caused by information flow.

The rest of this paper is organised as follows. In the following section, we introduce related fields and recent research works. Section “Proposed method” introduces the structure of the proposed model in detail. Section “Experiment” presents the experimental results on different datasets. Finally, the current work is summarised and future research directions are briefly described.

Related work

Image segmentation

Image segmentation is the basis of computer vision, image analysis and other related fields. In the last decade, significant progress has been made in image segmentation due to the emergence of fully convolutional network (FCN) (Long et al., 2015) and its variants, including SegNet (Badrinarayanan et al., 2017), UNet (Ronneberger et al., 2015), RefineNet (Lin et al., 2017) and others. Recently, multi-scale feature aggregation or attention mechanism has brought new technological breakthroughs for image segmentation (Yao et al., 2020; Yu et al., 2022). However, all models face the same limitations of data dependency. The volume of the training set and the accuracy of pixel-level labelling are the key factors that affect the model’s performance directly. Also, traditional models cannot identify new categories in the application phase that do not exist in the training set.

Few-shot learning for image segmentation

Different from the above methods, few-shot learning can apply the prior knowledge to predict new image categories. In image segmentation, the task of few-shot learning is to predict the label corresponding to each pixel, which is more challenging than image classification (Liu et al., 2021).

In Dong and Xing (2018), the idea of prototype learning was first proposed to guide image segmentation by measuring the similarity between the prototype and the query pixel. After that, many algorithms are investigated to obtain the prototype. For example, the similarity guidance network (SGOne) (Zhang et al., 2020) takes the average pooling of masks as a prototype. Till now, many new network structures for few-shot image segmentation have been proposed. For example, the learning receptive fields along the depth-axis for rgb-d scene parsing (CRNet) (Xing et al., 2020) proposes a cross-reference module to mine the common feature information between the support image and the query image. Pyramid graph networks (PGNet), bridging the intra-class and inter-class gaps in one-shot segmentation (BriNet) and democratic attention network (DAN) (Yang, Wang, et al., 2020; Zhang, Chen, et al., 2019; Zhang, Sun, et al., 2021) introduces dense pixel-to-pixel connections between the support image and the query image. The prototype mixture models (PMM) (Zhang, Wang, et al., 2021) uses expectation maximisation algorithm to generate multiple support prototypes to solve the problem that a single support prototype is insufficient to describe image features.

Fuzzy clustering

The traditional image segmentation method has little dependence on data and are strong interpretability (Koch et al., 2015; Wang et al., 2020). Even amid the crazy for deep learning, traditional machine learning methods are still hot (Hartigan & Wong, 1979).

Traditional pixel correlation measurement methods mainly include hard correlation and soft correlation. Hard correlation cannot be integrated into modern deep networks due to its non-differentiable nature, such as K-means (Hartigan & Wong, 1979). The emergence of fuzzy set extends the traditional set theory and provides a new tool to deal with soft correlation between sets (Papandreou et al., 2015). On the basis of fuzzy set theory, the fuzzy C-means (FCM) algorithm with simple principle and low complexity was proposed by DUNN (Bezdek et al., 1984).

Definition of problem

General terms are formally defined in this subsection. The support image set is represented as $S$ and the query image set is represented as $Q$. The number of labels and inputs in deep embedding learning is limited, and this model is also known as N-way K-shot classification (N-way: number of classes in a support set, K-shot: K examples in each category). The support image set is formalised as $S\text{ = { (}{x}_{1,1}\text{,}{c}_{1,1}\text{),(}{x}_{1,2}\text{,}{c}_{1,2}\text{),}\cdots \text{,(}{x}_{\text{N},\text{K}}\text{,}{c}_{\text{N},\text{K}}\text{)} }$ and the query image set is as $Q\text{ = { }{q}_1\text{,}\cdots \text{,}{q}_{\text{N}\times \text{M}}\}$, in which $x_{i,j}$ represents the example $j$ in category $i$, $c_{i,j}\text{ = { 1,2}\cdots \text{,N} }$. The train set is formalised as $S_{\mathrm{train}}\text{ = { (}{I}_{s/q}\text{,}{M}_{s/q}\text{)} }$ and the test set is as $S_{\mathrm{test}}\text{ = { (}{I}_{s/q}\text{,}{M}_{s/q}\text{)} }$, meeting the requirement $S_{\mathrm{train}}\cap S_{\mathrm{test}}\text{ = }\mathrm{\varnothing }$. $I\in R^{H\times W\times 3}$ is the RGB image and $M\in R^{H\times W}$ is the mask image.

Industrial significance

This paper proposes a method based on few-shot learning segmentation, which is a technique for segmenting images with limited annotated data. This technique has many important applications in various domains, such as medical image analysis, autonomous driving, machine vision, etc. For instance, tumours, tissues and cells in medical images can be segmented using only a few samples by this method, which can improve the accuracy and efficiency of diagnosis.

Proposed method

Different from the existing few-shot methods, which compress the query image into a supporting prototype and use a single-layer information fusion structure, the proposed model aims to grasp the multi-level information in four different sizes, breaking the limitations of a single scale. Besides, the proposed model introduces the idea of clustering to obtain pixel blocks with similar properties as supporting prototypes. Also, the model can reduce the loss of image information caused by cross-scale transmission effectively.

The overall architecture of the proposed network model is presented in Figure 2.

Figure 2. The overall architecture of the model.

Firstly, support image features and query image features are retrieved by feature extraction network with shared weights. The support image features (after pre-processing) and the query image features are input into the multi-level fuzzy clustering guidance module (MLFCG), in which prior knowledge with different levels can be retrieved from support image. Under the guidance of prior knowledge, the MLFCG module generates four new query image features. Next, to minimise the loss of image feature information and increase the resolution of the output image, four new query images are put into the cross-scale feature fusion module (CSFF). In the final stage, the feature information output by CSFF is input into the target network, in which $1\times 1$ convolution is performed for pixel classification.

Multi-level fuzzy clustering guidance module

The idea of fuzzy clustering is introduced in multi-level fuzzy clustering guidance module (MLFCG) to measure the soft correlation between pixels. The purpose is to obtain the cluster centre as a prototype.

As shown in Figure 3, the model uses four adaptive average pooling to get query features at four scales ($60\times 60$, $30\times 30$, $15\times 15$ and $8\times 8$). Hence, the spatial position information is first added to the support image, and the size of the support feature is changed from $H\times W\times C$ to $H\times W\times (C\text{ + 2)}$. In addition, to improve the accuracy of clustering and reduce the amount of computation, down-sampling is used to obtain the support mask to filter out the background information. Then, the feature map is compressed from $H\times W\times C$ to ($N_m,C+2$), where $N_m$ is the number of pixels of the corresponding mask image of the support image. The feature vectors of ($N_m,C+2$) are input into the Cluster Guiding Prototype (CGP) to calculate the support prototype. After four parallel CGPs, four-level support prototypes (${\tilde{F}}_{s1}$, ${\tilde{F}}_{s2}$, ${\tilde{F}}_{s3}$ and ${\tilde{F}}_{s4}$) corresponding to query features are obtained.

Figure 3. The multi-level fuzzy cluster guidance-extract prototype.

Next, the four support prototypes obtained from fuzzy clustering are adopted as prior knowledge to query features. As shown in Figure 4, query features and support prototypes are input into the Prototype Guiding Allocation (PGA). Under the guidance of support prototype with different levels, four new query features of different scales are generated. Next, CGP and PGA are explained in detail.

Figure 4. The multi-level fuzzy clustering guidance-use prototype.

Cluster guiding prototype (CGP)

This paper introduces the idea of clustering to aggregate image features with similar properties as a supporting prototype, which can provide more useful prior knowledge for subsequent operations. The specific implementation of this part is in Cluster Guiding Prototype (CGP). First, the random initialisation supports the original location of the prototype on space. Set the number of supported prototypes to $C$, and the corresponding objective function is shown as follows: (1) $F=\sum _{i=1}^C\sum _{j=1}^n{u}_{\mathrm{ij}}^m{d}^2(x_j,v_i)$(1) The objective function in equation (1) is under the constraints ${\sum }_{i=1}^c{u}_{\mathrm{ij}}=1$ for all pixels, where $n$ is the number of pixels, and $u_{\mathrm{ij}}$ is the fuzzy membership degree of pixel $j$ to support prototype $i$. $d$ is the metric to measure the distance, defined as follows. (2) $d=\sqrt{{(d_f)}^2+{(d_s/r)}^2}$(2) where $d_f$ is the Euclidean distance of image feature information, $d_s$ is the Euclidean distance of image spatial location information and $r$ is the balance factor.

The unconstrained objective function constructed by the Lagrange multiplier method is shown in equation (3): (3) $J=\sum _{i=1}^C\sum _{j=1}^n{u}_{\mathrm{ij}}^m{d}^2(x_j,v_i)+\sum _{j=1}^n{\lambda }_j\left(\sum _{i=1}^C{u}_{\mathrm{ij}}-1\right)$(3) where $\mathit{\lambda }=[{\lambda }_1,{\lambda }_2,\dots ,{\lambda }_n{]}^T$ is vector parameter in Lagrange Multiplier Method (LMM). Taking the derivative of $J$ with respect to $u_{\mathrm{ij}}$, and setting $\frac{\mathrm{\partial }J}{\mathrm{\partial }{u}_{\mathrm{ij}}}=0$, we will retrieve the optimal value of $u_{\mathrm{ij}}$ in minimising the Equation (3) as follows. (4) $\frac{\mathrm{\partial }J}{\mathrm{\partial }{u}_{\mathrm{ij}}}=m{u}_{\mathrm{ij}}^{m-1}{d}^2(x_j,v_i)+{\lambda }_j=0$(4) From equation (4), we obtain $u_{\mathrm{ij}}={\left(\frac{-{\lambda }_j}{m{d}^2(x_j,v_i)}\right)}^{\frac{1}{m-1}}$. Because of ${\sum }_{i=1}^C{u}_{\mathrm{ij}}={\sum }_{i=1}^C{\left(\frac{-{\lambda }_j}{m{d}^2(x_j,v_i)}\right)}^{\frac{1}{m-1}}=1$, we can infer ${\left(\frac{-{\lambda }_j}{m}\right)}^{\frac{1}{m-1}}=1/{\sum }_{i=1}^C\frac{1}{d{(x_j,v_i)}^{2/m-1}}$. Hence, the fuzzy membership degree of image pixels $j$ with respect to prototype $i$ can be calculated as equation (5): (5) $u_{\mathrm{ij}}=\frac{1}{{\sum }_{k=1}^C{\left(\frac{d(x_j,v_i)}{d(x_j,v_k)}\right)}^{\frac{2}{m-1}}}$(5) Taking the derivative of $J$ with respect to $v_i$ as Equation (6), and setting $\frac{\mathrm{\partial }J}{\mathrm{\partial }{v}_i}=0$, we will retrieve the optimal value of in minimising Equation (5) as follows. (6) $\frac{\mathrm{\partial }J}{\mathrm{\partial }{v}_i}=-2\sum _{j=1}^n{u}_{\mathrm{ij}}^m(x_j-v_i)=0$(6) The update equation of support prototype $v_i$ is retrieved as follows. (7) $v_i=\frac{{\sum }_{j=1}^n{u}_{\mathrm{ij}}^m{x}_j}{{\sum }_{j=1}^n{u}_{\mathrm{ij}}^m}$(7) The whole CGP process is delineated in Algorithm 1.

Table

Algorithm 1. Cluster guiding prototype (CGP)
Input: support feature $F_s$, support mask $M_s$, the number of prototype $C$, maximum iterations $\mathrm{maxIter}$ and minimal error threshold $\epsilon$;
Output: prototype;
Step 1. Extract masked features ${\tilde{F}}_s$ via support mask $M_s$;
Step 2. Initialise the membership matrix;
Step 3. Set iteration counter $\mathrm{it}=1$;
Step 4. Update support prototype $v^{\mathrm{it}}$ based on Equation (7);
Step 5. Update the fuzzy partition matrix $u^{\mathrm{it}}$ based on Equation (5);
Step 6. If $max\{u^{\mathrm{it}}-u^{\mathrm{it}-1}\}<\epsilon$ or $\mathrm{it}=\mathrm{maxIter}$ holds, Algorithms ends; else $\mathrm{it}=\mathrm{it}+1$, go to Step 4.

Prototype guiding allocation (PGA)

PGA adopts the supporting prototype to guide the query image to generate new query image features, presented in Figure 5.

Figure 5. The structure of prototype guiding allocation.

In PGA, the correlation is measured by the Cosine value between the support prototype and the query image feature. (8) ${C_i}^{x,y}=\frac{v_i\cdot {F_q}^{x,y}}{||v_i||\cdot ||{F_q}^{x,y}||}i\in \{1,2,\dots ,C\}$(8) where ${F_q}^{x,y}\in R^{c\times 1}$ is the feature of the query image at position $(x,y)$. The location index of the most associated support archetypes is determined by Equation (9). (9) $G^{x,y}=\underset{i\in \{0,\dots ,\text{C}\}}{\arg max}{C_i}^{x,y}$(9) Next, the guide map $G^{x,y}\in R^{h\times w}$ is recombined with the index values corresponding to the support prototype to get the location index table of the query image, denoted as $F_G\in R^{c\times h\times w}$. In the last layer of the clustering guidance module, the probability map $P$ is obtained by summing the similarity. At the end of the model, the final query image feature ${\tilde{F}}_q$ are obtained by concatenating the probability map, the cluster index table and the original image features. (10) ${\tilde{F}}_q=f(F_q\oplus F_G\oplus P)$(10)

Cross-scale feature fusion

To obtain a more reliable support prototype, the MLFCG method is proposed in this paper. In MLFCG, we use four prior knowledge to guide query images of four size levels, and finally generate four new query images. To fuse these four new query images, this paper proposes a Cross-scale feature fusion model (CSFF).

CSFF can make full use of and integrate feature information of different scales to reduce the loss caused by information flow. The structure of CSFF is shown in Figure 6.

Figure 6. The structure of cross-scale feature fusion.

As shown in Figure 6, four new query images of different sizes are input into the CSFF. Firstly, three new query image features with larger sizes are down-sampling. And the image features obtained by down-sampling are added to the upper image feature layer by channel connection. CSFF maintains a higher level of image feature detail by fusing different size image features. This feature fusion method integrates the features of different scales longitudinally and extracts the depth information of each scale feature horizontally. Next, we use $8\times 8$ convolution operations at each layer for information fusion.

Image features are connected at the end of the model to obtain the fused query image features. In addition, to further complement the image details, we concatenate the original query image features with the fused query image features.

Loss function

The loss function of the proposed model includes the cross entropy between the result and the reference image, and four cross-entropy losses of different scales, formalised as follows. (11) $L=L_1+\frac{\sigma }{n}\sum _{i=1}^n{L}_2^i\{i\in 1,2,\dots 4\}$(11) where $L_1$ is the cross entropy between the segmentation result image and the reference image, and $L_2^i$ is the cross entropy between the query image of different scales and the reference ground truth, which is computed in the cross-scale feature fusion module.$\sigma$ is the balance parameter, set to 1.0 in the proposed model.

Experiment

Datasets and evaluation metric

To verify the performance of the model, we used two datasets, the Pascal dataset and the COCO dataset. It is well known that Pascal consists of extensions of the Pascal VOC 2012 (Tian et al., 2022) and SBD (Nguyen and Todorovic, 2019) datasets, while COCO is well-known for its higher segmentation difficulty.

In the experiments, the 20 classes in the Pascal dataset were divided into four parts, each part containing five categories. The model is trained by cross-validation. Three parts are selected to train the model, and the remaining part is used to verify the effect of the model. In the test phase, 1000 support-query pairs were randomly sampled. Also, the 80 classes in the Pascal dataset are divided into four parts, each with 20 classes. The COCO validation set contains 40,137 images (80 classes). Because the number of images in COCO is much larger than that in Pascal, the 1000 query-support pairs used in previous work are insufficient. Therefore, during the evaluation, 20,000 support-query pairs were randomly selected to ensure reliable results.

In image segmentation, IoU (intersection-over-union) is often used as an evaluation index. In general, the higher the IoU, the higher the accuracy. When the number of classes in the training samples is unbalanced (for example, 49 sheep samples and 378 human samples), the cross-ratio cannot describe the performance accurately. To avoid the influence of class imbalance, mean intersection over union (mIoU) was adopted, which was defined as follows: (12) $\begin{array}{rl}\text{IoU}& =\frac{TP}{TP+FP+\mathrm{FN}}\end{array}$(12) (13) $\begin{array}{rl}\text{mIoU}& =\frac{1}{n_l}\sum _l\text{Io}{\text{U}}_l\end{array}$(13) where $\mathrm{TP},\mathrm{FP},\mathrm{FN}$ is the number of true positive, false positive and false negative pixels and $n_l$ is the number of category objects.

In addition, when the proportion of the area occupied by the target object in the overall image is small, the IoU value cannot evaluate the performance accurately. Therefore, the foreground and background values (FB-IoU) must be considered when evaluating the performance.

Implementation details

In the experiment, ResNet is adopted as the backbone network, combined with SGD optimiser. When Pascal dataset is trained, the initial learning rate is set to 0.0025 and the batch size is set to 4. For COCO dataset, the initial learning rate is set to 0.005 and the batch size is set to 8. The number of supported prototypes is initialised to 5.

Before training, image enhancement operations such as horizontal flip and rotation transformation are performed, and all image sizes are unified to $473\times 473$ (Pascal) and $641\times 641$ (COCO). The runtime environment is Pytorch, and the model run on an Nvidia-SMI GPU.

Ablation study

In the field of few-shot image segmentation, the information obtained after average pooling is often used as a supporting prototype. In order to verify that the support prototypes obtained by GPA module can guide query image segmentation more effectively, this paper designs a model B1 as a comparative experiment. To increase the persuasion of the experiment, the B1 model is almost identical to the proposed model, except for replacing the method of obtaining the prototype support from GPA to average pooling.

Ablation studies were performed on the Pascal datasets, as tabulated in Table 1, in which the column labelled Δ denotes the difference between 1-shot and 5-shot. In Table 1, the guidance prototype obtained under the guidance of fuzzy clustering can better grasp the image features better and can retrieve better mIoU values. The comparison means that the mask information of the supporting image is condensed into a supporting prototype through average pooling, the feature information in the supporting image is easily lost and the prior knowledge cannot be extracted.

Table 1. mIoU comparison of different models in Ablation study.

			1-shot					5-shot
Backbone	Method	Δ	Fold-0	Fold-1	Fold-2	Fold-3	Mean	Fold-0	Fold-1	Fold-2	Fold-3	Mean
ResNet50	B1	3.00	55.87	59.62	53.52	57.44	56.61	59.07	65.38	60.70	53.32	59.61
	B2	4.65	58.84	67.86	56.79	53.66	59.29	63.66	70.55	64.17	57.38	63.94
	our	3.69	58.76	69.58	57.22	56.33	60.46	63.27	70.89	64.46	57.98	64.15
ResNet101	B1	6.56	56.86	61.22	50.46	50.99	54.88	60.84	68.61	58.56	57.75	61.44
	B2	5.05	59.84	67.43	55.59	54.39	59.31	64.55	71.32	64.24	57.33	64.36
	Our	3.97	59.93	69.12	57.89	56.83	60.94	63.80	72.68	64.39	58.77	64.91

The bold values indicate the best result in each index.

In order to confirm that the support prototype in four spatial dimensions can reduce the negative impact caused by the inconsistent target objects in the support-query image, a B2 model is designed, in which CGP and PGA are simplified into a single structure. In B2, the corresponding support prototype is only learned from the original support image, and the query image features fused with guidance information are obtained by single-layer PGA and directly input into the target network. As shown in Table 1, the mIoU of the proposed model is almost the best, which means that the proposed model can achieve the best performance whether the background network is ResNet50 or ResNet101.

In summary, the cross-scale feature fusion module (CSFF) can significantly improve the image segmentation performance, compared with the single-scale model structure.

Comparison to state-of-the-art models

Pascal

The proposed model is compared with related methods separately on the Pascal dataset, and the results are presented in Table 2. According to the mIoU values, this model outperforms or compares the existing methods in the case of 1-shot and 5-shot in the same background network. It is worth mentioning that in the case of 5-shot on the Pascal dataset, the proposed model outperforms the state-of-the-art methods by an average of 3 percentage points. To increase the persuasiveness of the experimental results, mIoU and FB-IoU are adopted as indicators to evaluate the proposed model and state-of-the-art algorithms. According to the data in Table 3, the proposed model can almost achieve the highest level of image segmentation results regardless of which evaluation index is used.

Table 2. mIoU Comparison of the proposed model with state-of-the-art methods on Pascal.

			1-shot					5-shot
y	Method	Δ	Fold-0	Fold-1	Fold-2	Fold-3	Mean	Fold-0	Fold-1	Fold-2	Fold-3	Mean
ResNet50	CANet (Zhang, Lin, Liu, Yao, et al., 2019)	1.70	52.50	65.90	51.30	51.90	55.40	55.50	67.80	51.90	53.20	57.10
	PGNet (Zhang, Lin, Liu, Guo, et al., 2019)	2.50	56.00	66.90	50.60	50.40	56.00	57.70	68.70	52.90	54.60	58.50
	RPMMs (Yang et al., 2020)	2.15	55.15	66.91	52.61	50.68	56.34	56.28	67.34	54.52	51.00	57.30
	SIpNet (Gairola et al., 2020)	2.85	54.82	67.33	54.52	52.02	57.19	57.20	68.50	58.40	56.05	60.04
	PPNet (Liu et al., 2020)	10.5	47.83	58.75	53.80	45.63	51.50	58.39	67.83	64.88	56.73	61.96
	PFENet (Tian et al., 2022)	1.90	61.70	69.50	55.40	56.30	60.80	63.10	70.70	55.80	57.90	61.90
	CWT (Lu et al., 2021)	7.30	56.30	62.00	59.90	47.20	56.40	61.30	68.50	68.50	56.60	63.70
	ASGNet (Li et al., 2021)	4.65	58.84	67.86	56.79	53.66	59.29	63.66	70.55	64.17	57.38	63.94
	ours	3.69	58.76	69.58	57.22	56.33	60.46	63.27	70.89	64.46	57.98	64.15
ResNet101	FWB (Nguyen and Todorovic, 2019)	3.73	51.30	64.49	56.71	52.24	56.19	54.84	67.38	62.16	55.30	59.92
	DAN (Wang et al., 2020)	2.30	54.70	68.60	57.80	51.60	58.20	57.90	69.00	60.10	54.90	60.50
	PFENet (Tian et al., 2022)	1.30	60.50	69.40	54.40	55.90	60.10	62.80	70.40	54.90	57.60	61.40
	CWT (Lu et al., 2021)	6.70	56.90	65.20	61.20	48.80	58.00	62.60	70.20	68.80	57.20	64.70
	ASGNet (Li et al., 2021)	5.05	59.84	67.43	55.59	54.39	59.31	64.55	71.32	64.24	57.33	64.36
	Ours	3.97	59.93	69.12	57.89	56.83	60.94	63.80	72.68	64.39	58.77	64.91

The bold values indicate the best result in each index.

Table 3. Comparison of FB-IoU on Pascal.

	mIoU		FB-IoU
Method	1-shot	5-shot	1-shot	5-shot
OSLSM (Shaban et al., 2017)	40.80	43.95	61.30	61.50
Co-FCN (Rakelly et al., 2018)	41.10	41.40	60.10	60.20
AMP (Siam et al., 2019)	43.40	46.40	62.20	63.80
SG-One (Zhang et al., 2020)	46.30	47.10	63.10	65.90
CANet (Zhang, Lin, et al., 2019)	55.40	57.10	66.20	69.60
PGNet (Zhang, Lin, Liu, Guo, et al., 2019)	56.00	58.50	69.90	70.50
PANet (Wang et al., 2019)	48.10	55.70	66.50	70.70
PFENet (Tian et al., 2022)	60.10	61.40	73.30	73.90
Ours	60.50	64.04	72.99	75.07

The bold values indicate the best result in each index.

In order to show the performance more vividly, five sample images of different categories (including cows, cars, cats, sofas and birds) are presented in Figure 7. In Figure 7, the first two rows are the supporting image and the corresponding ground truths, the last two rows are the query image and the corresponding ground truths, and the last row presents the results of the proposed model. From Figure 7, we can see that the proposed model can achieve satisfying results even if there is a large difference between the support image and the query image. The results show that the proposed model is reasonable and reliable.

Figure 7. Image segmentation results on Pascal.

COCO

Compared with Pascal, COCO set has a larger amount of data and a more complex situation, hence almost all algorithms perform poor on COCO dataset. Table 4 shows that the proposed model still has a significant advantage over the state-of-the-art algorithms. As is shown, the proposed model outperforms the state-of-the-art methods in both 1-shot and 5-shot settings while using fewer parameters. And the column labelled Δ denotes the difference between 1-shot and 5-shot. The performance on 5-shot is better on 1-shot. The FB-IoU (Yang, Wang, et al., 2020) value of the corresponding models on COCO dataset are presented in Table 5. Compared with other algorithms, the proposed model can improve by at least one percentage point and even up to five percentage point in the value of FB-IoU. In addition, more accurate results can be obtained with ResNet101, and 5-shot setting generally provides more accuracy than the 1-shot setting.

Table 4. Comparison with state-of-the-arts on COCO with per-split results.

			1-shot					5-shot
y	Method	Δ	Fold-0	Fold-1	Fold-2	Fold-3	Mean	Fold-0	Fold-1	Fold-2	Fold-3	Mean
ResNet50	FWB (Nguyen and Todorovic, 2019)	2.5	17.0	18.0	21.0	28.9	21.2	19.1	21.5	23.9	30.1	23.7
	DAN (Wang et al., 2020)	5.4	–	–	–	–	24.2	–	–	–	–	29.6
	PPNet (Liu et al., 2020)	9.6	28.1	30.8	29.5	27.7	29.0	39.0	40.8	37.1	37.3	38.6
	RPMMs (Yang, Liu, et al., 2020)	4.9	29.5	36.8	28.9	27.0	30.6	33.8	42.0	33.0	33.3	35.5
	ASGNet (Li et al., 2021)	7.9	34.8	36.9	34.3	32.0	34.5	40.9	48.2	40.1	40.5	42.4
	CWT (Lu et al., 2021)	8.4	32.2	36.0	31.6	31.6	32.9	40.1	43.8	39.0	42.4	41.3
	RePRI (Boudiaf et al., 2021)	8.1	31.2	38.1	33.3	33.0	34.0	38.5	46.2	40.0	43.6	42.1
	Our	8.1	35.0	35.8	34.7	33.1	34.6	41.0	47.9	41.9	40.2	42.7
ResNet101	PGNet (Zhang, Lin, Liu, Guo, et al., 2019)	-	32.2	30.5	31.6	29.7	31.0	–	–	–	–	–
	PFENet (Tian et al., 2022)	5	34.3	33.0	32.3	30.1	32.4	38.5	38.6	38.2	34.3	37.4
	CWT (Lu et al., 2021)	9.6	30.3	36.6	30.5	32.2	32.4	38.5	46.7	39.4	43.2	42.0
	Our	5	36.7	37.5	34.1	33.1	35.3	40.2	39.8	38.9	36.7	38.9

The bold values indicate the best result in each index.

Table 5. Comparison of FB-IoU on COCO.

	mIoU		FB-IoU
Method	1-shot	5-shot	1-shot	5-shot
FWB (Nguyen and Todorovic, 2019)	21.2	23.7	–	–
DAN (Wang et al., 2020)	24.2	29.6	62.3	63.9
PPNet (Liu et al., 2020)	29.0	38.6	58.6	61.9
Our	31.6	40.3	65.5	64.7

The bold values indicate the best result in each index.

In Figure 8, the support-query image pair (including the category of cows, giraffes, humans, airplanes, surfers) is selected to show the advantages of the proposed model. The images in Figure 8 are arranged in the same order as in Figure 7. From Figure 8, the results of the proposed algorithm are satisfactory.

Figure 8. Image segmentation results on dataset COCO.

Discussion

Our method is designed for few-shot learning segmentation, which is a technique to segment images when there is a scarcity of high-precision annotated data. However, our method also faces some challenges and limitations. On one hand, our method relies on a fuzzy clustering algorithm to extract image features, but this algorithm requires pre-setting the number of clusters and the membership degree threshold, which may affect the clustering outcomes and the segmentation quality. On the other hand, our method uses image features of different scales to fuse with prior knowledge, but this also increases the computation and memory consumption, which may reduce the segmentation speed and efficiency.

Despite these limitations, our method still has some unique advantages and innovations. We believe that our method has great potential for improvement and application in the future.

Conclusions

Aiming at the problem that the existing supporting prototype cannot fit the image features accurately and lose more information in the single-level flow of image features, this paper proposed a multi-level clustering module and a cross-scale feature fusion module. Specifically, the multi-level clustering module is based on the idea of fuzzy clustering, and the feature vectors with similar properties in support images are used as support prototypes. The cross-scale feature fusion module can reduce the information loss caused by information flow between single layers and improve the accuracy of image segmentation. Extensive experiments and ablation experiments on both Pascal and COCO demonstrate the superiority of the proposed model, which means that the proposed model can achieve better performance than the current state-of-the-art algorithms.

Acknowledgements

The authors also gratefully acknowledge the reviewers’ helpful comments and suggestions, which will improve the presentation significantly.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This research was funded by National Natural Science Foundation of China NSF of China [grant numbers 62007017, 61873117, U22A2033, 62171209, 62176140] and Basic Research Project of Yantai Science and Technology Innovation Development Plan (2023JCYJ044).