DeFRCN-MAM: DeFRCN and multi-scale attention mechanism-based industrial defect detection method

ABSTRACT

With the technology development, industrial defect detection based on deep learning has attracted extensive attention in the academic community. Different from general visual objects, industrial defects have the characteristics of small sample, weak visibility and irregular shape, which hinder the application of related studies. According to these problems, a few-shot object detection (FSOD) method based on Decoupled Faster R-CNN (DeFRCN) is proposed in this paper. Firstly, it includes fine-tuning processing, because of the small sample characteristics. To adapt to the invisible characteristics of defects, we introduce the Feature Pyramid Network (FPN) and Residual Attention Module (RAM) into DeFRCN, which can enhance the capture ability of multi-scale features and feature association information. Furthermore, the feature representation ability is strengthened by parallel connecting of two channels, consisting of R-CNN head, box classifier and box regression models. Finally, it is completed that the pre-training, fine-tuning and testing of the proposed network, with DAGM 2007 and NEU-DET public industrial defect datasets as the base class and flange shaft defect data collected in the laboratory as the new class. To verify the effectiveness of the proposed one, we compare them with other classical FSOD methods. The superiority of the proposed method is obvious.

Introduction

Industrial defect detection aims to capture the appearance defects of various industrial products, which is one of the important technologies to ensure product quality and maintain production stability. The related studies can be applied to scenarios such as unmanned inspection, intelligent inspection, production control, anomaly traceability, etc. In addition, with the development of industrial imaging, computer vision, deep learning and other fields, industrial defect detection technology based on deep learning has become an effective method for product appearance inspection. It has attracted strong attention from academia and industry. However, different from the general objects, industrial defect has many characteristics, i.e. the shortage of defective samples, poor visibility of defects, irregular defects, etc. As a result, industrial defect detection is difficult to meet the requirements of high-precision and high-speed detection tasks at the same time, which hinders their practical application (Ren et al. 2021; Saberironaghi, Ren, and El-Gindy 2023; Tang et al. 2022). The main problems and the related studies of this field are simply shown in Table 1.

Table 1. The main problems and related studies for industrial defect detection task.

Task	Problems	Related studies
Industrial defect detection	The shortage of defective samples	Modify network structure and loss function to avoid overfitting phenomenon
		Design network structure to accelerate convergence
		Defect sample expansion
		Transfer learning
	Poor visibility of defects	Multi-scale feature fusion
	Irregular defects

First of all, the shortage of defective samples is mainly caused by the strict control of defective rate in the industrial production process, the limited number of defective samples, and the high cost of accurate labeling of defective samples. Hence, it is inevitable that the imbalance of positive and negative samples in industrial samples. This problem is the core difficulty in industrial defect detection. To solve this problem, relevant researchers focus on small-sample learning (Imoto et al. 2019; Tang et al. 2020), semi-supervised learning (Gao et al. 2020; He et al. 2019), weakly supervised learning(Niu et al. 2019; Zhu et al. 2017), self-supervised learning (Fei et al. 2020; Ristea et al. 2021), etc. Among them, small sample learning can be used to directly improve the performance of the model when samples are scarce. The main ideas include:

1. Modify network structure and loss function to avoid overfitting phenomenon caused by rare samples (Zhou et al. 2020; Xie et al. 2023)

2. Accelerate network convergence by designing network structure and stable optimization process (Tao et al. 2018)

3. Targeted design of corresponding defect data to achieve defect sample more comprehensive expression(Lin et al. 2020; Apostolopoulos et al. 2023)

4. Transfer the knowledge of other task models to the industrial defect detection task, and fine-tune the pre-trained model with a small amount of data (Jing, Ma, and Zhang 2019; Zhang et al. 2023)

Among them, it needs more theoretical experience as support that helps in network structure modification, loss function and optimization process design. The sample enhancement method can achieve performance superposition by combining different enhancement methods, but the improvement is limited. Moreover, if it is too large that the difference between the enhanced data and the real data, the difficulty of practical application will be enhanced. Few-shot object detection (FSOD) based on transfer learning, especially fine-tuning, has attracted more attention. The related methods refer to the re-use of network weights pre-trained on a baseline dataset to improve generalization capabilities on a new domain with limited data. This type of FSOD usually involves novel categories from the target domain. At present, researchers mainly focus on the selection of pre-trained models to speed up fine-tuning.

Secondly, poor visibility of industrial defects refers to low contrast between defects and surrounding background, insufficient significance and small difference between positive and negative samples. In addition, irregular defect means that the same industrial product may have different kinds of defects, and the same kind of defects may also have diversity characteristics, i.e., shape, size and color. It is necessary to study the network capable of detecting various types of defects to avoid missing detection (Akhyar et al. 2023; Cha et al. 2018). For example, Tao et al. provide a cascaded autoencoder for segmenting and localizing defects, which can adapt to various conditions (Tao et al. 2018). Similarly, the segmentation is applied to avoid missing detection (Tsai, Fan, and Chou 2021; Yang et al. 2022). Furthermore, to solve the poor visibility and irregular problems, the most classical solving idea is multi-scale feature fusion. Among them, Feature Pyramid Networks (FPN) (Lin et al. 2017) has the excellent performance on multi-scale problem, which is often introduced into the classical object detection network. In defect target detection, Li et al. (2018). detect steel surface defects based on the single-stage detector, i.e., YOLO, and integrate shallow features to improve the detection ability of small defects. Mei, Yang, and Yin (2018) use the image pyramid structure to scale the image to three different sizes, which can add noise randomly. Then, to recovery and defect detection, autoencoders are introduced. The above methods improve the detection effect of industrial defects. In the meantime, it is verified that the multi-scale feature fusion can improve the detection ability to industrial defects, consisting in characteristics of poor visibility and irregular shape.

Aiming at the problems of sample shortage, poor visibility and irregularity in industrial defect detection, this paper proposes a multi-scale attention-based FSOD model. Here, decoupled Faster R-CNN (DeFRCN) (Qiao et al. 2021) is the fundamental of the proposed method, whose parameters are pretrained by a large amount of data. Then, the small sample defect data are used to fine-tuning model parameters. On the basis of the classical DeFRCN, FPN is introduced in the backbone part to enhance the learning ability of different scales and feature associations of industrial defects. Moreover, to improve the adaptability of multi-scale feature, we use residual attention module (RAM) (Fei et al. 2017) to obtain the integration of attention maps in each channel and original feature maps. In addition, the R-CNN head, box classifier and box regression parts are connected in two-level parallel. Then, the relative relationship between feature channels is obtained through the outer product calculation to enhance the feature representation. The proposed method can adapt to actual defect, so the application in industrial field is promoted.

Related Works

FSOD Methods

In recent years, image object detection technology based on deep learning has made remarkable achievements. Here, many mature detection models have emerged. However, these models all need to use a large number of labeled samples for training. In actual industrial defect detection, it is difficult to obtain high-quality labeled defect samples. Therefore, it is limited that the application and popularization of these methods in actual industrial defect detection (Luo et al. 2020; Wen et al. 2022). On this basis, FSOD methods have been researched and developed gradually because of its small dependence on samples. According to the idea and model structure of FSOD, existing methods can be divided into metric learning(Hsieh et al. 2019; Lin et al. 2020; Zhang et al. 2019), data augmentation (Gong et al. 2020; Lin et al. 2020), model structure (Chen, Jiang, and Zhao 2020; Fan et al. 2020), fine-tuning (Kang et al. 2018; Sun et al. 2021; Wang et al. 2020), meta-learning (Zhang et al. 2021; Zhu et al. 2020), ensemble (Fan et al. 2021; Han et al. 2021), etc. Among them, the method based on metric learning is easy to implement. However, the positioning accuracy is relatively poor. In data augmentation part, if the difference is large between the augmentation samples and the real samples, practical application will be hindered. The methods based on model structure provide effective auxiliary information by constructing new model structure on the basis of conventional model detection. This kind of method needs more theoretical and empirical support and is not easy to implement. Moreover, meta-learning based methods are trained iteratively by task, and only a small number of iterations are needed for a specific task to achieve better performance. However, the iterative process of this method is prone to non-convergence. The method based on fine-tuning is to use a large amount of data to pre-train the model, and then use a small number of new samples to fine-tune some parameters. The detection accuracy of this method is relatively high, but it is sensitive to hyperparameters. The ensemble methods integrate multiple methods, which can draw on the advantages of these methods. However, its structure is relatively complex, causing large calculation and huge hardware requirements. Obviously, different types of methods have different characteristics. This paper focuses on the industrial defect detection task, which needs to obtain higher detection accuracy because of the high cost of missing detection. Therefore, this paper focuses on the FSOD methods based on fine-tuning.

Recently, fine-tuning based FSOD methods mostly use Convolutional Neural Network (CNN) to pre-train ImageNet classification. Then, the whole target detector is trained on the base class. Finally, fine-tuning is carried out on the novel class. Here, Wang et al (Wang et al. 2020). propose a fine-tuning based FSOD method whose backbone is Faster R-CNN. In fine-tuning stage, the parameters in feature extraction are frozen, but adjusted in classifier and regression. In addition, Wu et al (Wu et al. 2020). propose to use FPN as the main network of Faster R-CNN to solve the problem of unbalanced scale distribution. On this basis, each target could be obtained and transformed to different scales for subsequent PRN and detection processing. In addition, Qiao et al. (2021). extend Faster R-CNN by introducing a gradient decoupling layer for multilevel decoupling and a prototype calibration block for multitask decoupling, i.e., DeFRCN. The former redefines the feature forward operation and gradient backward operation, which decouples the back layer from the front layer. The latter one is a classification model based on offline prototype, which takes the proposal as input and boosts the original classification score and an additional pairwise score for calibration. The overall structure of DeFRCN is shown in Figure 1.

Figure 1. The overall architecture of DeFRCN model(Qiao et al. 2021).

Multi-Scale Object Detection Method

In view of the irregular characteristic of industrial defects, it is necessary to design a detection model which is suitable for defects of different sizes. Studies have shown that, in the general CNN target detection model, the shallow network can extract spatial information such as the edge of the target. With the deepening of the network, features are gradually abstracted and spatial information is constantly lost (Zeiler and Fergus 2014). In addition, object detection includes two subtasks: object classification and location. When objects with large size and rich details, it is necessary to provide stronger information as the basis for classification. When objects with small size and low tolerance of deviation, more fine-grained spatial information is needed to achieve for accurate localization. At present, the most common methods for constructing multi-scale feature expression include: (1) input image pyramid to extract features of different scales (Hao et al. 2017; Lu, Javidi, and Lazebnik 2015); (2) design parallel branches inside the network to build a spatial pyramid (Lazebnik, Schmid, and Ponce 2019); (3) integrate feature layers of different depths inside the network to construct feature pyramid (Cai et al. 2016; Liu et al. 2015), etc.

Firstly, the multi-scale target detection method based on image pyramid randomly inputs images with different scales in the training stage, which can force the neural network to adapt to the target detection of different scales. In the test stage, the same image is detected several times at different scales, and the results are finally integrated by the non-maximum suppression (NMS). However, this kind of method does not only require small batch size in training stage, which affects the accuracy of the model, but also hinder practical application. In addition, parallel branches with different parameters are designed in the network. Here, each branch extracts features in different spatial levels based on its own receptive field, which can also build a spatial pyramid. For example, the SPP module in SPP-NET (Kaiming et al. 2015) adopts the multi-scale block method of SPM to pool each block, so that feature maps of arbitrary size can be converted into feature vectors in fixed length. Finally, integration of feature layers in different depths can be realized by feature map pyramid. This type of method consists of feature fusion of different depths based on cross-layer connections and spatial pyramid construction based on parallel branches with different receptive fields. Among them, FPN is a typical representative of the method based on cross-layer connection (Lin et al. 2017). The basic idea is to detect multi-scale targets by combining the fine-grained spatial information of the shallow feature map and the information of the deep feature maps. On the basis of RPN, this algorithm adds a channel from top to bottom. Starting from the deepest feature map, after 1 × 1 convolution and upsampling processing, the current feature maps are aligned with the shallow feature map. Then, the new feature maps are fused by adding symmetric elements. In the same way, the fusion with the shallow feature map is realized.

Attention Mechanism

Due to its superior effect and convenience of plug and play, attention mechanism has been more and more widely used in deep learning tasks. It can be simply divided into two categories: single-path attention and multi-path attention. Single-channel attention includes Squeeze-and-excitation network (SE-NET), Efficient Channel Attention (ECA-NET), etc. Among them, SE-NET (Jie et al. 2017), which is provided in CVPR2018, uses the idea of attention mechanism. Firstly, interdependence between feature maps is modeled. Then, the importance of each feature map can be adaptively obtained by learning. Finally, the parameters update is based on the importance. Furthermore, ECA-NET (Wang et al. 2020) is an improvement of SE-NET. In this model, more accurate attention information is obtained by the local cross-channel interaction strategy without dimensionality reduction, the adaptive selection method of the size of one-dimensional convolution kernels and one-dimensional convolutional processing. In other words, attention information is obtained by summarizing the cross-channel information. In terms of multi-path attention, ResNeSt (Li et al. 2017) is an improvement of ResNet, including split-attention block. On the basis of ResNet, the raw images are input into k Cardinal branches in ResNeSt. Here, the outputs carry the weight information of different channels. In addition, Convolutional Block Attention Module Network (CBMA) (Woo et al. 2018) builds two submodules, i.e., Spatial Attention Module (SAM) and Channel Attention Module (CAM). The attention information in spatial and channel are integrated by them, respectively. On this basis, the Residual Attention Module (RAM) (Fei et al. 2017) is embedded with the depth separable convolution to obtain the attention map according to the features of each channel. Then, the final output is obtained by adding channel attentional map and spatial attentional map, sigmoid processing and multiplying with original features. This method has better adaptability to super-resolution problems.

The Proposed Method

The Architecture of the DeFRCN-MAM Model

The overall structure of the DeFRCN-MAM is shown in Figure 2. The fundamental of the proposed one is DeFRCN. The main improvement is to introduce FPN and RAM on backbone to improve the learning ability of multi-scale features, and obtain the association information of different channels and spatial feature maps. In addition, the R-CNN head, box classifier and box regression parts are connected in two-level parallel. Then, the relative relationship between feature channels is obtained through the outer product calculation to enhance the feature representation. Finally, the output of the network includes two parts: classification and location.

Figure 2. The architecture of DeFRCN-MAM model.

The Architecture of the DeFRCN

Different from standard Faster R-CNN, DeFRCN involves two gradient decoupling layers (GDL) and an offline prototype calibration block (PCB). Two GDLs are between backbone and RPN and between backbone and R-CNN, respectively, whose function is to adjust the degree of decoupling between the different modules. During forward and backward propagation, GDL performs a learnable affine transformation, $\mathcal{A}$, to the forward feature maps, and simply multiplies the backward gradient by a constant. The effectively decoupling can be carried out. In addition, during the backward propagation, GDL takes the gradient from the subsequent layer, multiplies it by a constant $\mathit{\lambda }\in \left[0,1\right]$ and passes it to the preceding layer. Mathematically, we can formally treat GDL as a pseudo-function ${\mathcal{G}}_{\left(\mathcal{A}\lambda \right)}$ defined by two equations describing its forward- and backward-propagation behavior as follows:

(1) ${\mathcal{G}}_{\left(\mathcal{A},\lambda \right)}\left(\mathit{x}\right)=\mathcal{A}\left(\mathit{x}\right)$(1)

(2) $\frac{\mathit{d}{\mathcal{G}}_{\left(\mathcal{A},\lambda \right)}}{\mathit{dx}}=\mathit{\lambda }{\mathrm{\nabla }}_{\mathcal{A}}$(2)

where ${\mathrm{\nabla }}_{\mathcal{A}}$ is the Jacobian matrix form the affine layer.

In addition, paralleled with box classifier module, PCB is used to further calibrate classifier score. Specifically, the PCB is equipped with a well-trained classification model and a new supporting prototype. Taken region proposals obtained by few-show object detection as input, PCB uses an additional prototype-based pairwise fraction to improve the original logistic regression score. Furthermore, since the PCB module is offline without any further training, it can be plug-and-play and easily equipped to any other architectures to build stronger few-shot detectors.

Multi-Scale Attention Mechanism

The backbone structure of the DeFRCN-MAM model is shown in Figure 3. To improve the expression of industrial defect target features, FPN and RAM are introduces into the classic ResNet-101 model. This improvement can enhance the learning ability of multi-scale features and obtain the correlation information of different channel and spatial feature maps.

Figure 3. The architecture of backbone.

In the task of FSOD, the various domains have different feature distribution and different feature representation. Hence, it is necessary to introduce RAM in backbone to improve feature representation by stacking. The architecture of RAM is shown in Figure 4.

Figure 4. The architecture of RAM.

RAM consists of residual unit, convolutional layer, pooling and upsampling processing, etc. Here, pooling and upsampling processing is used to extract multi-level semantic features. Residual unit can fuse multi-level semantic features and provide wide information to classification. In concretely, f(x) is input feature map of RAM. After three maxpooling and upsampling processing, the different level semantic features of $\mathit{f}\left(\mathit{x}\right)$ can be obtained. Then, attention parameters $\mathit{M}\left(\mathit{x}\right)$ can be extracted. Here, residual unit processing is carried out between pooling and upsampling. The dimensional of feature maps cannot be changed by residual unit, but the feature information is augmented. $f^{\prime }\left(\mathit{x}\right)$ can be obtained by several processing. Then, after feature information enhancement, the output feature map, $f^{\prime \prime }\left(\mathit{x}\right)$, can be obtained as:

(3) $\mathit{f}{ }^{\mathrm{\prime \prime }}\left(\mathit{x}\right)=\left(1+\mathit{M}\left(\mathit{x}\right)\right)\mathit{f}{ }^{\mathrm{\prime }}\left(\mathit{x}\right)$(3)

Experiments and Discussions

Industrial Defect Datasets

At present, the open datasets of industrial defect objects can be divided by the tasks, i.e., supervised classification, supervised detection, supervised segmentation, unsupervised segmentation, and weakly supervised segmentation. In this paper, DAGM 2007 and NEU-DET are taken as the research objects. Here, DAGM 2007 dataset is provided by Competition at DAGM 2007 Symposium, which contains 10 types of defect objects and is published for weakly supervised segmentation task. Its partial data is shown in the Figure 5(a). Among them, 6 classes with 150 defect images are in each category. The other 4 classes have 300 defect images each category. The NEU-DET (Song and Yan 2013) dataset is The Northeastern University Surface Defect (NEU-DET) database, which contains six typical defects of hot-rolled steel strips, i.e., crazing, inclusion, patches, pitted surface, rolled in scale and scratches. This dataset is published for supervised detection task. It is shown in Figure 5(b). It includes 1800 grayscale images with six different types of typical defects, each type containing 300 samples. NEU-DET provides annotations in “xml” format that indicate the category and location of defects in each image. In Figure 5(b), a yellow box indicates the location of each defect.

Figure 5. Open datasets, (a) DAGM 2007, (b) NEU-DET.

The experimental task in this paper are image samples of the flange shaft of automobile parts with defects. According to the detected defects of automobile parts, our team establishes the corresponding data acquisition environment. The collection environment is shown in Figure 6. Samples are collected by five cameras. Hereinto, there are four cameras around parts and one camera directly above it. Then, the data can be directly transmitted to the top computer through the data line. In view of this, we can establish a new part defects dataset.

Figure 6. Part image data acquisition environment.

In our new part defects dataset, there are 8-types defects, i.e., gear, quench, contusion, bump, crack, threaded hole shielding, unthreaded hole and threaded hole. The examples are shown in Figure 7.

Figure 7. Partial samples in defects dataset established by our team.

LabelImg software is used for image labeling of the collected samples. The labeling results are saved as files in “xml” format. It includes coordinate of defects, defects type, image size, image name, etc. The labeling images are about 1600. It is a typical few-shot object detection task, so we need to study FSOD methods.

Experimental Setting

Following the previous work of the researchers, we evaluate the performance of the proposed method by segmentation data in DAGM 2007, NEU-DET, flange shaft dataset established by our team. The data situations are shown in Table 2. The 5421 images in the three datasets are divided into training and test sets in a 1:1 ratio. Then, 16-types data in DAGM 2007 and NEU-DET is base class and data in flange shaft dataset is novel class. Furthermore, we set shots, k = 1, 2, 3, 5 and 10, in few-shot object detection. The assessment metrics are novel mean average precision (nAP), nAP50, nAP75, etc. In addition, the basic network is Faster R-CNN and the backbone is pre-trained ResNet-101 by ImageNet data.

Table 2. Defects data situation.

Classes	Class1	Class2	Class3	Class4	Class5	Class6	Class7	Class8	Class9	Class10
(a) DAGM 2007
Train	75	75	75	75	75	75	150	150	150	150
Test	75	75	75	75	75	75	150	150	150	150
Classes		Crazing		Inclusion	Pitted_surface		Patches	Rolled-in_scale		Scratches
(b) NEU-DET
Train		150		150	150		150	150		150
Test		150		150	150		150	150		150
Classes		Gear	Quench	Contusion	Bump	Crack	Threaded hole shielding		Unthreaded hole	Thread hole
(c) Flange shaft dataset
Train		20	20	20	20	622	14		25	20
Test		20	20	20	20	622	14		25	20

The experimental settings are shown in Table 3. We apply Stochastic Gradient Descent (SGD) to train the network optimization parameters.

Table 3. Training setting.

Name	Parameters
Minimum batch	8
Learning rate of base class	0.02
Learning rate of novel class	0.01
Momentum	0.9
Epoch	50

In addition, the experiment environment is combination of Intel Xeon Silver 4210 2.20 GHz × 20 of CPU and NVIDIA GTX 3090Ti 24GB × 2 of GPU. The implementation system is Ubuntu 18.04.6 LTS. In addition, we use Pytorch 1.7.1, cuda 11.0, cudnn 8.0.5 to establish the deep learning architecture. The training and the fine tuning situations are shown in Table 4.

Table 4. Training and fine tuning situations.

	Base class	Novel class
		1-shot	2-shot	3-shot	5-shot	10-shot
Samples		8	16	24	40	80
Epochs	11000	800	1200	1600	2000	4000

The experimental procedure is as follows:

1. On the basis of pre-trained ResNet-101-FPN trained by ImageNet, 16-types defect samples of DAGM 2007 and NU-DET are used to pre-train it again.

2. Fine tuning is performed based on the pre-divided k-shot samples.

3. Each fine-tuned model is evaluated by test set.

Ablation Study

According to the architecture of DeFRCN-MAM, we can find that the main contribution is the multi-scale feature fusion processing in backbone, i.e., FPN, and the parallel connection of R-CNN head, box classifier and box regression parts. In this part, we do ablation study and show the function of these contributions. The comparison results are shown in Table 5 by average precision (nAP), nAP 50 and nAP75. Here, the compared networks include classical DeFRCN, FPN revised DeFRCN, DeFRCN added parallel connection and DeFRCN with FPN and parallel connection revision.

Table 5. Ablation comparison results.

Shots	Method	nAP (%)	nAP50%)	nAP75%)
1	DeFRCN	14.75	22.12	14.18
	DeFRCN+FPN	14.23	21.85	13.17
	DeFRCN+Parallel connection	13.66	20.58	12.84
	DeFRCN+FPN+Parallel connection	16.65	25.03	16.66
2	DeFRCN	29.57	49.43	28.01
	DeFRCN+FPN	28.41	42.34	29.00
	DeFRCN+Parallel connection	27.21	43.25	26.20
	DeFRCN+FPN+Parallel connection	32.48	46.54	34.61
3	DeFRCN	35.28	53.59	36.04
	DeFRCN+FPN	34.89	48.37	34.73
	DeFRCN+Parallel connection	33.59	50.33	32.50
	DeFRCN+FPN+Parallel connection	39.28	54.22	41.46
5	DeFRCN	44.26	61.76	47.73
	DeFRCN+FPN	48.18	65.61	49.01
	DeFRCN+Parallel connection	45.07	60.76	46.35
	DeFRCN+FPN+Parallel connection	53.26	67.26	56.98
10	DeFRCN	57.43	74.04	61.93
	DeFRCN+FPN	64.15	75.41	67.57
	DeFRCN+Parallel connection	59.79	71.13	64.81
	DeFRCN+FPN+Parallel connection	64.97	76.29	70.91

According to Table 5, we can find that with the assistance of FPN and parallel connection, the nAP, nAP 50 and nAP75 of detection model have superiority in the most conditions. Moreover, the results of FPN revised DeFRCN are the most close to the optimal ones. Hence, it is obviously effective to improve detection results that the multi-scale feature fusion based on FPN. For example, nAP75s of FPN revised DeFRCN are 7.97% and 3.34% less than the optimal ones in 5-shot and 10-shot conditions, but higher than other ones. In short, it is verified that the contribution of multi-scale feature fusion and parallel connection to defect detection by this ablation study.

Detection Results and Analysis

The training process of DeFRCN-MAM in base class samples, which are in DAGM 2007 and NEU-DET datasets, is shown in Figure 8. It is obviously that the loss value of box regression and classification drop firstly sharply and then smooth descent. This phenomenon shows that the training process is normal. Moreover, the learning rate is dynamic. When loss value is relatively big, the learning rate is big too. On the contrary, when the loss value oscillates, the learning rate is small, which can drive parameters learning.

Figure 8. Loss and learning rate of training process based on base class samples.

On the basis of the model trained by base class samples, the fine tuning of model is carried out by samples in novel class. The defects in the novel class dataset are detected by the new model. This part of training process can be shown in Figure 9. We can find that the loss values of box regression and classification drop sharply in the front 1500 epochs and keep stable after that. It illustrates that the proposed model can fleetly transfer to the new dataset samples, i.e. flange shaft dataset. Compared with training process of samples in base class dataset, the fine-tuning model of samples in novel class dataset can be completed in a handful of epochs.

Figure 9. Loss and learning rate of training process based on novel class samples.

The detection results comparison between exposed methods and DeFRCN-MAM model is carried out in this part. Here, the exposed methods include FRCN+ft-full, TFA w/fc, TFA w/cos, few-shot object detection via contrastive proposal encoding (FSCE) and DeFRCN. In different sample quantity conditions, i.e. 1-shot, 2-shot, 3-shot, 5-shot and 10-shot, the detection result are compared, which is shown in Table 6. Here, the assessment metric is the mean average precision (mAP).

Table 6. Detection mAP (%) comparison between exposed methods and DeFRCN-MAM.

Class	Method	1-shot	2-shot	3-shot	5-shot	10-shot
Gear	FRCN+ft-full (Wang et al. 2020)	18.20	27.24	40.75	55.65	77.96
	TFA w/fc (Wang et al. 2020)	18.95	11.00	7.94	3.73	24.97
	TFA w/cos (Wang et al. 2020)	23.02	16.34	9.40	9.94	33.53
	FSCE (Sun et al. 2021)	14.87	21.20	23.26	38.90	68.92
	DeFRCN (Qiao et al. 2021)	16.58	25.49	36.82	39.76	57.28
	Our	24.74	21.08	27.90	39.98	69.01
Quench	FRCN+ft-full (Wang et al. 2020)	8.18	22.68	28.62	28.29	51.91
	TFA w/fc (Wang et al. 2020)	0.91	5.28	4.30	4.17	8.28
	TFA w/cos (Wang et al. 2020)	0.36	3.63	5.16	3.61	7.00
	FSCE (Sun et al. 2021)	5.39	16.93	24.75	32.61	50.78
	DeFRCN (Qiao et al. 2021)	14.44	25.10	27.68	31.34	53.40
	Our	9.41	27.96	29.40	46.47	60.69
Contusion	FRCN+ft-full (Wang et al. 2020)	4.97	10.03	6.76	29.13	35.80
	TFA w/fc (Wang et al. 2020)	0.08	0.78	0.21	0.76	1.16
	TFA w/cos (Wang et al. 2020)	0.37	0.10	0.36	6.71	0.95
	FSCE (Sun et al. 2021)	4.47	5.99	11.35	22.05	37.05
	DeFRCN (Qiao et al. 2021)	4.97	8.50	14.09	26.69	45.47
	Our	4.95	11.00	18.19	34.00	48.64
Bump	FRCN+ft-full (Wang et al. 2020)	9.92	24.90	34.41	39.95	68.47
	TFA w/fc (Wang et al. 2020)	7.07	16.90	12.75	6.74	22.73
	TFA w/cos (Wang et al. 2020)	6.67	1.23	12.10	1.71	23.74
	FSCE (Sun et al. 2021)	7.73	28.58	36.81	41.94	66.32
	DeFRCN (Qiao et al. 2021)	7.92	32.86	37.23	49.97	64.80
	Our	9.90	39.29	40.23	48.45	65.06
Crack	FRCN+ft-full (Wang et al. 2020)	0.89	1.07	1.00	1.80	5.42
	TFA w/fc (Wang et al. 2020)	0.00	0.49	0.06	0.05	0.09
	TFA w/cos (Wang et al. 2020)	0.00	0.02	0.08	0.10	0.11
	FSCE (Sun et al. 2021)	0.98	1.43	1.17	1.50	6.11
	DeFRCN (Qiao et al. 2021)	0.59	0.94	0.98	0.97	2.50
	Our	0.79	0.79	0.79	0.87	4.31
Threaded hole shielding	FRCN+ft-full (Wang et al. 2020)	46.36	57.06	56.54	69.75	81.55
	TFA w/fc (Wang et al. 2020)	36.54	44.22	42.05	55.99	67.82
	TFA w/cos (Wang et al. 2020)	34.01	43.09	49.79	59.60	66.83
	FSCE (Sun et al. 2021)	45.38	53.32	65.37	70.32	79.84
	DeFRCN (Qiao et al. 2021)	45.92	53.73	53.66	67.78	77.65
	Our	48.92	62.14	68.01	71.17	84.40
Unthreaded hole	FRCN+ft-full (Wang et al. 2020)	2.84	19.07	36.64	55.25	70.99
	TFA w/fc (Wang et al. 2020)	1.42	13.04	29.67	34.18	51.86
	TFA w/cos (Wang et al. 2020)	3.70	8.29	15.02	12.39	28.33
	FSCE (Sun et al. 2021)	2.98	41.69	49.60	56.00	78.84
	DeFRCN (Qiao et al. 2021)	13.01	59.35	63.98	76.35	81.28
	Our	10.65	55.29	56.29	70.22	82.22
Thread hole	FRCN+ft-full (Wang et al. 2020)	15.65	32.76	45.15	62.73	75.80
	TFA w/fc (Wang et al. 2020)	4.65	6.62	11.27	21.78	34.77
	TFA w/cos (Wang et al. 2020)	4.61	6.15	8.14	26.71	28.59
	FSCE (Sun et al. 2021)	8.03	11.40	27.55	52.68	72.80
	DeFRCN (Qiao et al. 2021)	14.54	30.57	47.83	61.19	77.04
	Our	19.54	29.39	46.46	62.88	80.04

According to the results of Table 6, it can be seen that the proposed DeFRCN-MAM can obtain the optimal detection results in a variety of defect detection, even if the number of samples is different. In especially, threaded hole shielding defects obtains the optimal mAP value in 1-shot, 2-shot, 3-shot, 5-shot and 10-shot cases, which is higher than the existing methods by 2.56%, 5.08%, 2.64%, 0.85% and 2.85% at least. The advantage of the proposed method is obvious. In addition, for quench, contusion and bump defects, the proposed method has better detection results in most cases. However, for gear, crack and unthreaded hole, the detection effect of the proposed method is inferior to the existing classical methods. From the characteristics of different types of defects, it can be seen that the detection performance of the proposed method is significantly improved for industrial defects with small size and poor clarity. Although the detection effect of the proposed method is not the strongest for defects with relatively strong visibility, such as gear and crack, it is still close to the existing optimal detection results. This experiment verifies that DeFRCN-MAM has excellent detection capability for irregular defect targets.

In order to further prove the detection advantages of DeFRCN-MAM method, it are statistically analyzed that the comprehensive detection effects of different detection methods in the case of multiple data samples. Moreover, the evaluation indexes are nAP, nAP50 and nAP75, respectively. The comparison result is shown in Table 7.

Table 7. Detection result comparison based on several metrics.

Shots	Method	nAP (%)	nAP50%)	nAP75%)
1	FRCN+ft-full (Wang et al. 2020)	13.38	17.89	13.90
	TFA w/fc (Wang et al. 2020)	8.70	15.83	9.86
	TFA w/cos (Wang et al. 2020)	9.09	15.05	10.27
	FSCE (Sun et al. 2021)	11.23	18.64	11.36
	DeFRCN (Qiao et al. 2021)	14.75	22.12	14.18
	Our	16.11	25.43	15.46
2	FRCN+ft-full (Wang et al. 2020)	24.35	33.75	27.28
	TFA w/fc (Wang et al. 2020)	12.20	20.11	12.53
	TFA w/cos (Wang et al. 2020)	9.85	17.33	9.14
	FSCE (Sun et al. 2021)	22.57	33.84	23.98
	DeFRCN (Qiao et al. 2021)	29.57	49.43	28.01
	Our	30.87	44.93	31.26
3	FRCN+ft-full (Wang et al. 2020)	31.24	41.57	34.89
	TFA w/fc (Wang et al. 2020)	13.53	22.56	13.85
	TFA w/cos (Wang et al. 2020)	12.51	21.27	10.38
	FSCE (Sun et al. 2021)	29.98	41.39	33.46
	DeFRCN (Qiao et al. 2021)	35.28	53.59	36.04
	Our	35.91	49.17	37.23
5	FRCN+ft-full (Wang et al. 2020)	42.82	53.02	48.10
	TFA w/fc (Wang et al. 2020)	15.93	26.55	16.79
	TFA w/cos (Wang et al. 2020)	15.09	24.87	16.16
	FSCE (Sun et al. 2021)	39.50	55.70	42.65
	DeFRCN (Qiao et al. 2021)	44.26	61.76	47.73
	Our	46.75	62.70	51.15
10	FRCN+ft-full (Wang et al. 2020)	58.49	67.72	62.75
	TFA w/fc (Wang et al. 2020)	26.46	40.01	30.01
	TFA w/cos (Wang et al. 2020)	23.64	36.30	23.62
	FSCE (Sun et al. 2021)	57.58	71.48	62.60
	DeFRCN (Qiao et al. 2021)	57.43	74.03	61.93
	Our	61.79	73.53	65.71

The results of Table 7 show that the optimal detection results of DeFRCN-MAM are guaranteed even if with different sample sizes, especially under nAP and nAP75 indexes. Under nAP50 index, the detection results of DeFRCN-MAM are also close to the existing optimal detection results. The experimental results further verify that the proposed method has strong industrial defect detection capability.

In addition, Figure 10 shows partial defect detection results in the case of 10-shot. The results can visually show the detection performance of DeFRCN-MAM. Among them, the accuracy of unthreaded hole, threaded hole and quench are above 0.95, which further demonstrate the adaptability of DeFRCN-MAM to industrial defects.

Figure 10. Some detection results of DeFRCN-MAM.

Conclusion

The proposed DeFRCN-MAM is proposed in this method, facing the problems of small samples, weak visibility and irregular shape of industrial defect objects. Here, DeFRCN is the fundamental of the proposed one. Here, the core contribution of DeFRCN, i.e., GCL and PCB, adapts to few-shot detection problem. Moreover, we introduce FPN and RAM in the backbone, to enhance the multi-scale feature expression of the model and the learning of the feature correlation. So that, the proposed model can better adapt to the characteristics of weak visualization and irregular shape of industrial defects. In addition, the feature expression is enhanced by connecting R-CNN head, box classifier and box regression in parallel. In experiments part, we pre-train the proposed model with the public industrial defect datasets, i.e., DAGM 2007 and NEU-DET, as the base class. Then, we take the flange shaft datasets measured in the laboratory as the new class to fine-tune the model and conduct testing of industrial defects. The rationality of DeFRCN-MAM structure is verified by ablation study part. Moreover, the results of DeFRCN-MAM are compared with several classical FSOD methods, including FRCN, TFA, FSCE and DeFRCN. The results show that DeFRCN-MAM has the strongest ability to detect industrial defects with poor visibility and small size, even if the corresponding data volume is different. In especially for threaded hole shielding, the optimal mAP values are obtained under 1-shot, 2-shot, 3-shot, 5-shot and 10-shot, and at least 2.56%, 5.08%, 2.64%, 0.85% and 2.85% higher than other methods, with obvious advantages. However, when detecting defects with strong visibility, the proposed method is still close to the existing optimal detection results. Furthermore, the DeFRCN-MAM, proposed in this paper, has stronger adaptability to industrial defects and better detection effect. This study also provides the design idea of FSOD method for the subsequent industrial defect detection based on deep learning.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The work was supported by Beijing Natural Science Foundation [4202015]; and Chinese University Industry-University-Research Innovation Fund - Blue Dot Distributed Intelligent Computing Project [2021LDA06002].