Development Of A Vision- based Anti-drone Identification Friend Or Foe Model To Recognize Birds And Drones Using Deep Learning

ABSTRACT

Recently, the growing use of drones has paved the way for limitless applications in all the domains. However, their malicious exploitations have affected the airspace safety, making them double-edged weapons. Therefore, intelligent anti-drone systems capable of recognizing and neutralizing airborne targets become highly required. In the existing literature, most of the attention has been centered on recognizing drones as unique airborne target, whereas the real challenge is to distinguish between drones and non-drone targets. To address this issue, this study develops an Identification Friend or Foe (IFF) model able to classify the aerial targets in foe or friend categories by determining whether the aerial target is a drone or bird, respectively. To achieve this objective, artificial intelligence and computer vision approaches have been combined through transfer learning, data augmentation and other techniques in our model. Another contribution of this work is the study of the impact of depth on the classification performance, which is demonstrated through our experiments. A comparison is performed based on eight models, where EfficientNetB6 shows the best results with 98.12% accuracy, 98.184% precision, 98.115% F1 score and 99.85% Area Under Curve (AUC). The computational results demonstrate the practicality of the developed model.

Introduction

Recently, drones have become widespread in several domains thanks to the progress made in Artificial Intelligence (AI), security and transportation fields (Gupta, Kumari, and Tanwar 2021; Kumar et al. 2021; Kurt et al. 2021; Serrano-Hernandez, Ballano, and Faulin 2021; Spanaki et al. 2021). Thus, they are considered a revolution of the 21st century. With the rapid development of intelligent techniques that range from image and speech recognition to self-decision-making in autonomous robots, drones have benefited from it. Moreover, they are demonstrating their high ability to provide real-time and cost-effective solutions in a variety of areas including security, surveillance and smart city monitoring (Kangunde, Jamisola, and Theophilus 2021; Lin Tan et al. 2021).

Nonetheless, they may be exploited maliciously and anarchically for miscellaneous hostile objectives such as spying, smuggling drugs and contraband, cyber-attacks and terrorism. Their overnight invasion and unrestricted deployment are threatening public safety and security. Thus, drone activities need to be controlled properly and efficiently. This can be ensured only by the anti-drone systems, which are cutting-edge devices relying on a reverse working process to take down the aerial target lawfully and safely (U.S.C. Title 49 - TRANSPORTATION, n.d.). In fact, the anti-process goes through four main phases, namely, Detection, Identification, Tracking and Interception (DITI) as explained in (Yasmine, Maha, and Hicham 2022).

The two first phases aim to recognize the airborne target in real-time and with high confidence scores. The detection and identification seek to depict and confirm a foreign presence in the sky based on the generated signals (Park et al. 2021; Taha and Shoufan 2019) and to label the intruder object to “Friend or Foe” category. Knowing the degree of lethality and hazard of the potential target, the identification should be done accurately for safety and security issues, since false alarm responses due to missed or confused detections may lead to an unsuccessful anti-drone process and a doomed interception that could have weighty damages for both the target and the environment under operation.

In general, within safeguarded areas and critical infrastructures, the use of drones improperly or maliciously can pose significant privacy and security risks (Kolamunna et al. 2021). As drones have become more prevalent among civilians, the related technical, security, and public safety issues must be addressed, regulated, and prevented (Al-Sa’d et al. 2019). Thus, anti-drone systems are used to counter potential threats posed by unauthorized or malicious drone activity, especially in highly sensitive locations such as airfields and military bases (Allahham, Khattab, and Mohamed 2020). The effectiveness of an anti-drone system often relies on the incorporated recognition module to provide comprehensive coverage and accurate identification of drones. In addition, the analysis of the drone’s payload allows to determine the lethality degree of the drone. Furthermore, the anarchic and rogue drone deployment poses potential collisions, environmental disturbances, airspace distress, privacy threats, security concerns and safety issues. Therefore, an Automatic Drone Identification (ADI) protection mechanism is proposed in (Svaigen et al. 2021) to detect and avoid malicious drones using the generated acoustic signals. Another paper (Casabianca and Zhang 2021) leverages the acoustic signal to develop a CNN in order to improve the drone’s detection performance.

Actually, an anti-drone should be able to recognize the main types of airborne targets and especially distinguish between the prevalent ones; which are the drones and birds. Since they share the same airspace and altitudes; mainly the low altitude airspace up to 32 000 ft as an upper limit. At this altitude, recognizing the flying targets turns out to be a real challenge regarding their similarities which increases the likelihood of false detections.

It is a significant task for an anti-drone system to distinguish accurately between drones and birds. Bird flights can cause far-reaching impacts within safeguarded areas and critical infrastructures such as airports and military bases. Thus, it is of high importance to effectively distinguish between birds and drones to avoid doomed interception and collateral damages where anti-drone measures are deployed. For instance, bird strikes with deployed drones and airplanes can result in severe damage to the environment, which poses a safety concern and a potential accident. The collision of birds with drones can damage the frame and the parts of the drone, compromising flight safety. Efforts to mitigate any potential negative impacts on birds involve the use of an appropriate recognition model within the anti-drone system. The birds can be considered as false-positive detections for anti-drone if the used recognition model determines the bird as a drone and thus the system is falsely triggered causing a failed neutralization. Due to the complexity of the system, this might cause unnecessary alerts or interruptions. As anti-drone technologies aim to detect and mitigate threats from drones, they must distinguish them from harmless airborne objects, such as birds.

Furthermore, detecting the identity of the airborne targets is crucial since the drones and birds have similar movement behaviors, radar-cross sections, very close flying altitudes and speeds, and also their radar signals share similar signal amplitude as well as the fluctuation of time series and spectrum structure (Gong et al. 2019). Moreover, when the distance increases, the generated radio and acoustic signals cannot be recognized and distinguished properly (Alaparthy, Mandal, and Cummings 2021; Fuhrmann et al. 2017; Patel, Fioranelli, and Anderson 2018; Torvik, Olsen, and Griffiths 2016). For this reason, a thorough study of the detection and identification model is of great importance in reinforcing aerial security and building effective anti-drone systems. Based on the research work done in [10,17–19], the visual-based detection is the most advantageous in view of the quality and quantity of information delivered by the Electro-Optical (EO) and Infrared (IR) sensors. Recent visual recognition tasks with image detection and tracking use Convolutional Neural Networks (CNN) as a foundational part to process and extract the visual features from the input images to provide a probability distribution over a set of categories (Isaac-Medina et al. 2021).

Further, visual detection methods using suitable AI algorithms are often considered one of the most accurate means for anti-drone detection due to several reasons such as:

• Adaptability and flexibility of deployment in various locations and weather conditions.

• Reduced false alarms since the visual detection methods are the most suitable to distinguish between drones and other flying objects, reduce false alarms and misidentification of non-threatening airborne targets, e.g. birds.

• Real-time and effective identification of the visual methods allows the anti-drone automated system to visually confirm the presence of a drone or not rapidly with the use of a suitable pre-trained model based on the related requirements and challenges. Thus, the incorporated identification models can rapidly identify the type of the airborne intruder and assess its potential threat level.

Furthermore, the collected images and videos are more reliable to recognize and to fit the intruder into its defined class « drone or bird » by gathering the visual cues; such as the appearance feature, e.g., colors, contour lines, geometric forms and edges, etc., and the motion across consecutive frames (Shi et al. 2018). Recent advances in AI and computer vision have reinforced the visual detection module in the anti-drone systems, in terms of accuracy and processing time. The integration of AI algorithms has enabled to take a significant step forward in making the anti-drone system intelligent, flexible and autonomous.

To address the existing challenges, we have combined AI techniques, computer vision approaches and deep learning to develop a novel recognition model aiming to distinguish between the prevalent aerial targets in the sky.

In this work, we develop an advanced detection model able to distinguish between drones and birds. Actually, computational experiments are conducted by training, validating and testing the model on a real-world dataset. The computational results demonstrate the practicality of the proposed model.

The main novelty and contributions of this paper can be summarized as follows:

• Proposing a backbone model suitable for anti-drone deployment.

• Developing an Identification Friend or Foe (IFF) backbone model that serves as a binary classification backbone model in an anti-drone system to recognize the types of the aerial targets.

• Study the effect of the depth of the models on the performance.

In the following, Section II provides a summary of the state-of-the-art studies on airborne target recognition, whereas the problem-solving methodology is presented in Section III. Section IV highlights the proposed system setup with the used techniques and parameters. The experimental results and the comparison with the benchmark are detailed in Section V. Finally, we conclude and address the advantages and limitations of the proposed model as well as the perspective research directions in Section VI.

Related Work

To date, the recent advances in AI and Machine Learning (ML) have significantly accelerated the improvements done on both drones and anti-drone systems. Thus, the anti-drone systems have become intelligent and autonomous in terms of decision-making. Moreover, their capabilities are enhanced with the fusion of AI approaches with deep learning algorithms, computer vision techniques and more precisely image recognition.

The four phases of the DITI process aim to automate some or all the operations to improve the system performance using AI techniques. Thus, the anti-drone becomes a cutting-edge device by adopting and fusing knowledge from AI, Internet-of-Things and robotics to respond effectively to the security threats posed by the malicious drones (Choi 2022; Ding et al. 2018). Since the anti-drone relies mainly on the detection model used, it is important to select suitable algorithm and methodology, optimal parameters and appropriate data to achieve satisfactory results.

One cannot talk of AI and computer vision without talking of image recognition and classification algorithms and CNN architectures. The achieved advances have reinforced the effectiveness of the visual detection in anti-drone systems, in terms of accuracy and processing time. The drone recognition have shifted from using traditional methods that use low-level handcrafted features and classical classifiers (Ganti and Kim 2016; Gökçe et al. 2015; Lai, Mejias, and Ford 2011; Rozantsev, Lepetit, and Fua 2016; Unlu, Zenou, and Rivière 2018; Wang et al. 2018; Wu, Sui, and Wang 2017) to more automated ones represented most of all by deep neural networks (Wang, Fan, and Wang 2021) considered as a “black-box” solution for most of the problems (Osco et al. 2021; Seidaliyeva et al. 2020). In fact, the anti-drone system effectiveness depends mainly on the reliability and validity of the results delivered by the recognition model.

For this reason, several later research studies have focused on developing novel drone detection strategies by leveraging different AI approaches with the purpose of reaching high-level confidence results.

In the following, we review the existing models in the literature as single and binary airborne target recognition models, which include detection and classification.

Single Target Recognition Models

The single airborne target recognition refers to drone recognition as a single target. Indeed, the proposed methods train the models only on one target class, i.e., drones. The latter is a limitation for anti-drone deployment in view of the omnipresence of the birds in the sky.

In (Garcia, Min Lee, and Kim 2020), the authors used Faster R-CNN with ResNet-101 to detect drones, reaching an accuracy of 93.40%. However, the developed model is not trained on the bird dataset and thus it is unable to properly identify birds. Using visible-band and infrared datasets, the paper (Isaac-Medina et al. 2021) has shown that the precision counts more than the time when detecting small drones with the selected Faster R-CNN that yields the best precision rate. The proposed model didn’t find the optimal performance compromise to detect different sizes of drones. Further, the authors in (Behera and Bazil Raj 2020) have trained Yolov3 during 150 epochs to detect drones. However, the overall performance does not fulfill the anti-drone requirements. Further, the paper (Ashraf, Sultani, and Shah 2021) has demonstrated that a two-stage approach performs better than the one-stage one for drone detection thanks to its small memory consumption, fast speed and excellent capability of capturing detailed spatio-temporal information. Another drone detection method presented in (Zeng et al. 2021) has tested three CNN namely, R-CNN, YOLOv3, SSD with different backbone architectures, i.e., VGG16, ResNet 50, DarkNet 53 and DenseNet 201. Results showed that all the models achieve high results rates as long as the dataset is representative and varied. However, there is a need to finetune the model on the anti-drone research task. Deep reinforcement learning was also used and implemented with Transfer Learning (TL) for drone detection and guidance as proposed in (Çetin, Barrado, and Pastor 2020). TL improves the overall performance since it was faster in reaching the threshold reward around 30K steps with a higher asymptotic performance. Another paper (Çetin, Barrado, and Pastor 2021) has shown that the combination of Darknet-53 and EfficientNet-B0 shows good results in terms of precision measured.

Recently, the authors in (Fujii, Akita, and Ukita 2021a) have proposed to use CenterNet model to detect only birds without considering drone class. They have shown that the selection of appropriate data augmentation techniques has a direct impact on the detection performance, e.g., random flip, random scaling, random cropping. The developed model achieved 72.13 mAP.In (Zheng et al. 2021), the authors have focused on proposing a dataset that covers a wide range of practical scenarios with viewing angles, different background scenes, relative distance, flying altitude, and lightning conditions. Eight representative deep learning models have been evaluated to assess the key challenges in the problem of air-to-air UAV detection. RetinaNet model showed the best performance. However the ground-to-air detection is not considered. Based on an extensive experimentation comparison, the paper (Zhao et al. 2022) has selected Faster R-CNN with VGG16 as a backbone. The selected model is fused with a tracking algorithm to detect and track the target with to keep it within the field of view. Similarly, the paper (Ge et al. 2022) has integrated a deep Neural Network (DNN) as visual target detector with the Kalman Filter tracking model.

Recently, the paper (Lee, Chin, and Ho 2023) has proposed a novel method based on the use of pre-trained model and Yolov4 algorithm with the Kalman Filter to track the target motion. The model performance achieved an average precision of 90.04%. However, the model is developed based on only the drone class.

Binary Target Recognition Models

Binary recognition models make reference to drone and bird recognition models based on a twofold input dataset.

In (Mahdavi and Rajabi 2020), the authors have shown that CNN models outperform Support Vector Machine (SVM) and K-Nearest Neighbors (KNN) in detecting drones using extracts from films in which drones and birds appear, especially when increasing the convolutional layers and epochs that has direct impact on the accuracy. However, the used dataset lacks of variability with respect to the most encountered drones and birds in the sky. The authors in (Upadhyay, Murthy, and Raj 2021) have used a 15-layer series CNN to classify the airborne targets as drones and birds. The model achieved 87.5% accuracy. However, the developed model is not suitable for real-time deployment due to the included data acquisition module and the processing time of the developed model. In order to upgrade the visual automated methods integrated into the anti-drone system, the paper (Oh, Lee, and Kim 2019) has proposed to examine the most suitable model for drone classification by comparing the classification accuracy, processing time, and network size of the representative CNN models. The experimental results have shown that the shallow and simple structure models produce better training results when classifying a small number of labels. Knowing that time is a significant challenge when detecting airborne targets, there is a need to minimize the inference time without affecting the performance of the model.

The work reported in (Samadzadegan et al. 2022) has implemented binary detection and recognition with YOLO v4 to make accurate distinction. Their proposed method sequentially achieved 84% mAP, 81% IoU, and 83% accuracy, which can be improved to reach higher detection performance that is highly required to reinforce airspace safety. The work reported in (Pawełczyk and Wojtyra 2020) addresses a new drone dataset tested on MobileNetv1 model, resulting in an accuracy and F1 that does not exceed 70% and 60.2%, respectively.

In 2019, IEEE International Conference on Advanced Video and Signal-based Surveillance (AVSS) has launched the first edition of Drone-vs-Bird Detection Challenge, whose primary objective was to correctly detect drones and birds appearing in video sequences without false alarms.

The winning team of the first edition (Coluccia et al. 2019) has proposed the use of two separate CNN; U-Net and ResNet. The proposed model has reached 0.73 F1 rate; exceeding thus the other algorithms. In the second edition (Coluccia et al. 2021), the combination of Cascade R-CNN architecture with ResNeXt as backbone obtained the best overall average precision performance of 80%. The paper (Coluccia et al. 2022) presents the results of the 5th edition of the “Drone-vs-Bird” detection challenge with the use a new dataset comprising 77 video samples of eight commercial drone and two fixed wings. The first ranking team proposed the use of Yolov5 model integrating synthetic data, negative images during training to reduce false-positive predictions and a scoring method to assess the performance of a track (Akyon, Altinuc, and Temizel 2022). The team achieved the best performance in terms of the adopted performance metrics: 0.910 recall and 0.796 AP. In order to integrate these models as recognition modules within anti-drone systems, there is a need to adopt the models to the anti-drone requirements and deployments. Also, it is important to address the overfitting challenge which has not been investigated and detailed accordingly in the existing related work. Table 1 highlights the main contributions as well as the advantages and limitations of the mentioned research works in this section.

Table 1. Comparative table of the related work.

Paper	Proposed method and AI approaches	Size of the dataset	Advantages	Limitations
(Garcia, Min Lee, and Kim 2020)	Combining Faster R-CNN with a ResNet-101 as backbone network.	NA	Good detection performance	Assessing the model with only one metrics. Model unable to detect birds
(Isaac-Medina et al. 2021)	Using Yolov3 and DETR algorithms	NA	Cross modality detection with optical and infrared signals	The dataset includes only one target class.
(Behera and Bazil Raj 2020)	Yolov3	NA	Explainable insight related to the model used	The detection is relatively low for anti-drone system requirement
(Ashraf, Sultani, and Shah 2021)	Two-stage model	NPS and FL drone dataset	Using a segmentation-based approach with pixel and channel-wise attention	Lack of the used experimental and computational parameters
(Zeng et al. 2021)	Faster R-CNN, SSD, YOLOv3	13 803 images	Using CNN for object detection backbone.	Need to finetune the selected model and the used dataset on the research task
Çetin, Barrado, and Pastor (2020)	Reinforcement learning	AirSim environments	Use of TL to improve the detection performance	Bird class is not considered during the development of the model
(Çetin, Barrado, and Pastor 2021)	Darknet-53 and EfficientNet-B0	2,000 images	Real-time deployment	No sufficient learned knowledge
(Fujii, Akita, and Ukita 2021b)	CenterNet	21 837 images	Use of varied bird dataset	Drone target class is not considered
(Mahdavi and Rajabi 2020)	8-layers CNN model	712 images	Experimental comparisons with SVM and KNN	The used dataset is not representative
(Upadhyay, Murthy, and Raj 2021)	15-layer CNN model	Micro-doppler signals	Detecting drones and birds using the generated radar signals	Not suitable for real-time deployment
(Oh, Lee, and Kim n.d..)	Alexnet, Resnet, Squeeznet and other models	7000 images	Experimental comparison of several deep CNN models.	Time consuming model
(Samadzadegan et al. 2022)	Yolov4	10000 images	Efficient detection in crowded backgrounds and high similarity between drones and birds	Relatively low achieved confidence scores
(Pawełczyk and Wojtyra 2020)	MobilNetv1	56 821 images	Use of a pre-trained model on COCO dataset and finetuning it on the used dataset	Need to include more confidence metrics and improve the detection results.
(Lee, Chin, and Ho 2023)	Yolov4	9 332 images	Use of pre-trained model with the detection and interception modules	Considering only classes of drone category
(Zheng et al. 2021)	RetinaNet	13 000 images	Comparing eight deep learning models	Not including ground-to-air detection
(Zhao et al. 2022)	Faster-RCNN-VGG16	10 000 images	Fusing detection and tracking algorithms	Considering only classes of drone category
(Ge et al. 2022)	DNN with Kalman filter	NA	Effective detection in challenging scenarios	Tracking information is not investigated

In this paper, we have focused and worked on the existing technical and conceptual gaps in the detection of aerial targets to respond to the existing needs. Therefore, we have designed a binary classification backbone model for an anti-drone system to determine whether the aerial target is a drone or a bird. Indeed, drones and birds are the most encountered airborne targets in both urban and rural areas. We have experimented an advanced model by selecting a representative dataset, integrating the most recent computer vision techniques and comparing several deep learning models to find the best combination.

Using a backbone model in anti-drone system is significantly important for an effective target detection and identification. The combination of this backbone model with the real-time detection model generates results which are assessed and considered during the tracking and interception phases.

In the following, we are going to develop an Identification Friend or Foe (IFF) model to determine if the aerial target is a drone or bird using transfer learning, optimal hyperparameters and suitable state-of-the art architectures.

Proposed Methodology

In most previous studies, most of the attention has been centered on improving detecting drones as unique targets while the real challenge consists of distinguishing drones and non-drone objects.

In this study, we develop a proper model able to recognize and distinguish drones and birds. Also, we propose to enhance the recognition system by developing a twofold dataset representing the largest combination of backgrounds and foregrounds, in different environments and weather conditions.

Some of the challenges in visual-based drone detection and recognition are:

1. Resemblance in physical and behavioral attributes of drones and birds at high altitudes and long distances.

2. Existence of drones in a populated background, changing weather conditions, varying resolutions, and different illumination situations.

3. The difficulty of identifying drones and birds to their small size.

To address these challenges, different data augmentation techniques are applied along with transfer learning to improve visual recognition.

Data Acquisition

In supervised learning, all is about learning from the data and the model are data driven (Taha and Shoufan 2019). The training process learns the input output relationship and maps the function that binds the output to the inputs thanks to the acquired knowledge going from the obvious to underlying patterns that results in mapping a function between the input and the output. Once trained, the model makes prediction by assigning the unseen data to the category it belongs to using the acquired knowledge.

Thus, dataset preparation is a crucial task for our supervised image recognition task. In fact, we are looking for both high quality and large quantity of data for creating a model with minimal bias and variance, representing a wide range of real cases.

In this work, we have carefully collected images with different types of drones and birds with the purpose to highlight the maximum possible combination of the most encountered aerial targets in different environments. We have selected 20 000 instances mainly from (Caltech-UCSD Birds-200-2011, n.d; Pawełczyk and Wojtyra 2020). The dataset includes different types and categories of drones and birds captured in different altitudes, weather conditions and locations. It is important to note that the selected dataset ensures that the trained model can adapt to a variety of situations and environments.

Further, to overcome overfitting and to improve the generalization ability of the model, we have used three data augmentation techniques through changing brightness range, applying zoom effect and horizontal flip on the images. In this way, each image generates three augmented ones, added to the original one, resulting in increasing the whole database four times. To the best of our knowledge, we have used the largest database of images of drones and birds for a binary classification model.

The Figure 1 shows a selection of drones and birds images used.

Figure 1. A selection of bird and drone images from the collected twofold dataset.

The selection of appropriate training, validation, and testing configurations is an important step to ascertain optimal performance without overfitting. This is performed by dividing the dataset into 3 sets: training, validation, and testing. Assuming that there is a need for hyperparameter tuning, a validation set creation is necessary aside from the testing set.

The training, validation, and testing were chosen, and the probability was set to 0.7, 0.2, and 0.1. A probability of 0.7 means that 70% of the images will be tagged for training, 20% of the images will be assigned for validation, and the remaining 10% of the images will be tagged for testing.

Transfer Learning and Pre-Trained Models

Transfer Learning Process

To avoid training models from scratch and to address the problem of data scarcity and resource inefficiency, the visual-based detection methods leverage massively the Transfer Learning (TL) approach. Accordingly, TL is incorporated to improve the performance of an original model, the speed and accuracy of the new recognition tasks using the knowledge gained from a training recognition source on a similar target task (Hua et al. 2021). In visual automated detection, the transfer learning addresses cross domain learning problems by extracting visual features from data in a related source task and transferring them on a target one (Ling, Zhu, and Li 2015); which improves significantly the generalization ability of feature extraction process of the pre-trained model (Exploring the Efficacy of Transfer Learning in Mining Image-Based Software Artifacts | Journal of Big Data | Full Text, n.d.; Yang et al. 2021). Indeed, the acquired knowledge is transferred from a pre-trained model on a large dataset source; for instance ImageNet, to a new target task. The use of pre-trained models assures the extraction of the fundamental visual features on the new target domain, in this case, airborne targets model. This is achieved by applying the preliminary learned parameters from the pre-trained model on the new one which reduces data requirement; so that the final trained model is both accurate and fast (Zhu et al. 2021). The used ImageNet dataset was created to further the visual recognition tasks, and it represents the largest database spanning for 1000 object classes (Krizhevsky, Sutskever, and Hinton 2017).

In our binary classification problem, there are two unseen target classes: the drone and birds classes. The used transfer learning is established by freezing much of the convolutional layers by using them as feature extractors without changing the learned parameters, then fine-tune the last ones on our targeted dataset by updating their weights during backpropagation to customize the model with the new recognition categories (Imai, Kawai, and Nobuhara 2020). Figure 2 highlights the process of knowledge transfer applied on our classification problem. Figure 2(a) shows the training phase in which the parameters are trained and fine-tuned. On the other hand, Figure 2(b) illustrates the testing phase which provides the class probability of the target.

Figure 2. Transfer learning framework used from pre-trained models to our target model; divided on a) a training and b) a testing phases.

Backbone Pre-trained Models

So far, there are several powerful deep learning models used for visual object recognition tasks that achieved outstanding performance. We are going to test four architectures that have already demonstrated their effectiveness, with different depths. Further, these models depend mostly upon the provided visual information and their resolution (Kannojia and Jaiswal 2018), which is satisfied by our dataset. The pre-trained models are selected on the basis of their architecture and performance in image recognition since their learned parameters and knowledge are transferred and fine-tuned on the drone-bird recognition task.

We implement two versions of each pre-trained model using different number of layers so that the depth is increased to analyze the difference and study the impact. The selected architectures are VGG, ResNet, DenseNet and EfficientNet.

System Setup

In this section, we detail the used setup and configuration of the proposed model, adjusted according to the anti-drone requirements and needs.

Experimental Design

The proposed models are trained on a laptop a NVIDIA® GeForce RTX 3050, Intel® Core i7 -11,370 H Processor 3.3 GHz 12 M with 16GB of memory and desktop NVIDIA Quadro P4000, Intel(R) Xeon(R) W-2155@ 3.30 GHZ with 32GB of memory and Windows as OS. Our experiments are executed using Tensorflow deep learning framework.

During our training process, the parameter and hypermarameters are carefully selected and then tested to be in accordance with the binary classification task. Table 2 details the used parameters and hyperparameters. Actually, the parameters are determined from the general context. Meanwhile, the hyperparameters are adjustable parameters that are tuned several times before finding the optimal ones.

Table 2. Parameters and hyper-parameters setting.

	Parameters	Value/type
CNN Parameters	Epochs	[20, 30, 40, 50]
	Batch size	16
	Image size	224, 224, 3
	CNN classes	2
	Loss Activation	Binary cross entropy Relu and Sigmoid
	Optimizer	Adam
	Dataset size	28 600 images
	Metrics	Accuracy, F1 score, precision, confusion matrix
CNN Hyperparameters	Learning rate Drop factor Learning rate	0.3 0.0001
	Fine-tuning layers	GlobalAveragePooling, Dropout and Dense
	Decay	Learning rate/epochs
	Dropout	0.2

Moreover, layer regularization techniques are used to avoid the overfitting risk and to speed up the training process, which results in lesser variance. In this work, we have found that dropping randomly 20% of the neurons at each iteration is the optimal value for our problem.

Since, we are developing a backbone model to be integrated within the anti-drone system, it’s important to save the best performing weights of the model for each epochs under h5 files by using ModelCheckpoint technique. Further, the EarlyStopping allows to stop training when the performance is getting worse, especially when the model stops improving with the purpose to retain the optimal generalization performance. After several trials, the early stopping patience is adapted to each set of epochs when the approximation and complexity errors get close to each other detect and the dominance of the variance part begins as explained in details in (Prechelt n.d.).

System Model

The overall research flow diagram with the significant followed steps is illustrated in Figure 3. We start by feeding the selected dataset to the model which is augmented and split into three sets: training, validation and testing. After, the training process starts using the specified parameters and techniques. The validation process is done in parallel to assess the reliability of tuned hyperperparameters. Thereafter, we test the model on an unseen set of images to get the performance of the model with numerical confidence scores and visual results.

Figure 3. Proposed research methodology.

Evaluation Protocol

In this paper, we are using predefined metrics to assess the proposed models.

Used Metrics

In order to evaluate the performance of the proposed classification models and methods, we adopt accuracy, precision, F1 score and confusion matrix as performance metrics. Indeed, these metrics allow better assessment of the models and methods in the case of binary classification problems since they rely on the verified and missed detection with True Positives (TP) which are drones, True Negatives (TN) representing the Birds, False Positives (FP) and Negatives (FN) are the falsely classified drones and birds. Table 3 details the relation between true and predicted labels.

Table 3. True and false responses.

	Predicted Labels
		Drone	Bird
True Labels	Drone	True Positive (TP)	False Negative (FN)
	Bird	False Negative (FN)	True Negative (TN)

The used metrics are defined by the following equations.

The accuracy indicates the number of successful detected drones and birds.

(1) $\mathrm{Accuracy}=\frac{\sum TP+TN}{\sum TP+TN+FP+FN}$(1)

The precision and recall show the validity of the positive detected aerial targets.

(2) $\mathrm{Precision}=\frac{\sum TP}{\sum TP+FP}$(2)

(3) $\mathrm{Recall}=\frac{\sum TP}{\sum TP+FN}$(3)

The F1 score measures the proposed models’ accuracy. Usually, it is used in the case of binary classification, with positive and negative samples.

(4) $\mathrm{F}1\mathrm{score}=2.\frac{\mathit{precision}\times \mathit{recall}}{\mathit{precision}\times \mathit{recall}}$(4)

The confusion matrix presents the rate of true positive and false positive in the true and predicted classes of the two classes in a matrix format.

Experimentation and Results

In this section, the experiments of the proposed methodology are presented and explained through numerical results, graphs and prediction visualizations. Also, we discuss the impact of the depth of the pre-trained models on the overall performance. At the end, we present ablation experiments and comparison between our proposed model and the existing benchmark instances.

Impact of the Depth

Several researches have demonstrated that deeper architectures their ability to create profound analysis of the abstract representations of the input and to fit classifications functions better as pointed out in (Zhong, Ling, and Wang 2019). In fact, deeper models perform better in the presence of larger datasets (Schindler, Lidy, and Rauber n.d..). Thus, we have tested to deepen the architectures by increasing the number of Hidden Layers (HL). The selected architectures are VGG16, VGG19, ResNet50, ResNet152, DenseNet121, DenseNet201, EfficientNetB1 and EfficientNetB6. The results of Table 4 show that when the number of HL increases, the performance improves significantly. It is remarked that there is an improvement of the metrics when the model version gets deeper. According to Table 5, ResNet has proven to show the largest improvement when increasing the depth of the architecture with about 47.25% on average. Because DenseNet and EfficientNet have already an important number of HL with 121 and 237, respectively, they didn’t show much improvement with only an average of 1% and 0.3% only. Thus, it is shown that the overall performance depends mostly on the depth of the model. Indeed, the deeper architectures perform better till a fixed threshold depth where the models’ performance don’t increase much anymore. This transition is the stagnation phase. Further, we have found that the threshold depth for this binary classification problem is fixed at 100 layers with a margin of ±5% common and applied to all the models. Thus, EfficientNet-B1 and EfficientNet-B6 are retained as the best performing models. This is explained also by the considerable number of parameters that are used during the training with trainable and non-trainable parameters. The used parameters by the pre-trained models are shown in Table 6. The non-trainable parameters refer to the feature extraction process where the parameters are kept frozen and used as they were trained on the pre-trained model while the trainable parameters refer to the parameters used when fine-tuning the model on our target dataset. The VGG models are fine-tuned in full due to their small number of layers. From this table, it is shown the number of layers and parameters has a high impact on the performance since the largest models extract the maximum amount of informative and discriminative features.

Table 4. Implementation of the models during 20 epochs.

Architecture	Accuracy	Precision	Recall	F1 score
VGG16	0.7795	0.7804	0.7795	0.77932
VGG19	0.82692	0.8271	0.82692	0.8269
ResNet50	0.51	0.51	0.51	0.509
ResNet152	0.96654	0.96764	0.96654	0.96652
DenseNet121	0.95654	0.95655	0.95654	0.95654
DenseNet201	0.96769	0.96898	0.96769	0.96767
EfficientNetB1	0.97808	0.979	0.97808	0.97808
EfficientNetB6	0.98115	0.98184	0.98115	0.98115

Table 5. Impact of the depth.

Architecture/ Metric	VGG	ResNet	DenseNet	EfficientNet
Accuracy	5.73%	47.23%	1.15%	0.3%
Precision	5.64%	47.29%	1.28%	0.2%
Recall	5.73%	47.23%	1.15%	0.3%
F1 score	5.75%	47.33%	1.15%	0.3%

Table 6. The configuration of the pre-trained models.

Model	Layers	Trainable parameters	Non-trainable parameters	Total parameters
VGG16	16	3,323,842	0	3,323,842
VGG19	19	2,160,962	0	2,160,962
ResNet50	50	18,064,386	42,880	18,107,266
ResNet152	152	52,749,314	141,184	52,890,498
DenseNet121	121	6,719,810	79,104	6,798,914
DenseNet201	201	17,535,554	220,928	17,756,482
EfficientNetB1	237	5,361,106	55,015	5,416,121
EfficientNetB6	813	36,413,258	211,767	36,625,025

Performance Behavior

Founding that we have experimented a stagnation phase starting from a common threshold depth, we will use graphical analysis of the performance of EfficientNetB1 and EfficientNetB6 to make better evaluations since their results are close. Figure 4 shows the performance behaviors in terms of accuracy, loss and Area Under Curve (AUC) during the first 20 epochs. The generated graphs have satisfactory behaviors where the accuracy and AUC are exponentially increasing without overfitting while loss is decreasing promptly till approaching zero. However, EfficientNetB6 reaches higher values in terms of accuracy, loss and AUC. This can be explained by its deeper architecture and significant number of layers and also its computational time which is high compared to the other models and mostly EfficientnetB1, as it is depicted in Table 5. This confirms the reliability of the retained model. Further, we have investigated their behavior during 50 epochs, and it is confirmed that EfficientNetB6 has the better performance as shown in Figure 5.

Figure 4. Performance progress during 20 epochs of a) EfficientNetB1 and b) EfficientNetB6 models.

Figure 5. Evolution of the performance during 50 epochs.

Ablation Experiment

As explained earlier, we have selected carefully appropriate Data Augmentation (DA) and Fine-Tuning (FT) regularization techniques for developing the aforementioned model. In order to assess and verify their respective contributions in the overall performance, we have conducted ablation experiments and comparison with the proposed model. The ablation experiment is presented in Table 7.

Table 7. Ablation experiments.

Test	DA	FT	Accuracy	Precision	F1 scoore
1	$\times$	$\times$	0.93115	0.93949	0.93083
2	yes	$\times$	0.96308	0.96531	0.96303
3	$\times$	yes	0.97923	0.97994	0.97922
Proposed model	yes	yes	0.98115	0.98184	0.98115

We have tested our retained model without DA and FT techniques, and then with each of them separately to analyze their impact. Based on the presented results, it can be seen that the integration of each technique increases significantly the results.

The integration of DA and FT separately and both of them have increased the results by 3.3%, 4.9% and 5%, respectively, in terms of accuracy. This validates their important impact on the proposed model.

Considering different altitudes, flight patterns, and environmental conditions, the dataset reflects the diversity of distances encountered in practical settings. Across these varied distances, our model has been trained to recognize and differentiate drones and birds. As result, we observed that the model’s accuracy may vary with distance. Model accuracy is enhanced at shorter ranges, where detailed features can be captured, but decreases slightly at longer distances due to reduced feature resolution. Therefore, to mitigate the impact of different distances on classification accuracy, our training process augmented the dataset to include observations from a variety of distances and incorporated techniques to make the model more robust against distance-related variations.

Comparison with Benchmark

The results of our retained EfficientNetB6 model and the aforementioned existing models discussed in Section II are presented in Table 8. To the best of our knowledge, we confirm that our proposed model achieves the highest confidence score in terms of accuracy, precision, recall and F1 score and outperforms the other models. This is explained by integrating different AI approaches during the training process such as applying data augmentation on the varied dataset, adopting an appropriate methodology that corresponds to the current recognition problem as well as using transfer learning approach from EfficientNet-B6 pre-trained model which is already trained on ImageNet dataset and then fine-tune it on our airborne target dataset. Also, the selected combination of the trainable and non-trainable parameters has proven its effectiveness on our model. Further, the recognition models are data driven requiring a considerable and satisfactory dataset in terms of quality and quantity to enhance their performance and make them more robust.

Table 8. Comparison between our proposed model and the existing ones.

Papers	Accuracy	Precision	Recall	F1 score
(Samadzadegan et al. 2022)	83%	84%	84%	83%
(Pawełczyk and Wojtyra 2020)	70%	-	-	60.2%
(Coluccia et al. 2019)	-		-	73%
(Coluccia et al. 2021)	-	80%	-	-
(Coluccia et al. 2022)	-	0.796	0.910	-
Our model	98.115%	98.184%	98.115%	98.115%

The combination of all the aforementionned approaches has significantly enhanced the performance of the selected model when recognizing drone and birds.

The confusion matrix of the EfficientNetB6 model obtained during the test step is presented in Figure 6 with the rate of correct and false prediction.

Figure 6. Confusion matrix of EfficientNetB6 model.

Prediction Visualization

Finally, we have integrated a visualization technique in order to illustrate how the models are performing the classification of drones and birds based on the true and false predictions; which are the key elements behind the adopted metrics. A selection of the delivered positive and negative responses is presented in Figure 7.

Figure 7. Samples from the predicted true and false responses.

The performance of the aforementioned models is explained by the rate of positive and negative responses when predicting the unseen instances. Hence, the most accurate models predict correctly the true positives and negatives predictions, and the other way around. The predictions made by EfficientNetB6 are the most accurate in determining whether the airborne target is a drone or bird, which is explained by the high achieved confidence scores during the testing process.

Gradient-Weighted Class Activation Map (Grad-CAM)

Further, we have integrated prediction visualization techniques to have more intuitive outputs and to visualize the performance of the models. The original and predicted aerial targets on the image are highlighted. Besides, the GRAD-CAM allows to see the activation maps that are responsible for the prediction ensured by the last Convolution, Pooling and activation layers. Figure 8 highlights a selection of the detected birds and drones using Grad-CAM technique.

Figure 8. Grad-CAM visualization targets.

Discussion

The developed model successfully responds to the anti-drone needs and fulfills the related challenges and requirements to recognize efficiently and properly the most encountered airborne targets, which are drones and birds. The conducted experiments have shown that using transfer learning and finetuning significantly enhances the detection performance. Through the conducted experiments, we have found that using the backbone model allows for feature extraction in the real-time detection module, allowing it to focus more on decision-making rather than raw data processing. This proposed model serves as backbone for real-time detection during the anti-drone deployment with a complementary real-time detection model which includes more targets such as airplanes, dayframes and building (Yasmine, Maha, and Hicham 2023).

Conclusion and Perspectives

This paper has proposed an airborne identification friend or foe model aiming to determine if the aerial target is a drone or bird. This backbone model aims to improve the overall performance of the anti-drone systems since the whole process relies on the information delivered by the detection model. Accordingly, we have developed a binary classification method following an appropriate methodology using transfer learning, fine-tuned regularization techniques and data augmentation. These latter have enhanced the overall performance.

Also, we have shown that the model depth impacts to a large extent the overall performance since deeper models perform better. Most importantly, there is a common threshold depth from which all the models have similar behavior and then tend to converge with only an increase in the computational time versus a small improvement in the results. This is the stagnation phase. Results showed that EfficientNetB6 model achieved the best performance with 98.12% accuracy, 98.184% precision, 98.115% F1 score, and 99.85% AUC. The EfficientNetB6 results exceed the existing methods in the literature. Finally, it can be concluded that using pre-trained models and fine-tune them on our target dataset following the proposed methodology achieves better results compared with other existing approaches.

Since this proposed model addresses drone and bird classification, the addition of a complementary real-time tracking model presents an interesting research avenue. Also, the reliability of the model could potentially be improved by developing a fine-grained classification model of the types of drones and also by adding new airborne targets that are susceptible to be assimilated to drones and birds which is worth considering it in our upcoming research.

In addition, we are going to combine this backbone model with real-time detection and tracking modules to get the exact locations and predict the movements of the foe target.

Disclosure Statement

The authors have no relevant financial or non-financial interests to disclose.

Data Availability Statement

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

Additional information

Funding

The authors declare that no funds, grants, or other support were received during the preparation of this manuscript.