Dynamic fuzz testing of UAV configuration parameters based on dual guidance of fitness and coverage

Abstract

ArduCopter's configuration parameter verification defects may cause the Unmanned Aerial Vehicle (UAV) in abnormal status. However, traditional UAV configuration parameter defect detection methods based on fuzz testing lack guidance design and inadequately detect configuration parameter defects. This paper proposes an improved configuration security defect analysis method based on fuzz testing. Using the fitness feedback mechanism based on the CAG neural network to guide the generation of fuzz testing cases, and using multiple coverage feedback mechanisms to guide the exploration direction of fuzz testing. Experimental results show that this method almost covers ArduCopter's position and attitude controller, guiding the UAV into abnormal states such as spin and crash, and detecting specific instances of configuration parameter defects.

KEYWORDS:

• Configuration defects
• fuzz testing
• fitness
• coverage
• feedback

1. Introduction

UAVs are flight carriers that do not need to be directly controlled by personnel. Due to their affordability and versatile applications, UAVs have been widely used in military, scientific research, commercial, and other fields. However, with the continuous development of UAV flight control system functions, numerous configuration parameters bring flexible function configuration and many configuration safety risks.

Fuzz testing has some advantages in discovering security issues related to configurations. For UAV flight control systems, fuzz testing involves the generation of configuration parameters, which are then uploaded to the UAV in a random or semi-random manner. Subsequently, the programme’s state is monitored to detect potential programme errors. Popular fuzz testing tools such as Microsoft’s OneFuzz (2023), Google's AFL (Bai et al., 2018) and Honggfuzz (Bohme et al., 2017) are mainly used to test traditional applications, system kernels, browsers, and network protocols. They detect problems with programmes by detecting memory leaks or programme crashes. However, these tools cannot be directly applied to UAV flight control systems due to limitations such as fault detection and input space. To address this problem, RVFuzzer (Chen et al., 2018) exploits input validation bugs to upload binary search mutated configuration parameters to the UAV and monitors unexpected situations during the flight. Similarly, LGDFuzz (Chen et al., 2019) generates test cases by genetic algorithms, uses the UAV states to determine the effectiveness of fuzz testing, and generates reasonable ranges by balancing high stability and low adaptability associated with narrowing the range of configuration parameters. PGFuzz (Fioraldi et al., 2020) uses temporal logic formulas with time constraints to tailor the inputs to the UAV spatially and uses the distance between the system states and the rule violation as the feedback for fuzz testing. The above work pioneered the application of fuzz testing in the UAV flight control systems without using coverage as an orientation for fuzz testing. Using coverage metrics instead of observations of UAV states can provide a more intuitive and accurate evaluation of the results of fuzz testing (Han et al., 2022).

To address the aforementioned problems, this paper proposes a dynamic fuzz testing method for UAV configuration parameters based on dual feedback-driven of fitness and coverage, which uses the fitness feedback mechanism based on the CAG neural network and the multiple coverage feedback mechanism to perform dynamic fuzz testing on the configuration parameters of the UAV flight control system, thereby improving upon limitations of traditional UAV fuzz testing tools and more effectively detecting potential configuration parameter defects in ArduCopter. In summary, this paper makes the following contributions:

1. For the lack of feedback and guidance in traditional fuzz testing tools for UAV flight control systems, this paper proposes a dynamic fuzz testing method based on dual guidance of fitness and coverage. The method utilises fitness function feedback to drive the mutation of configuration parameters and multiple coverage metrics feedback to drive the scheduling of configuration parameters, so as to guide the exploration direction of fuzz testing and detect configuration defects more efficiently.

2. For the high time cost of UAV configuration parameter mutation, this paper utilises the CAG neural network model combining causal convolution, gated loops and attention to predict UAV states. The model accurately predicts the attitude data of ArduCopter in a temporal sequence, avoiding time-consuming simulation validation of mutation, thus improving the efficiency of fuzz testing.

2. Related works

The many limitations in UAV flight control systems lead to traditional software testing techniques (ArduPilot, 2023; Bai et al., 2018; Bohme et al., 2017) not detecting configuration security issues in UAV flight control systems very well. In cases where the flight control system has a substantial codebase, configuration parameters comprise a significant portion of the input space. The intricacies of static code analysis techniques pose challenges due to high complexity, unpredictable time costs, and lack of insight into the dynamic environment. Dynamic programme analysis, on the other hand, can monitor the programme running process and better analyse the dynamic impact of different configuration parameters on the UAV. However, there is a lack of standardised criteria for determining faults in flight control systems, and many input validation bugs do not even exhibit system-level symptoms until certain conditions are met at runtime.

To address these issues, RVFuzzer (Chen et al., 2018) establishes a control instability detection mechanism to detect control system flaws by observing the operating behaviour of a robotic vehicle. The control instability detection mechanism results are also utilised as feedback to guide input generation for more effective discovery of input validation bugs (Chen et al., 2018). While RVFuzzer (Chen et al., 2018) focuses on using UAV control states as feedback drivers, the method proposed in this paper not only references UAVs' abnormal control states, but also introduces coverage feedback drivers from traditional fuzz testing (Geatpy, 2020; Kim et al., 2021) to UAV fuzz testing. PGFuzz (Fioraldi et al., 2020) further expands the input space to include user commands, configuration parameters, and environmental factors and also proposes strategy-oriented fuzz testing for robotic vehicles in response to traditional fuzz testing methods that focus only on memory corruption vulnerabilities or flight control system stability issues testing framework. PGFuzz (Fioraldi et al., 2020) expresses desired policies via temporal logic formulas with time constraints, using these policies as a guiding metric for fuzz testing, and generates fuzzing inputs that minimise the distance between current state and policy violation. However, this static analysis method is not robust enough to handle policy violations that require dynamic execution to be triggered. Instead, our approach combines dynamic black-box fuzz testing with static white-box coverage analysis by combining the advantages of both in order to eliminate most of the defects. LGDFuzz (Chen et al., 2019) detects range specification bugs in parameters through a genetic algorithm-based mutation strategy. It calculates the most suitable range of parameter values while balancing stability and adaptability when narrowing the range of configuration parameters (Chen et al., 2019). Wang et al. proposed a parameter validation method based on mixture density network and fuzz testing based on LGDFuzz (Chen et al., 2019) for discovering input validation bugs in the parameter mechanism of the flight control system (Kim et al., 2019). Ye et al. fuzz the flight control system by generating high quality multi-dimensional test cases to uncover denial of service vulnerabilities (Lee et al., 2021). These methods also rely solely on anomalous state guidance of the UAV and lack a multi-dimensional feedback-driven approach.

In response to challenges such as extensive input space, inconsistent fault determination conditions and lack of coverage feedback, this paper proposes a dynamic fuzz testing method for UAV configuration parameters. The method performs feedback-driven fuzz testing on specific ArduCopter configuration parameters, enhancing the efficiency and effectiveness of the fuzz testing.

3. Uav fuzz testing framework

3.1. General framework of fuzz testing methods for UAVs

This paper proposes a dynamic fuzz testing method for UAV configuration parameters based on dual guidance of fitness and coverage. The method initially generates multiple sets of configuration parameter test cases (seeds) through randomisation, within the configuration safety range. The scheduler uploads these seeds to the flight control system, which screens the seeds that will enable the UAV to perform the task in its entirety by simulating the flight through the emulator. Subsequently, a fitness-guided differential evolutionary algorithm is employed to introduce mutations to the seeds. The scheduler uploads mutated seeds to the flight control system, and performs different flight tasks and tests to guide the scheduling of seeds and evaluate the effectiveness of the fuzz testing. This evaluation includes the assessment of code line coverage, function coverage, code branch coverage, and failure rate of a particular component. The framework of the fuzz testing is shown in Figure 1.

Figure 1. The Framework of the UAV Fuzz Testing.

3.2. Generation of the initial configuration parameter test case

3.2.1. Track files

Software in the Loop (SITL) performs simulated autonomous flights using trajectory files for route guidance. ArduCopter has many trajectory files built into its source code for various tests. Among them, AVCMission is used as the track file for the SparkFun UAV competition (Lemieux & Sen, 2018), where the UAV needs to take off autonomously, follow a prescribed route and land in a designated location; ArduCopter_Tests contains several types of track files used by ArduCopter's AutoTest framework to test specific features.

3.2.2. Configuration parameter selection

ArduCopter offers over a thousand dynamically adjustable configuration parameters in different code libraries (Lemieux & Sen, 2018). This paper focuses on the configuration parameters strongly related to the UAV’s flight state in the AUTO flight mode. Specifically, ArduCopter predominantly uses the positional-attitude controller to control the UAV's flight state. Among them, the position controller contains tripartite controllers for velocity, acceleration, and position in both horizontal and vertical dimensions. In this paper, the PID gain configuration parameters and feedforward gain configuration parameters of six position controllers are selected. The attitude controller contains rate, acceleration, angle and angular velocity quadratic controllers in three dimensions: roll, pitch and yaw. This paper selects twelve PID gain configuration parameters, feedforward gain configuration parameters and gain limit configuration parameters for the attitude controller. Meanwhile, this paper also selects five configuration parameters related to waypoint speed and acceleration. Details of the above parameters can be found in Appendix Table A1.

3.2.3. Generation and filtering of configuration parameter test case

Most test cases generated randomly are not valid in terms of syntactic format, and such test cases will be rejected directly by the UAV flight control system, resulting in a tremendous waste of computational resources. To balance the legitimacy and diversity of configuration parameters, this paper uses random generation to create several distinct sets of configuration parameters. These parameter sets are generated within the vicinity of the parameter ranges specified in Appendix Table A1. The scheduler subsequently injects these configuration parameters into the ArduCopter, and identical flight routes are repeatedly executed to assess the UAV's attitudes resulting from these parameter mutations. To prevent the configuration parameters that can guide to abnormal states from contaminating the training dataset of the neural network, the logs of normal states are filtered by a data consistency test and split into two groups. One group will be converted into a supervised learning dataset and further divided into a training set and a test set applied to the neural network. In contrast, the other group will be directly used to construct a pool of configuration parameter test cases for participation in fuzz testing.

3.3. Mutation of configuration parameters test case

3.3.1. Mutation method

While randomly generated configuration parameter test cases maximise data diversity, blind exploration poses efficiency challenges for fuzz testing. Mutation mechanisms with goal-directness can generate more configuration parameters that have the potential to trigger errors, thereby enhancing the efficiency of fuzz testing (OneFuzz, 2023; Peng et al., 2018; Robert, 2023). Inspired by the combination of machine learning and genetic algorithms by Chen (Schumilo et al., 2017) and Yan (SparkFun Electronics, 2022) et al., this paper uses the Geatpy differential evolutionary algorithm (Wang, 2021), which incorporates CAG (Causal Convolution-Attention-GRU) neural networks, as a mutation method for configuration parameters. We use fitness function as the primary feedback to assess strengths and weaknesses of configuration parameters, performing operations like selecting, mutating, and crossover on individuals. This ultimately generates the test case set after continuous iteration. The differential evolution algorithm model with fusion of CAG neural networks is shown in Figure 2.

Figure 2. Differential Evolution Algorithm Model with Fusion of Neural Networks.

3.3.2. Population initialisation

The default values of the configuration parameters constitute the initial configuration parameter population $P\left(0\right)$. $P\left(0\right)$ contains a number of $N_1$ configuration parameter individuals $I_{x,y}$, where $x$ is the individual serial number, $y$ is the individual dimension; $N_2$ is the number of individual dimensions; $I_{x,y}^{\max }$ and $I_{x,y}^{\min }$ respectively represent the upper and lower bounds of the value range of the $y$ dimension configuration parameter of the $x$ individual; $I_{x,y}^{\mathrm{icm}}$ represents the increment of the $y$ dimensional configuration parameter of the $x$ individual; the $\mathrm{Random}$ function can generate random numbers in a given range of intervals. The initial configuration parameters population $P\left(0\right)$ and individual $I_{x,y}$ are constrained as shown below: (1) $\{\begin{array}{l}\{I_{x,y}\in P(0)|I_{x,y}^{\min }\le I_{x,y}\le I_{x,y}^{\max };x=1,2,\cdots ,N_1;y=1,2,\cdots ,N_2\}\\ I_{x,y}=\mathit{\text{Random}}\left(\left[0,1\right]\right)\ast |I_{x,y}^{\min }-I_{x,y}^{\max }+2\ast I_{x,y}^{\mathrm{icm}}|+I_{x,y}^{\min }-I_{x,y}^{\mathrm{icm}}\end{array}$(1)

3.3.3. Fitness function feedback

As a feedback indicator of individual evolution, the fitness function is responsible for guiding the inheritance of better configuration parameters to the next generation of populations. This paper compares the deviation distance between the predicted flight states data and the real flight states data of the UAV at the same moment to determine individual performance. Specifically, this paper utilises neural networks to calculate the predicted flight state values such as roll angle and roll angle rate of UAV under a specific individual, and evaluates the deviation between the predicted values and the true values. A higher deviation indicates that the predicted flight state values due to this configuration parameter deviate more from the true value, and this individual is more likely to carry anomalous configuration parameters. The fitness function is calculated as follows: (2) $\begin{array}{c}\mathrm{Fitness}=\mathrm{Sum}\left(|D^P-D^T|\right)\end{array}$(2) Where $D^P$ denotes the flight state values predicted by the neural network and $D^T$ denotes the real flight state values. $\mathrm{Sum}$ function calculates the sum of all the matrix elements and takes the result as the individual's fitness. Generally, individuals with higher fitness are more likely to be the solutions solved by evolution algorithms. Tracking highly adaptable individuals in each generation of populations is of great significance for guiding the direction of evolutionary searches.

3.3.4. UAV status prediction

UAV flight logs constitute time-ordered data, where data from the previous time step strongly influences the data in the subsequent time step. Given the causal correlation of UAV state data over time, causal convolutional networks can model data's forward and backward dependence in time order (Yan et al., 2017). Meanwhile, to reduce the loss function and improve prediction accuracy, this paper trains a CAG neural network model incorporating causal convolutional network, attention mechanism, and gate recurrent unit to estimate UAV attitude. Specifically, the CAG neural network model is trained to predict the maximum conditional probability $D^P$ of the next time step's triaxial angle and angular rate by learning important features from previous $N$ time steps' state information, sensor data, and configuration parameters in flight logs. The Mean Squared Error (MSE) is used to evaluate the training effect of the neural network. In the calculation of the fitness function, the use of neural networks to predict the UAV state caused by an individual instead of simulated or actual flights to verify the UAV state caused by an individual can improve the speed of the calculation of the fitness function and optimise the efficiency of the configuration parameter test case mutations. Figure 3 illustrates the structural information of the CAG neural network model.

Figure 3. Structural of the CAG neural network model.

3.3.5. Evolutionary operations

In order to generate superior individuals to guide UAVs into abnormal flight states, this paper performs a differential evolution operation of the fusion neural network on the initial configuration parameter population $P\left(0\right)$ to explore the potential superior individuals in an iterative manner of the population.

1. Elite Copy Selection. The pre-selection operator adopts the elite replication selection strategy, which directly selects the most adapted individuals from the current population to the next generation population. Those Individuals who are not selected will not participate in the ongoing round of evolutionary iterations. This selection method prevents the chromosomes of optimal individuals from being disturbed by subsequent evolutionary processes. Elite Replication Selection is accomplished as follows: (3) $\begin{array}{c}\mathrm{ECS}\left(F\left(I_{x,y}\right)\right);x=1,2,\cdots ,N_1;y=1,2,\cdots ,N_2\end{array}$(3)

2. Differential Mutation. Differential mutation operations are performed to generate new individuals by using different individuals to interfere with the target individuals in the new population obtained after elite replicative selection. The process of differential mutation is shown below: (4) $\begin{array}{c}{V}_{x,y}^g=I_{r0}^g+F_1\ast \left(I_{r1}^g-I_{r2}^g\right)+F_2\ast \left(I_{\mathrm{best}}^g-I_{r0}^g\right)\end{array}$(4) Where $V_{x,y}^g$ is the mutant individual corresponding to the target individual $I_{r0}^g$, $g$ denotes the number of evolutionary generations, $F$ is the mutant scaling factor, $I_{r1}^g$ and $I_{r2}^g$ are the individuals that are different from each other within the population, and $I_{\mathrm{best}}^g$ is the individual with the highest fitness. both $I_{r1}^g-I_{r2}^g$ and $I_{\mathrm{best}}^g-I_{r0}^g$ form the difference vector.

3. Binomial Distribution Crossover. In order to promote genetic and population diversity, a binomial distribution crossover by a crossover probability factor is needed for each individual and its offspring mutation vectors to generate new trial individuals. The formula for crossover is shown below: (5) $U_{x,y}^g=\{\begin{array}{ll}{V}_{x,y}^g;& \mathit{\text{if}}\mathit{\text{Random}}\left(\left[0,1\right]\right)\le \mathit{\text{XVOR}}\mathit{\text{or}}y=y_{\mathrm{rand}}\\ I_{x,y}^g;& \mathit{\text{if}}\mathit{\text{Random}}\left(\left[0,1\right]\right)>\mathrm{XVOR}\end{array}$(5) Where $U_{x,y}^g$ is the new trial individual, $V_{x,y}^g$ is the $y$ dimensional configuration parameter of the $x$ individual after cross-reorganisation, $I_{x,y}^g$ is the $y$ dimensional configuration parameter of the $x$ individual before cross-reorganisation. $\mathrm{XVOR}$ is the crossover probability factor, and $y_{\mathrm{rand}}$ is the random component that ensures that at least one configuration parameter of the trial individual is provided by the mutant individual.

4. One-to-One Survivor Selection(OTOS). Each individual in the trial population is subjected to OTOS with individuals in the same position in the target population retaining the more highly adapted individuals. The formula for OTOS is shown below: (6) $I_x^{g+1}=\{\begin{array}{ll}{U}_x^{g+1};& \mathit{\text{if}}\mathit{\text{Fitness}}\left(U_{x,y}^g\right)\le \mathit{\text{Fitness}}\left(I_{x,y}^g\right)\\ I_x^g;& \mathit{\text{if}}\mathit{\text{Fitness}}\left(U_{x,y}^g\right)>\mathit{\text{Fitness}}\left(I_{x,y}^g\right)\end{array}$(6) Where $I_x^{g+1}$ is the next generation of individuals selected by the OTOS by comparing fitness, $U_{x,y}^g$ is the current-generation trial individual, $I_x^{g+1}$ is the current-generation target individual, and $\mathrm{Fitness}$ is the fitness function.

Suppose the maximum number of evolutionary generations preset by the algorithm is reached or the optimal solution of the population reaches the accuracy error preset by the algorithm. In that case, the differential evolution algorithm is terminated and the better-adapted individuals in the last generation of the population are selected as the output of the mutation. Conversely, the last generation of the population is used as the initial population for the next round of differential evolution to continue the evolution.

3.4. Configuration parameters test case application

3.4.1. Simulation verification

In this paper, the configuration parameters generated in Section 3.3 are applied to ArduCopter for verification to verify whether the test cases can guide the UAV into abnormal states. UAV status categorisation and specific information are covered more in the subsequent Test Oracle section.

The ArduCopter flight control system can implement various configuration parameter test case verification methods. This paper selects two verification approaches: SITL simulation and AutoTest. Specifically, the SITL simulation verification method uploads configuration parameters constructed by the differential evolution algorithm fused with neural networks to the simulation environment. It then starts the ArduCopter flight control system cyclically to perform the same flight mission, verifying each set of configuration parameters. The method detects UAV abnormal states by obtaining STATUSTEXT information stream. Our method can count abnormal UAV flight states caused by each configuration parameter set and provide coverage information for the related codebase. Meanwhile, the AutoTest method passes the set of configuration parameter test cases directly to the ArduCopter flight control system and loops through the execution of unit and functional tests. This method eliminates the repetitive creation of simulation scenarios, thus increasing testing efficiency. This paper combines the SITL simulation verification method and the AutoTest method to calculate code line coverage, function coverage, code branch coverage, and fault discovery rate for the ArduCopter-related codebase.

3.4.2. Test oracle

Configuration test cases generated through fuzz testing may guide the UAV to assume multiple attitudes. It is essential to determine if the UAV's attitude is within safety bounds to detect errors. By monitoring its operational attitude, ArduCopter can determine whether the UAV is in an abnormal state. However, due to limitations of the UAV's onboard computer performance, ArduCopter cannot determine all abnormal states. This paper refines the state category definition of UAVs by adding test oracles, and analyses the STATUSTEXT data in the MAVLink information stream in real-time at the ground control station to determine UAV's specific state. The particular test prophecies are shown in Table 1.

Table 1. UAV status test oracle.

UAV status	Implications
Pass	The UAV completes the established flight mission and lands safely.
Timeout	The UAV fails to arrive at the next waypoint within the time threshold.
PreArm Failed	The UAV's motor refuses to unlock when there is an abnormality in the sensor or configuration item.
Potential Thrust Loss	The UAV fails to reach the desired state even when the throttle is saturated.
Course Deviation	The UAV deviates from the established course and exceeds the distance threshold.
Yaw Imbalance	The UAV is caught in a left-right swing in the yaw direction.
Spin	The UAV with excessive yaw imbalance is caught in a rapid spin.
Failsafe	The UAV encounters an unknown malfunction that activates protective measures.
Crash	All of the above abnormal states may cause the UAV to crash or touch down too fast.

3.4.3. Coverage feedback

As the first-level feedback of the evolutionary algorithm, fitness can guide individuals causing the UAV's abnormal state to be retained for the next generation population, thus building test cases with higher expectations for configuration parameters. In contrast, not all test cases can explore more code paths. Using coverage feedback mechanism as the second-level feedback can filter test cases that positively contribute to exploration so that they can continue to join the mutation pool or seed pool for subsequent fuzz testing.

Coverage feedback mechanism includes code line coverage, function coverage, and branch coverage feedback and is also a generalised metric for examining fuzz testing effectiveness. Specific coverage information is described in Section 4.2 Experimental Evaluation Index. Coverage feedback mechanisms can evaluate the progress of fuzz testing, and experiments can continue to be executed until the coverage target is reached or end the round of fuzz testing after the coverage enters the growth bottleneck, and refine the test cases that are helpful for the coverage growth and put them into subsequent fuzz testing. Incorporating coverage feedback mechanism also reduces time required for subsequent fuzz testing. By filtering test cases that enhance coverage and eliminating unhelpful ones for coverage improvement, the size of the test cases in the seed pool decreases dramatically, improving the efficiency of subsequent fuzz testing.

Specifically, this paper uses GCOV and LCOV to count the coverage information of ArduCopter. GCOV instruments additional code when compiling the ArduCopter source code and traces the execution path of the regulation when the programme is run. A raw file containing coverage data is generated upon programme execution. Convert coverage statistics source files from LCOV and Genhtml into user-friendly visual reports providing code line coverage, function coverage, and code branch coverage data for specific code bases and uncovered blind code paths. By filtering out test cases that positively contribute to code line coverage, function coverage, and code branch coverage to be added to the mutation pool, the second round of fuzz testing expects high-performing initial seeds to mutate into more adaptive individuals covering more code paths and exploring unknown configuration parameter defects.

4. Experiments

4.1. Experiment environment

The hardware environment comprises an Intel Xeon Bronze 3204 CPU, NVIDIA RTX 3080 GPU and 64GB RAM. The software environment comprises the Ubuntu 22.04 operating system, ArduPilot V4.4.1 open-source autopilot software, SITL simulator, MAVProxy ground control station software, Python 3.10.6 development language and TensorFlow 2.12.0 deep learning framework.

4.2. Experimental evaluation index

4.2.1. Failure rate

The failure rate is the ratio of the number of configuration parameters that can guide the UAV into an abnormal state to the total number. It is one of the most important metrics for measuring the quality of test cases generated by fuzz testing. A higher failure rate indicates a greater likelihood of uncovering vulnerabilities in configuration parameters and related code. Based on the UAV status category defined in Section Test Oracle, this paper counts the number and proportion of test cases in the configuration parameters that can guide the UAV to abnormal states to evaluate the quality of the configuration parameters.

4.2.2. Coverage rate

Coverage is the ratio of the total number of code statements, function methods, and branching paths that are executed during testing. It is a critical evaluation metric for measuring the effectiveness of fuzz testing execution. This paper uses code line coverage to determine whether all executable statements in the AC_AttitudeControl position attitude controller codebase are executed, uses function coverage to determine whether all logical functions are tested, and combines this with code branch coverage to determine whether fuzz testing covers different decision paths.

4.3. Analysis of experimental results

Regarding the failure rate, this paper tallies the count and percentage of various state types appearing in 4661 sets of configuration parameter test cases guiding UAVs in the SITL simulation and verification phase. The percentage of each state type is shown in Figure 4, examples of some UAV abnormal states are shown in Figure 5, and specific results of quantity and severity levels for each status type are listed in Table 2.

Figure 4. Percentage of UAV states.

Figure 5. Examples of UAV abnormal states.

Table 2. Quantity and severity of UAV states.

States	Quantity	Severity
Pass	450	INFO (6)
Timeout	715	CARTICAL (2)
PreArm Failed	656	EMERGENCY (0)
Potential Thrust Loss	1644	EMERGENCY (0)
Course Deviation	648	WARNING (4)
Yaw Imbalance	152	EMERGENCY (0)
Spin	75	EMERGENCY (0)
Failsafe	69	WARNING/CARTICAL (4/2)
Crash	252	EMERGENCY (0)

From Figure 4, Figure 5 and Table 2, it can be observed that the percentage of fuzz testing cases that can guide the UAV to abnormal states is 90.35%. Among these, only 656 groups of test cases can be detected as incorrect by the ArduCopter PreArm check programme, and the test cases that can cause an abnormal state after the UAV takes off account for 83.62% of all test cases that can cause abnormal states. The data indicates that most configuration parameters generated by fuzz testing are immune to the PreArm check programme and guide the UAV to abnormal states after takeoff.

After multiple rounds of fuzz testing, ArduCopter exhibited numerous unintended scenarios. This paper reproduces the detected configuration defect errors by reconstructing the same configuration test cases. A typical configuration parameter defect case is the floating point overflow error. According to the parameter list provided in the ArduCopter user manual (OneFuzz, 2023), the range of values for the PSC_ACCZ_I (Vertical Acceleration Controller I Gain) configuration parameter is [0.000-3.000]. However, the checking mechanism for this configuration parameter in ArduCopter is "is_positive(get_accel_z_pid().kI())", which indicates whether PSC_ACCZ_I is a positive floating point number. When the user configures PSC_ACCZ_I to be 0 according to the manual, it will trigger a floating-point overflow error. This error leads to the position controller's unlock check failure, causing the motor to reject unlocking and resulting in a PreArm Failed error. More seriously, when using all default configuration parameters to make the UAV take off normally, dynamically loading configuration parameter test cases generated by fuzz testing, the UAV will accept them unconditionally. Although some configuration parameters require effect after flight control system restart, they still threaten UAV stability.

For the coverage rate, this paper uses the same configuration parameters test cases to apply the AutoTest framework to ArduCopter and counts the code line coverage, function coverage, and code branch coverage of the AC_AttitudeControl library within the AC_AttitudeControl.cpp, AC_AttitudeControl_Multi.cpp, and AC_PosControl.cpp files in the AC_AttitudeControl library. The experimental results are shown in Table 3.

Table 3. Comparison of our coverage statistics with ArduCopter's coverage statistics.

	Line coverage		Function coverage		Branch coverage
Controller	ArduCopter	Ours	ArduCopter	Ours	ArduCopter	Ours
AC_AttitudeControl	75.8%	80.4%	89.2%	86.5%	–	71.8%
AC_AttitudeControl_Multi	94.0%	97.6%	100%	100%	–	97.9%
AC_PosControl	90.6%	91.0%	90.6%	90.6%	–	74.7%

From the coverage statistics table, it can be learned that the code line coverage of AC_AttitudeControl achieves 80.4%, surpassing ArduCopter's official test result (Ye, 2020) by 4.6%. Furthermore, the function coverage achieves 86.5%, and the code branching coverage achieves 71.8%. AC_AttitudeControl_Multi achieves 97.6% code line coverage, surpassing ArduCopter's official test result by 3.6%. Additionally, function coverage for AC_AttitudeControl_Multi achieves 100%, while code branch coverage reaches 97.9%. AC_PosControl achieves 91.0% code line coverage, surpassing ArduCopter’s official test result by 0.4%; function coverage achieves 90.6% and code branch coverage achieves 74.7%. It is important to note that ArduCopter does not officially track branch coverage, as the current LCOV version disables it by default. For files that did not achieve complete coverage, the relevant code lines, function functions or code branches were not covered because they were not invoked by the configuration parameters examined in this paper. Additionally, there are some unreachable redundant codes in ArduCopter, but they still participate as the denominator of the coverage. Furthermore, certain codebases unrelated to the position and attitude control of ArduCopter were not included in the fuzz testing in this paper. The coverage statistics presented in Table 3 fully demonstrate the effectiveness of the dynamic fuzz testing method for configuration parameters based on dual guidance of fitness and coverage and also inspire future work to explore the uncovered code paths and other codebases. The initial round of fuzz testing substantially reduced the size of the seed pool and improved the efficiency of fuzz testing by screening test cases that could effectively improve coverage. Subsequent fuzz testing attempts to cover more unexplored code paths by continuing to mutate high-quality seeds in the seed pool.

5. Conclusions

In this paper, we propose a dynamic fuzz testing method for UAV configuration parameters guided by both fitness and coverage. We utilise a differential evolutionary algorithm fused with CAG neural networks with fitness as the first level of guidance to mutate the parameters related to UAV position and attitude control. Additionally, we use the coverage of UAV position-attitude control libraries as the second level of guidance for scheduling configuration parameter test cases. Meanwhile, the CAG neural network is used to estimate the UAV state during the fitness function calculation. Experimental results show that the dynamic fuzz testing method for UAV parameters based on dual guidance of fitness and coverage can guide UAVs to over eight abnormal states and detect specific configuration parameter defects. Most of the UAV position-attitude controller related codebase coverage exceeds official ArduCopter data, proving the method's effectiveness. In future work, we plan to combine white-box testing techniques such as static analysis to complete the testing of UAV configuration parameters. In addition, defective configuration parameters are repaired by designing syntax and semantic constraints for configuration parameters.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by National Key Research and Development Program of China [grant number: 2020YFB1005704].

Appendix

Table A1. ArduCopter V4.4.1 Fuzz testing configuration parameter list.

Parameter	Definition	Defaults	Increment	Range	Parameter	Definition	Defaults	Increment	Range
PSC_VELXY_P	Horizontal velocity P gain	2	0.1	[0.1,6]	ATC_RAT_PIT_D	Pitch axis rate D gain	0.0036	0.001	[0,0.05]
PSC_VELXY_I	Horizontal velocity I gain	1	0.01	[0.02,1]	ATC_RAT_PIT_FF	Pitch axis rate feedforward	0	0.001	[0,0.5]
PSC_VELXY_D	Horizontal velocity D gain	0.5	0.001	[0,1]	ATC_RAT_YAW_P	Yaw axis rate P gain	0.09	0.005	[0.1,2.5]
PSC_VELXY_FF	Horizontal velocity feedforward	0	0.01	[0,6]	ATC_RAT_YAW_I	Yaw axis rate I gain	0.009	0.01	[0.01,1]
PSC_VELZ_P	Vertical velocity P gain	5	1	[1,8]	ATC_RAT_YAW_D	Yaw axis rate D gain	0	0.001	[0,0.02]
PSC_VELZ_I	Vertical velocity I gain	0	0.01	[0.02,1]	ATC_RAT_YAW_FF	Yaw axis rate feedforward	0	0.001	[0,0.5]
PSC_VELZ_D	Vertical velocity D gain	0	0.001	[0,1]	ATC_ANG_RLL_P	Roll axis angle P gain	4.5	0.1	[3,12]
PSC_VELZ_FF	Vertical velocity feedforward gain	0	0.01	[0,1]	ATC_ANG_PIT_P	Pitch axis angle P gain	4.5	0.1	[3,12]
PSC_ACCZ_P	Vertical acceleration P gain	0.5	0.05	[0.2,1.5]	ATC_ANG_YAW_P	Yaw axis angle P gain	4.5	0.1	[3,12]
PSC_ACCZ_I	Vertical acceleration I gain	1	0.1	[0,3]	ATC_RATE_FF_ENAB	Body-frame rate feedforward enabled	1	1	[0/1]
PSC_ACCZ_D	Vertical acceleration D gain	0	0.001	[0,0.4]	ATC_RATE_P_MAX	Maximum angular velocity in pitch axis	0	10	[0,1080]
PSC_ACCZ_FF	Vertical acceleration feedforward	0	0.001	[0,0.5]	ATC_RATE_R_MAX	Maximum angular velocity in roll axis	0	10	[0,1080]
PSC_POSXY_P	Horizontal position P gain	1	0.01	[0.5,2]	ATC_RATE_Y_MAX	Maximum angular velocity in yaw axis	0	10	[0,1080]
PSC_POSZ_P	Vertical position P gain	1	0.01	[1,3]	ATC_ACCEL_P_MAX	Maximum acceleration in pitch axis	1000	1000	[0,180000]
ATC_RAT_RLL_P	Roll axis rate P gain	0.135	0.005	[0.01,0.5]	ATC_ACCEL_R_MAX	Maximum acceleration in roll axis	1000	1000	[0,180000]
ATC_RAT_RLL_I	Roll axis rate I gain	0.135	0.01	[0.01,2.0]	ATC_ACCEL_Y_MAX	Maximum acceleration in yaw axis	1000	1000	[0,72000
ATC_RAT_RLL_D	Roll axis rate D gain	0.0036	0.001	[0,0.05]	ATC_ANGLE_BOOST	Angle Boost increases output throttle	1	1	[0/1]
ATC_RAT_RLL_FF	Roll axis rate feedforward	0	0.001	[0,0.5]	WPNAV_SPEED	Waypoint horizontal speed target	1000	50	[20,2000]
ATC_RAT_PIT_P	Pitch axis rate P gain	0.135	0.005	[0.01,0.5]	WPNAV_SPEED_UP	Waypoint ascent rate target	250	50	[10,1000]
ATC_RAT_PIT_I	Pitch axis rate I gain	0.135	0.01	[0.01,2]	WPNAV_SPEED_DN	Waypoint descent rate target	150	10	[10,500]