Applicability of UAV-based optical imagery and classification algorithms for detecting pine wilt disease at different infection stages

ABSTRACT

As a quarantine disease with a rapid spread tendency in the context of climate change, accurate detection and location of pine wilt disease (PWD) at different infection stages is critical for maintaining forest health and being highly productivity. In recent years, unmanned aerial vehicle (UAV)-based optical remote-sensing images have provided new instruments for timely and accurate PWD monitoring. Numerous corresponding analysis algorithms have been proposed for UAV-based image classification, but their applicability of detecting different PWD infection stages has not yet been evaluated under a uniform conditions and criteria. This research aims to systematically assess the performance of multi-source images for detecting different PWD infection stages, analyze effective classification algorithms, and further analyze the validity of thermal images for early detection of PWD. In this study, PWD infection was divided into four stages: healthy, chlorosis, red and gray, and UAV-based hyperspectral (HSI), multispectral (MSI), and MSI with a thermal band (MSI&TIR) datasets were used as the data sources. Spectral analysis, support vector machine (SVM), random forest (RF), two- and three-dimensional convolutional network (2D- and 3D-CNN) algorithms were applied to these datasets to compare their classification abilities. The results were as follows: (I) The classification accuracy of the healthy, red, and gray stages using the MSI dataset was close to that obtained when using the MSI&TIR dataset with the same algorithms, whereas the HSI dataset displayed no obvious advantages. (II) The RF and 3D-CNN algorithms were the most accurate for all datasets (RF: overall accuracy = 94.26%, 3D-CNN: overall accuracy = 93.31%), while the spectral analysis method is also valid for the MSI&TIR dataset. (III) Thermal band displayed significant potential in detection of the chlorosis stage, and the MSI&TIR dataset displayed the best performance for detection of all infection stages. Considering this, we suggest that the MSI&TIR dataset can essentially satisfy PWD identification requirements at various stages, and the RF algorithm provides the best choice, especially in actual forest investigations. In addition, the performance of thermal imaging in the early monitoring of PWD is worthy of further investigation. These findings are expected to provide insight into future research and actual surveys regarding the selection of both remote sensing datasets and data analysis algorithms for detection requirements of different PWD infection stages to detect the disease earlier and prevent losses.

KEYWORDS:

• Pine wilt disease
• unmanned aerial vehicles
• optical imagery
• classification algorithms

1. Introduction

Pine wilt disease (PWD) is a devastating invasive disease that affects Pinus species and is caused by the pine wood nematode (PWN, Bursaphelenchus xylophilus). PWN is a North American native that has spread rapidly around the world through cerambycid beetles of the genus Monochamus (Morimoto and Iwasaki 1972; Escuer, Arias, and Bello 2004). Within its infectious range, it endangers forests, plantations, and even ecosystems. The area, size, and extent of PWD damage is growing worse because of ongoing global climate change and extreme weather. According to relevant statistical reports in China, it has been reported in 18 provinces in China, with an infection area of 1.8 million hectares, and had already killed 19.5 million trees by 2020 (Zhang et al. 2021; Sun et al. 2021). Nonetheless, as a major quarantine method, the prompt burning or felling of affected trees is the only way to control the disease (Zhang et al. 2021; Yu et al. 2021; Jackson et al. 1981). Thus, identifying and removing diseased trees from a site before the disease spreads to other (healthy) trees is the main priority in PWD management. The commonly used PWD damaged tree detection technique, on the other hand, has always focused on artificial field research in actual forest management activity. It is frequently necessary to combine physical and chemical testing procedures, such as microscopic detection, polymerase chain reaction (PCR), and loop-mediated isothermal amplification (LAMP) methods to obtain a correct judgment (Wu et al. 2021). These methods require highly skilled grassroots workers (such as local forest station workers), professional tools, and individuals with professional backgrounds in pathology, entomology, and other related fields (Tang et al. 2021). More importantly, these methods are based on point sampling, which is not only unable to meet the real-time requirements of PWD field research but also lacks spatial integrity (Carnegie et al. 2018; Choi et al. 2017).

The development of optical remote sensing technology opens new avenues for forest damage diagnosis, detection, and location (Choi et al. 2017; Lee, Cho, and Lee 2007; Ju et al. 2014; Zhang et al. 2018). In recent decades, unmanned aerial vehicle (UAV) technology has been routinely used in PWD investigation to provide additional data support for natural resources monitoring (Iordache et al. 2020; Yu et al. 2021). Kim et al. (2015) used a UAV to collect time-series hyperspectral images and successfully analyzed the PWD distribution characteristics. Huang et al. (2018) employed a fixed-wing UAV to monitor dead pine trees for PWD on an individual tree scale with an accuracy of over 80%. Previous studies have shown that UAVs are low in cost, easy to operate, and adaptable, making them ideal in forest disease detection, particularly for PWD (the susceptible trees wilt and eventually die within 2–3 months after infected by Bursaphelenchus xylophilus). Compared to the use of satellite and ground remote sensing for forest disease investigation, UAV technology is less affected by atmospheric effects and can provide imaging data that are not limited by tree height or ground environment. Thus, we selected the UAV as the platform for this study.

From the data perspective, UAV-based visible images, multispectral images (MSI), and hyperspectral images (HSI) are currently used in forest disease investigations (Iordache et al. 2020; Wu et al. 2020). Xu et al. (2020) used UAV-based RGB images to achieve a PWD-infected pine tree detection with an overall accuracy of 82.42%, while under the same conditions, Huang (2020) and Kim et al. (2015) used MSI data and HSI data for infected trees detection, respectively. While many studies have shown the usefulness of spectral imaging techniques in identifying PWD-infected trees because of changes in water, pigmentation, and even canopy structure in the host itself during this process (Kim et al. 2013, 2018; Zhang et al. 2020), there is no clear description of how to select the optimal data for practical applications. For example, Iordache et al. (2020) found that changes in the red and near-infrared spectral bands can better indicate the progression of the PWD; however, this conclusion results from both the MSI and HSI datasets. Furthermore, despite changes in transpiration rate being the primary symptom of PWD, thermal images have been shown to be effective in identifying water changes in agriculture and other forest diseases (Golomb et al. 2015; Liu et al. 2021). However, to the authors’ knowledge, there are few effective hyperspectral or multispectral UAV-based sensors that cover the 1100–2500 nm for detecting PWD-infected pine tree and few studies have been conducted on thermal imaging. Thus, we will investigate the capabilities of various UAV-based data sources in identifying the various stages of PWD infection as well as the prospective applications of thermal imaging.

From the algorithm perspective, many UAV-based image classification algorithms have been developed in recent decades. Spectral analysis method is an earlier algorithm used in disease and pest monitoring (Ju et al. 2014; Kim et al. 2013). In the scientific literature, some relevant sensitive bands, spectral/vegetation/disease indices have been suggested and published. Machine learning algorithms, such as support vector machine (SVM), random forest (RF), and artificial neural network (ANN), which effectively combine spectral and spatial features, have been continuously developed, achieved satisfactory classification results, and successfully used for MSI and HSI datasets in PWD tree-damage classification. Takenaka et al. (2017) used the SVM algorithm to categorize the various stages of PWD-damaged degrees trees based on a combination of aerial laser scanning and high-resolution space-borne images, with a highest classification accuracy reaching of 98.5%. Yu et al. (2021) used the RF algorithm to successfully estimate the stage of PWD infection in sampled trees. However, many studies have indicated that selecting the appropriate parameter combination, kernel function, and classification feature for these methods remains difficult (Zhang, Wang, and Zhang 2020; Romero, Gatta, and Camps-Valls 2015; Liu 2017). In recent years, because deep learning in computer vision is currently booming, it has been possible to identify damaged trees in forests by using an intelligent analysis technique for remote sensing images. Qin et al. (2021) used UAV-based MSI to diagnose pine nematode diseases using the suggested spatial-context-attention network (SCANet). The average overall accuracy was 79.33%, with a precision of 0.86 and a recall rate of 0.91. Wu et al. (2021) employed a faster region-based convolutional network (Faster R-CNN) and You Only Look Once version 3 (YOLOv3) to diagnose the PWD-infected trees at an early stage. The AI detection score was 0.772. It is not difficult to find that similar to data source selection, it is also important to discuss the recognition abilities of various algorithms for different PWD infection-stages detection while using different data sources.

Therefore, this study primarily focused on assessing the applicability of UAV-based optical imagery and classification algorithms for detecting PWD at different infection stages. And the usefulness of UAV-based thermal imaging was analyzed using mature image classification algorithms to fill the gap left by thermal imaging in PWD identification. The goals of this research were to answer three scientific questions in remote sensing-based PWD actual investigation: 1) which data source is the most appropriate for different PWD infection-stage detection; 2) which data analysis method is the comparatively optimum choice of corresponding data source under different PWD infection stages; and 3) whether the thermal imaging system is capable of meeting the demands of PWD investigation, especially for the early detection of infection.

2. Study area and data

2.1. Study area

The study area was concentrated in two typical regions where the PWD occurred. The first one is Caomiaozi, which is located in the LinGang Economic and Technological Development Zone, Weihai City in Shandong Province, China (37°14′~37°20′N, 122°03′~122°10′E, Figure 1(a)-Site a). The second site is Xiangshan East Village, which is located in ChaoHu City in Anhui Province, China (31°16′~32°0′N, 117°25′~117°57′E, Figure 1(a)-Site B).

Figure 1. Location of the study areas. (a) represents the location of the study areas; (b) and (c) denote the true color composite images from 639 nm, 550 nm, and 470 nm of Site a and Site B used in this study; (d) and (e) are the live photo of the study area.

Caomiaozi lies in a warm temperate semi-humid monsoon climate zone with an average annual air temperature of 12.7°C. The average annual relative humidity reached 68%. The average precipitation in this area is 158.6 mm in late spring (4 to 5 months) and early autumn (9 to 10 months), and average temperature during this time is 20–25°C. This site is a pure Red pine (Pinus densiflora Sieb. et Zucc) forest with a canopy density of approximately 57% and scattered understory vegetation (Figure 1d). The trees are 10–15 years old with an average height of approximately 5.8 m. The average diameter at breast height (DBH) and average crown width are 10.98 cm and 2.1 m, respectively. Furthermore, this location has a low mountainous topography and is in a development zone with traffic and frequent trade. All of these climatic characteristics, host vegetation distribution, topography, and socio-economic factors are conducive to the spread of PWD.

The Xiangshan East Village has a subtropical monsoon climate. The average annual temperature is 12–20°C, and the average rainfall is over 1000 mm. The area has a frost-free season that lasts more than 200 days per year. This site contains Masson pine (Pinus massoniana Lamb.) and broad-leaved mixed forests. There are a few scattered broad-leaved deciduous trees in the region; however, the understory vegetation is relatively dense (Figure 1E). The trees are 15–17 years old. The dominant tree species in the survey area had an average DBH and crown width of 15 cm and 2.5 m, respectively. The canopy density was over 70%. Similarly, this area has a high incidence of PWD.

2.2. Field data acquisition

The ground survey was conducted in Caomiaozi in September 2019 and in Xiangshan East Village in October 2019. Caomiaozi, identified as Site A, was chosen as the model training set, and Xiangshan East Village, identified as Site B, was chosen as the test site. At these two sites, trees with different degrees of PWN damage were observed.

We set up five 30 m × 30 m square sample plots at Site A. The coordinate data of the four corners of each sampling plot were recorded (the red dots in Figure 1(b)) using a handheld differential global positioning system (DGPS, Version S760, South Surveying & Mapping Technology Co., Ltd. Guangzhou, China). Red pine trees in each sampling plot were chosen for sampling. We validated the healthy and PWD infected trees using the Behrman funnel technique for morphological identification first (Yu et al. 2021) and then used DGPS to locate their coordinates. There were 437 healthy and 75 infected trees in these five sampling plots. Then, using the resin secretions and growth vigor of the trees, as well as the color of the needles (the tops and one-year branches were the main considerations, and the general state of the sample tree was also taken into account), infected trees were classified into three infection levels, in combination with the classification criteria shown in Xu et al. (2011). These three levels of damage severity correspond to the three phases of infection mentioned in Section 3.1. In total, 20 trees were in the chlorosis stage, 40 in the red stage, and 15 in the gray stage. These 75 infected sample trees were used as the investigated data to first offer visual references for drawing the regions of interest (ROIs) of different infection- stages trees and then validate the accuracy of the classification results. These 512 sample trees were not used for training the categorization models.

Due to the topographical conditions of the research site, we did not create sample plots inside Site B. Instead, 20 randomly discolored Masson pine trees were chosen to serve as the sample trees (containing all the infection stages in this research, and recorded the position of each trees with DGPS, the red dots in Figure 1(c)). Totally, there were five trees in chlorosis stage, nine in red stage, and six in the gray stage.

2.3. UAV-based remote sensing images acquisition and pre-processing

A DJI Matrice 600 six-rotor UAV (DJI, Shenzhen, China) was used as the data acquisition platform and two UAV-based imaging systems were constructed to acquire ultra-high-resolution remote sensing data. The first was a UAV-based multispectral system mounted with a MicaSense Altum multispectral camera (MicaSense, Seattle, WA, USA), which had five multispectral bands (blue, green, red, red-edge, and near-infrared) and an integrated long-wave thermal infrared band (Simpson et al. 2021). The second was a UAV-based hyperspectral imaging system equipped with a push-broom Nano-Hyperspec hyperspectral sensor at 400–1000 nm with 270 bands (Headwall Photonics, Boston, USA).

Considering the actual vegetation conditions and areas that needed to be covered, the flying height was set at 120 m, with 75% overlap set in the forward and lateral directions for multispectral images, and 30% lateral overlap between hyperspectral scan lines. Each imaging equipment flew twice to cover Site A and once to Site B. Following this, rigorous pre-processing was performed, including image mosaic, reflectance correction, radiometric calibration, and geometric corrections. To provide multispectral calibration information before and during light changes, a calibration plate and separate Downward Light Sensor (DLS2, Micasense, Seattle, WA, USA) were used. A standard whiteboard and white tarp were placed on the ground within the flying area, and the ground spectrum was measured using an ASD Field Spec Pro Spectrometer (Analytical Spectral Devices, Boulder, CO, USA) for auxiliary radiometric correction. The Savitkzy-Golay smoothing method was used to reduce noise in the hyperspectral images. Finally, two hyperspectral orthophoto images with a spatial resolution of 0.14 m were acquired, as well as two multispectral orthophoto images with a spatial resolution of 0.053 m and two related thermal images with a spatial resolution of 0.062 m.

We resampled the multispectral images using a lower spatial resolution thermal imaging dataset. Then, green forestry was separated from the background (including rivers, bare land, roads, buildings, agricultural lands, shadows, and cars) using threshold segmentation. Table 1 lists the segmentation indices and thresholds for each dataset. In locations with extremely complex vegetation characteristics, manual outlining was partially employed to exclude understory vegetation (some parts of Site B).

Table 1. The segmentation indices and thresholds of each dataset.

Dataset	Segmentation indices	Thresholds
HSI	$\mathrm{N}\mathrm{D}\mathrm{V}\mathrm{I}\left(860,640\right)=\left(R_{860}-R_{640}\right)/\left(R_{860}+R_{640}\right)$	−1≤NDVI(860,640)≤0.7
	$\mathrm{S}\mathrm{i}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{e}\mathrm{I}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{x}=R_{550}+R_{766}-R_{679}$	$0\le \mathrm{S}\mathrm{i}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{e}\mathrm{I}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{x}\le 0.2$
MSI/MSI&TIR	$\mathrm{R}\mathrm{V}\mathrm{I}=\mathrm{R}\mathrm{e}\mathrm{d}/\mathrm{N}\mathrm{I}\mathrm{R}$	$\mathrm{m}\mathrm{i}\mathrm{n}\le \mathrm{R}\mathrm{V}\mathrm{I}\le 0.105\hspace{0.17em}$*

*indicates the effective values of RVI are between the minimum value and 0.105.

3. Methods

In this study, we used the spectral analysis (building a simple spectral index by sensitive band selection and using threshold segmentation to classify different infection stages), RF, SVM, two- and three-dimensional convolutional network (2D- and 3D-CNN) classification algorithms to detect the different PWD infection stages on individual trees. All classification algorithms were tested on three different datasets (hyperspectral image, HSI; multispectral image, MSI; and multispectral image with the thermal band, MSI&TIR). The classification performance of the thermal band was analyzed by comparing the classification accuracy of both MSI, MSI&TIR, and composited RGB with TIR band (RGB&TIR) data sets. Figure 2 shows the primary workflow of the experiments.

Figure 2. UAV-based remote-sensing data analysis processing framework for a comprehensive assessment of different PWD infection stages.

3.1. PWD infection-stage categorization and data set construction

PWD infection was classified into four stages: (1) healthy stage; (2) chlorosis stage; (3) red Stage; and (4) gray stage according to the “Technical regulation on quarantine for the pine wilt disease” (National standards of P.R.C, GB/T 23,476–2009) and previous studies (Yu, Ren, and Luo 2021; Xu et al. 2011). Using the GPS information to determine the location of each sample tree, the typical visual characteristics of each infected stage was summarized based on the visual representation of the trees at the corresponding location on the image. Figure 3 shows typical images and photos of pine trees at four different PWD infection stages and the associated evaluation standards.

Figure 3. Typical images of different PWD infection stages and corresponding categorized standards of each stage in this study.

Referring to the visual standards provided by the 95 infected investigated trees (75 in Site A and 20 in Site B), the ROIs of trees at different infection stages were formed by manually drawing the tree crown canopy on each dataset. Finally, there were 266, 604, and 163 trees in the chlorosis, red, and gray stages at Site A, and 56, 104, and 62 trees at Site B, respectively. Because the infected trees were removed as part of the background during the threshold segmentation described in Section 2.3, different infection stage trees were plotted manually from the resampled images, and the ROIs of healthy trees were formed using green forestry imaging.

The manually drawn ROIs in Site A served as the training set, whereas the 75 diseased trees under investigation served as the validation set. Site B was used as the test set. Table 2 provides detailed information on the three datasets as well as their construction criteria for different classification methods. Because there were many healthy trees in both Sites A and B, the ROIs of healthy trees were chosen at random, and the final number matched to the chlorosis stage.

Table 2. The detailed information of training, validation, and test sets for different classification algorithms.

Algorithms	Training set construction criterion	Detailed information*
		Stages	Training set	Validation set	Test set
2D-CNN & 3D-CNN	Taking each pixel as the center, constructing the space-spectral cube of 19 × 19 × 270 (for HSI) and 43 × 43 × 270 (for MSI and MSI&TIR) size as the sliding window size (for 3D-CNN) and setting 1 pixel as the sliding step size, all extracting space-spectral cube and the corresponding label as sample data of each dataset. The space-spectral cube size of 2D-CNN are 19 × 19 × 3 (for HSI), 43 × 43 × 5 (for MSI) and 43 × 43 × 6 (for MSI&TIR).	Healthy	39582	21983	21072
		Chlorosis	39582	21567	20055
		Red	74156	40562	34101
		Gray	31247	13795	15444
RF & SVM	Tacking the center point of each ROI as the center, selecting all pixels points within 3 × 3 size and the corresponding label as sample data of each data set.	Healthy	2394	21983	21072
		Chlorosis	2394	21567	20055
		Red	5427	40562	34101
		Gray	1476	13795	15444
Spectral Analysis	Extracting the mean spectral reflectance of each ROI and the corresponding label as sample data of the training data set, tacking the center point of each ROI and the corresponding label as sample data of the test data set.	Healthy	266	21983	21072
		Chlorosis	266	21567	20055
		Red	603	40562	34101
		Gray	163	13795	15444

*The samples number of training sets for different algorithms has different sample size statistical basis. 2D-CNN and 3D-CNN are based on the space-spectral cube; RF and SVM are based on the pixel number; and spectral analysis is based on the canopy crown ROI number. The validation and test set were counted according to the actual surveyed pixels of diseased trees.

3.2. Different classification algorithms

3.2.1. Spectral analysis

The spectral vegetation index (SVI) is an effective and simple method for spectral data analysis and for detecting changes in plant physiology and chemistry (Zhang et al. 2020). In this study, we constructed three SVIs for the three datasets. This procedure consists of three steps: (I) acquiring the mean spectral information, (II) selecting a sensitive wavelength, and (III) developing a spectral index. It should be noted that Step (II) is only applicable to the HSI dataset.

The mean spectral reflectance of each ROI at Site A was extracted, and each training set contained a total of 1299 spectral samples. For the HSI dataset, these 1299 spectral samples were utilized to select sensitive bands and than extracted the associated spectra of the sensitive bands to construct spectral index. For the MSI and MSI&TIR datasets, these 1299 spectral samples were used directly for spectral index construction.

Successive projection algorithm (SPA) was used and implemented in Matlab R2016a to find wavelengths sensitive to PWD. SPA employs simple projection operations in a vector space to obtain subsets of wavelengths with minimal collinearity. The principle of variable selection is that the new variable selected is the one among all the remaining variables that has the maximum projection value on the orthogonal subspace of the previously selected variable. Root mean square error (RMSE) was used as the evaluation criterion to determine the final optimal bands (Zhang et al. 2018).

To select the ideal band combination, we drew spectral curves of different stages for each dataset using the final sensitive bands in the HSI dataset and the mean spectral reflectance of the MSI and MSI&TIR datasets. The process of spectral index creation and classification threshold determination is comprises the following important components. (1) Examination of the changing trend of the spectral curve and locating bands with noticeable differences. (2) The normalization different spectral index (NDSI), ratio spectral index (RSI), difference spectral index (DSI), and other band combinations were calculated. (3) The sample scatter and index value of each spectral index was plotted to determine whether they could successfully distinguish between different infection stages. (4) All possible combinations were exhaustively searched to find the optimal index (containing the combination way and weights distribution). (5) Scatter plot of samples of different infection stages (X-axis) and index values (Y-axis) were generated to estimate the related thresholds for disease class separation. Corresponding to the three data sets of HSI, MSI, and MSI&TIR, three spectral indices were finally formed based on their classification results.

3.2.2. Machine learning algorithms

Two typical machine learning classification algorithms, SVM and RF classifiers, were used to classify the different infection stages in the three image datasets. These two machine learning algorithms were implemented in PyCharm by using Python 3.8.6.

As a non-parametric supervised machine learning algorithm, the key point of the SVM classifier for solving the class separation problem is to find the optimal hyperplane (Wietecha et al. 2019). Gaussian radial basis (G-RBF) kernel was selected for SVM classification, and two parameters – the penalty coefficient (C) and the kernel parameter (γ)—were determined using the grid search algorithm. The best classification parameters combination of C and γ was trained using Site A’s data. The RF classifier is also a robust and flexible non-parametric classification approach. It is friendly to high-dimensional, large data, and multi-classification tasks (Xu et al. 2012). The decision tree is a key point for the RF classifier. We also used the data from Site A to train the structure and determine the number of decision trees from {1000, 500, 150, 100}.

3.2.3. Deep learning algorithms

The capacity of deep learning algorithms to automatically learn complex characteristics assists in simplifying the data interpretation classification process. In recent years, convolutional neural networks (CNN) have made unprecedented advances in the field of computer vision and can significantly improve the accuracy of classification jobs, particularly those involving hyperspectral data. Considering that this study used both HSI and MSI (MSI&TIR) images, we compared both 2D- and 3D-CNN techniques.

2D-CNN can efficiently extract spatial information and share the weights, which significantly decreases the amount of information that the model can deliver. Eq. (1)(1) $v_{ij}^{xy}=f\left(b_{ij}+{\sum }_m{\sum }_{h=0}^{H_i-1}{\sum }_{w=0}^{W_i-1}{k}_{ijm}^{hw}{V}_{\left(i-1\right)m}^{\left(x+h\right)\left(y+w\right)}\right)$(1) describes a 2D convolution operation:

(1) $v_{ij}^{xy}=f\left(b_{ij}+{\sum }_m{\sum }_{h=0}^{H_i-1}{\sum }_{w=0}^{W_i-1}{k}_{ijm}^{hw}{V}_{\left(i-1\right)m}^{\left(x+h\right)\left(y+w\right)}\right)$(1)

where $v_{ij}^{xy}$ is the value at ($x,y)$ in the $j$-th feature map in the $i$-th layer, $b_{ij}$ is the offset and $m$ is the index of all feature maps connected to the current layer in the $\left(i-1\right)$ layer, $k_{ijm}^{hw}$ represents the value of the convolution kernel at $\left(h,w\right)$ (Zhang, Zhao, and Zhang 2020).

Ji et al. (2013) first proposed 3D-CNN to manage the challenge of extracting information from spatial and temporal dimensions, fostering significant progress in computer vision and other fields. The capacity to extract three-dimensional information concurrently using an end-to-end technique is ideal for hyperspectral image classification. Eq. (2)(2) $\begin{array}{rl}& v_{ij}^{xyz}=\\ & tanh\left(b_{ij}+{\sum }_m{\sum }_{h=0}^{H_i-1}{\sum }_{w=0}^{W_i-1}{\sum }_{s=0}^{S_i-1}{k}_{ijm}^{hws}{V}_{\left(i-1\right)m}^{\left(x+h\right)\left(y+w\right)\left(z+s\right)}\right)\end{array}$(2) describes the 3D convolution operation:

(2) $\begin{array}{rl}& v_{ij}^{xyz}=\\ & tanh\left(b_{ij}+{\sum }_m{\sum }_{h=0}^{H_i-1}{\sum }_{w=0}^{W_i-1}{\sum }_{s=0}^{S_i-1}{k}_{ijm}^{hws}{V}_{\left(i-1\right)m}^{\left(x+h\right)\left(y+w\right)\left(z+s\right)}\right)\end{array}$(2)

where $\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}\left(\cdot \right)\hspace{0.17em}$is the hyperbolic tangent function, ${\mathrm{b}}_{\mathrm{i}\mathrm{j}}$ is the bias for the feature map, ${\mathrm{S}}_{\mathrm{i}}$ is the size of the 3D kernel along the spectral dimension, ${\mathrm{k}}_{\mathrm{i}\mathrm{j}\mathrm{m}}^{\mathrm{h}\mathrm{w}\mathrm{s}}$ is the $\left(\mathrm{h},\mathrm{w},\mathrm{s}\right)$ value of the kernel connected to the $m$-th feature map in the previous layer (Ji et al. 2013).

In terms of image classification, 3D convolution encompasses both spatial and spectral convolution operations. We used a pixel-level-based 3D-CNN classification model established by Zhang, Zhao, and Zhang (2020) to evaluate the specific circumstance of PWD damage (the infected trees are rather spread).

In this study, the tree canopy of most field investigation trees was less than 3 m, thus, each patch was set to 19 × 19 for HSI and 43 × 43 for MSI and MSI&TIR datasets. For 2D-CNN, three principal components were generated from the 270 bands of the original data using the PCA method for HSI dataset, and 19 × 19 × 3 data were extracted as original features. And for the MSI and MSI&TIR datasets, the original features were 43 × 43 × 5 and 43 × 43 × 6, respectively. The network included three convolutional layers and two pooling layers (Figure 4(a)). The size of the convolution kernel was 5 × 5, and the numbers of convolution kernels per layer was 32, 64, and 128.

Figure 4. Architecture of the 2D- and 3D-CNN models for HSI dataset. (A) is the architecture of 2D-CNN and (B) is for 3D-CNN.

For 3D-CNN, the model added the Dropout to the last 3D convolution and the first fully connected layers. It sets the output of the neurons to 0 according to a set probability of 0.5. The batch size and the number of epochs were set to 64, and 300, respectively. The training process was completed using an SGD optimizer. A Rectified linear unit (ReLU) was used as the activation function. The Grid search method was used to select a number from {0.01, 0.03, 0.001, 0.003, 0.0001, 0.0003} as the learning rate to train the model for 300 epochs. Based on the changes in accuracy and loss during training and the classification results, the optimal learning rate was set as 0.001. For the MSI and MSI&TIR image datasets, the deep learning classification network contained two 3D convolutional layers with a fixed kernel 3 × 3 × 3. Tensorflow and Keras, two open source deep learning frameworks, were used to construct the input dataset and implement deep learning approaches, respectively. The operating platform hardware configuration included an Intel (R) Core (TM) i7–7700 CPU @ 3.60 GHz, 32GB memory, and NVIDIA TITAN Xp COLLECTORS EDITION GPU.

3.3. PWD infection-stage classification by thermal band

Two simple tests were designed to investigate the PWD infection-stage classification performance of the thermal band. On the one hand, thermal band was used as a new dataset to classify the infection stages using threshold segmentation. On the other hand, we extracted blue, green, and red bands from the original MSI data set to create new RGB composite images, which were then combined with the thermal band as a new RGB&TIR dataset. RGB&TIR dateset was then used to classify different PWD infection stages using the relevant optimal classification algorithm (the algorithm that had the best classification performance for different PWD infection-stage detection using the MSI&TIR dataset). The MSI and MSI&TIR datasets were used to compare all categorization results from these two experiments.

3.4. The comparison of PWD-infected Trees Detection and Spatial Resolution Response

Two comparison tests were designed to verify the reliability of the correlation analysis results. We selected the dataset with a relatively good classification effect as the standard dataset and chose the classification method that had the best classification effect corresponding to the selected standard data set as the standard classification method. These two comparative experiments are as follows:

1. Detection of PWD-infected trees. The “PWD-infected tree” was created by combining all three stages of infection (chlorosis, red, and gray) into one type. The background was described for all other ground types and healthy trees. In other words, the classification process was reduced to a simple binary classification problem. This is because in the actual PWN prevention and control policy, once a PWD infected tree is discovered, it is cut down regardless of the degree of infection.

2. PWD infection-stage classification using images with different spatial resolution. Based on the regularly used spatial resolution of satellite remote sensing images, the standard images were resampled into 0.3 m (equivalent to worldview-3), 0.8 m (corresponding to Gaofen-2), and 1.5 m (corresponding to SPOT6) spatial resolution. Resampled images with different spatial resolutions were classified using the standard classification method.

3.5. Accuracy assessment

To evaluate the reliability of the corresponding datasets and algorithms statistically, the confusion matrix, overall accuracy (OA), and kappa coefficient (Kappa) were calculated. The infection-stage information from the field survey, containing both the GPS and infection stages was used to stacking with each classification map to show a more intuitive effect. In addition, the time and economic cost, principle, and computational complexity were within the scope of the comprehensive evaluation indices.

4. Results

4.1. Sensitive band selection and spectral indices construction

The SPA was used to select sensitive bands of the HSI dataset. The results are shown in Figure 5. As the number of selected optimal bands increases, the RMSE shows a significant drop when the number of variables included in the model is four. Considering the selected band number and analysis accuracy, the final number of selected bands was four, with an RMSE of 0.42. The final selected sensitive bands were 413 nm, 691 nm, 720 nm, and 911 nm.

Figure 5. SPA bands selection result. (a) Variation in RMSE as the number of selected bands increase. (b) Final optimal selected bands.

According to the spectral index construction steps described in Section 3.2, the NDSI, DSI, RSI and other band combinations were calculated, their scatterplots were drawn based on the sensitive band selection results (HSI dataset). The sample band combination scatterplot of the HSI dataset is shown in Figures 6(a,b). Figure 6(a) shows that $\mathrm{N}\mathrm{D}\mathrm{S}\mathrm{I}\left(691,720\right)$ was successful in separating trees in the healthy stage, however, trees in the other three infection stages, particularly the red and gray stages, could not be separated. The simple index $0.5\times {\mathrm{R}}_{911}-3\times {\mathrm{R}}_{413}$ was useful for identifying healthy and gray stage trees (Figure 6(b)). It is worth noting that these two simple indices are somewhat, but not significantly, effective in distinguishing between chlorosis and red stages. The PWD infection-stage classification index for the HSI dataset ($\mathrm{P}\mathrm{H}\mathrm{S}\mathrm{I}$) was therefore constructed to widen the gap between the chlorosis and red stages, while also ensuring that the two simple indices already differentiated between the healthy and gray stages. Based on carefully searching for all possible combinations and comparing their scatterplots, the final PHSI developed here is shown in Eq.(3)(3) $\mathrm{P}\mathrm{H}\mathrm{S}\mathrm{I}=\hspace{0.17em}5.5\times {\mathrm{R}}_{911}-33\times {\mathrm{R}}_{413}+\mathrm{N}\mathrm{D}\mathrm{S}\mathrm{I}\left(691,720\right)$(3) The scatterplot (Figure 6(e)) indicates that the healthy, chlorosis, red, and gray stages had thresholds of 1.0, 0.5, and 0, respectively.

Figure 6. Scatterplot of different simple indices and the classification performance and thresholds of three construction spectral indices. (a) and (b) are the simple indices for the HSI dataset; (c) and (d) are the simple indices for the MSI dataset; (d), (e) and (f) are the classification performance of each spectral indices for HSI, MSI and MSI&Tir dataset. The different color dots are the spectral index values of four corresponding infection stages.

(3) $\mathrm{P}\mathrm{H}\mathrm{S}\mathrm{I}=\hspace{0.17em}5.5\times {\mathrm{R}}_{911}-33\times {\mathrm{R}}_{413}+\mathrm{N}\mathrm{D}\mathrm{S}\mathrm{I}\left(691,720\right)$(3)

Similarly, Figures 6(c,d) show the two simple indices generated by the scatterplot analysis of the MSI dataset. $\mathrm{N}\mathrm{D}\mathrm{S}\mathrm{I}\left(668,717\right)$ can effectively distinguish between healthy, chlorosis, and the other two infection-stage trees, while $0.7\times {\mathrm{R}}_{840}-3\times {\mathrm{R}}_{475}$ can better distinguish the gray stage trees. Like $\mathrm{P}\mathrm{H}\mathrm{S}\mathrm{I}$ construction, the final PWD infection-stage classification index ($\mathrm{P}\mathrm{M}\mathrm{S}\mathrm{I}$) is shown in Eq. (4)(4) $\mathrm{P}\mathrm{M}\mathrm{S}\mathrm{I}=8.4\times {\mathrm{R}}_{840}-36\times {\mathrm{R}}_{475}+\mathrm{N}\mathrm{D}\mathrm{S}\mathrm{I}\left(668,717\right)$(4) The thresholds corresponding to the healthy, chlorosis, red, and gray stages, as determined by the scatterplot (Figure 6(f)), were 2.0, 1.1, and 0.65, respectively. For the MSI&TIR, the final PWD infection-stage classification index ($\mathrm{P}\mathrm{T}\mathrm{S}\mathrm{I}$) is shown in Eq. (5)(5) $\mathrm{P}\mathrm{T}\mathrm{S}\mathrm{I}=8.4\times {\mathrm{R}}_{840}-36\times {\mathrm{R}}_{475}+\mathrm{N}\mathrm{D}\mathrm{S}\mathrm{I}\left(668,717\right)-0.1\times {\mathrm{R}}_{11000}$(5) , and the thresholds were −1.5, −2.65, and −3.3, respectively. Figure 6(g) shows its scatterplot.

(4) $\mathrm{P}\mathrm{M}\mathrm{S}\mathrm{I}=8.4\times {\mathrm{R}}_{840}-36\times {\mathrm{R}}_{475}+\mathrm{N}\mathrm{D}\mathrm{S}\mathrm{I}\left(668,717\right)$(4)

(5) $\mathrm{P}\mathrm{T}\mathrm{S}\mathrm{I}=8.4\times {\mathrm{R}}_{840}-36\times {\mathrm{R}}_{475}+\mathrm{N}\mathrm{D}\mathrm{S}\mathrm{I}\left(668,717\right)-0.1\times {\mathrm{R}}_{11000}$(5)

4.2. Classification results of different image sets with different methods

Table 3 shows the final classification parameter combinations as well as the classification accuracy of the validation set under the different classification algorithms and datasets. It is clear that the HSI and MSI&TIR data sets can achieve higher classification accuracy using RF as well as the two deep learning algorithms. The results of the spectral analysis were the lowest. Under this condition, we used the same model for PWD classification on the test set, and carefully analyzed the categorization outcomes of the test set to confirm this finding.

Table 3. The overall accuracy and kappa coefficient of validation set under different classification algorithms for different datasets.

Algorithms		HSI	MSI	MSI&TIR
Spectral Analysis	Model	$\mathrm{P}\mathrm{H}\mathrm{S}\mathrm{I}=\hspace{0.17em}5.5\times {\mathrm{R}}_{911}-33\times {\mathrm{R}}_{413}+\mathrm{N}\mathrm{D}\mathrm{S}\mathrm{I}\left(691,720\right)$	$\mathrm{P}\mathrm{M}\mathrm{S}\mathrm{I}=8.4\times {\mathrm{R}}_{840}-36\times {\mathrm{R}}_{475}+\mathrm{N}\mathrm{D}\mathrm{S}\mathrm{I}\left(668,717\right)$	$\mathrm{P}\mathrm{T}\mathrm{S}\mathrm{I}=8.4\times {\mathrm{R}}_{840}-36\times {\mathrm{R}}_{475}+\mathrm{N}\mathrm{D}\mathrm{S}\mathrm{I}\left(668,717\right)-0.1\times {\mathrm{R}}_{11000}$
	OA/%	73.54	84.37	85.15
	Kappa	0.68	0.72	0.86
SVM	Model	SVM(20,0.5)^a	SVM(100,0.2)	SVM(100,0.17)
	OA/%	87.94	86.14	89.65
	Kappa	0.86	0.77	0.88
RF	Model	RF-1000^b	RF-100	RF-100
	OA/%	93.63	87.20	97.22
	Kappa	0.91	0.83	0.97
2D-CNN	Model	three convolutional layers and two pooling layers with the fixed kernel of 5 × 5	three convolutional layers and two pooling layers with the fixed kernel of 5 × 5	three convolutional layers and two pooling layers with the fixed kernel of 5 × 5
	OA/%	89.62	86.45	94.91
	Kappa	0.81	0.81	0.92
3D-CNN	Model	Five 3D convolutional layers with the fixed kernel of 3 × 3 × 7	Two 3D convolutional layers with the fixed kernel 3 × 3 × 3	Two 3D convolutional layers with the fixed kernel 3 × 3 × 3
	OA/%	92.62	86.32	97.58
	Kappa	0.90	0.81	0.96

^aSVM(i,j) indicates that the SVM classification is performed using a parameter combination of (penalty coefficient, the kernel parameter) in the G-RBF kernel.

^bRF-n indicates that the fandom forest classification is performed using the n trees.

4.2.1. Classification accuracy assessment

The classification confusion matrices for different classification algorithms using different test datasets are shown in Figure 7. It can be observed that the healthy and gray stages are accurately characterized regardless of the algorithm used for any given data set. With the MSI&TIR dataset, the classification accuracy of healthy stage trees by SVM and RF classification methods achieved 100%.

Figure 7. The classification confusion matrices (percent) of different algorithms with different datasets. The horizontal axis is the ground truth and the vertical axis is the classification result.

The classification accuracy of the same dataset using different classification algorithms varies significantly in the longitudinal view of Figure 7. For the HSI dataset, $\mathrm{P}\mathrm{H}\mathrm{S}\mathrm{I}$ displayed the lowest classification accuracy. The chlorosis stage was misclassified as healthy stages, the healthy and red stages were misclassified as chlorosis stage, with misclassification rates of 16.85%, 17.39% and 52.16%, respectively. This is essentially the same as the classification performance of the $\mathrm{P}\mathrm{H}\mathrm{S}\mathrm{I}$ shown in Figure 6E. SVM and 2D-CNN techniques perform almost identically, with red and gray stage classification accuracies exceeding 94%. Regardless of the stage, the machine learning and deep learning algorithms produced better classification results, especially the RF and 3D-CNN algorithms, which achieved greater than 70% classification accuracy. For the MSI dataset, the RF and 3D-CNN classification algorithms also have the highest classification accuracy, whereas the spectral analysis method of $\mathrm{P}\mathrm{M}\mathrm{S}\mathrm{I}$ has the lowest classification accuracy. Misclassification continues to be most common in the chlorosis and red stages. Regardless of the algorithm used, the chlorosis stage had the lowest classification accuracy. When compared to that of the other two datasets, the correct classification accuracy rate of the chlorosis stage based on the MSI&TIR dataset was significantly higher, with even the lowest classification technique, PTSI, achieving 79.47%. The best classification results were obtained using the RF and 3D-CNN algorithms, and the SVM and 2D-CNN results for the MSI&TIR data set were excellent.

In terms of the horizontal axis of Figure 7, while the chlorosis stage classification accuracy of the HSI and MSI datasets is not perfect, the RF and 3D-CNN produce the best classification results for each infection stage. The categorization results of the HSI and MSI&TIR datasets were much better than those of the MSI dataset. The correct classification rates of chlorosis and red stages, in particular, exceeded 92% using the MSI&TIR dataset. When using the HSI and MSI datasets, the misclassification rates of chlorosis and red stages under the spectral analysis method were extremely high, but they improved significantly when using the MSI&TIR dataset. Similarly, while the SVM and 2D-CNN algorithms have low classification accuracy, the MSI&TIR dataset produced the best results. Overall, regardless of the classification algorithm used, the MSI&TIR dataset produced the best categorization results. Although HSI has high classification accuracy, the misclassification of the chlorosis and red stages should not be ignored.

Table 4 shows the OA and Kappa of the different classification algorithms for the different test sets. The MSI dataset had the lowest classification accuracy of all tested algorithms. Even the best RF achieved an OA of only 84.79%, a difference of less than 3% than that of the worst spectral analysis classification in MSI&TIR. For the MSI&TIR dataset, even when using the PTSI classification method, the OA and Kappa attained over 81% and 0.74, respectively, which were much better than the other datasets. In other words, the RF and 3D-CNN algorithms performed consistently and effectively across three different datasets, with OA of more than 81%, even on the MSI dataset. In contrast, the spectral analysis method was less stable, with a major difference in accuracy across the three datasets. Comparing Tables 3 and 4, despite the fact that the OA and Kappa acquired from Site B were generally lower than those obtained from Site A, MSI&TIR had the best classification accuracy under RF and 3D-CNN algorithms, regardless of area, whereas the spectral analysis method had the most unstable classification technique and the lowest classification accuracy for the MSI dataset.

Table 4. The overall accuracy and kappa coefficient of test set under different classification algorithms for different datasets.

Algorithms		HSI	MSI	MSI&TIR
Spectral Analysis	OA/%	63.10	81.22	81.60
	Kappa	0.48	0.74	0.75
SVM	OA/%	84.90	81.54	92.32
	Kappa	0.79	0.75	0.90
RF	OA/%	87.60	84.79	94.26
	Kappa	0.83	0.79	0.92
2D-CNN	OA/%	85.27	81.33	90.48
	Kappa	0.79	0.75	0.87
3D-CNN	OA/%	87.46	81.39	93.30
	Kappa	0.83	0.74	0.90

In addition, we performed a two-way analysis of variance (ANOVA) on the OA of the classification of the four models on both the validation and test sets to further illustrate the variability between the models. The results are shown in Table 5. Comparing the three F-statistics and the corresponding F-critical values, we found that the F-statistics of the dataset were all greater than the F-critical values, indicating that the data and the algorithm have significant effects on classification accuracy. Furthermore, all three P-values in the results are less than 0.05, indicating that the data, algorithm, and their interaction have a significant effect on classification accuracy.

Table 5. The two-way ANOVA results.

Source	SS	df	MS	F	P-value	F crit
Methods	605.26	4	151.32	12.82	<0.001	3.06
Data sets	358.82	2	179.41	15.20	<0.001	3.68
Interaction	292.32	8	36.54	3.10	0.03	2.64
Internal	177.05	15	11.80
Total	1433.46	29

4.2.2. Classification mapping

Figure 8 depicts the distribution of different PWD infection stages using different algorithms for each dateset. It can be observed that the classification result of PHSI is the worst, with essentially no classification of the chlorosis stage accomplished. In addition, the HSI dataset has such significant clustering that it cannot be properly differentiated from the individual tree scale. In contrast, the tree distribution of the MSI and MSI&TIR datasets seems to be more realistic. It is worth noting that the classification results of tree crowns in red and gray stages are more prominent and clustered in the MSI and MSI&TIR datasets. The chlorosis stage classification results show the “Pepper and Salt” phenomena, which is particularly noticeable in places with large areas of bare land and low canopy density. In particular, the chlorosis stage is successfully represented when the MSI&TIR dataset is combined with the RF algorithm, which has an exceptionally good classification performance. According to the classification results in Figures 7 and 8, the MSI&TIR dataset had the best results under the RF and 3D-CNN algorithms.

Figure 8. The classification results of different algorithms using different datasets.

To comprehensively analyze the accuracy of the classification results, tree crowns in different infection stages were labeled, and the field observations were overlaid with the map obtained from RF classification using the MSI&TIR dataset. The results are shown in Figure 9. By comparing the locations of the investigated trees and the infection stages, we found that the gray stage classification was 100% correct, with the largest misclassification occurring in the chlorosis stages. This is consistent with the previous quantitative analysis results (Figure 7 and Table 4).

Figure 9. Comparison mapping with investigation results. Circles and crosses indicate the remote sensing classification results and investigation results, respectively.

4.3. Classification results of using thermal images

Table 6 shows the test results using thermal images to classify the PWD infection stages. As can be observed, first, the classification accuracy of single thermal images was the worst and was inadequate to meet investigation requirements. However, the chlorosis stage classification accuracy was not significantly lower than that of the other stages, as with the other data sources. Second, while the composited RGB dataset had lower classification accuracy in the chlorosis and red stages, the RGB&TIR dataset was significantly better at identifying trees in the healthy, chlorosis, and gray stages, especially in the chlorosis stage. The classification accuracy exceeded the recognition accuracy under the MSI dataset. Third, an overall comparison of the results of the datasets with TIR band (RGB&TIR and MSI&TIR) and without the TIR band (RGB and MSI) shows that increasing the TIR band was highly significant in improving chlorosis stage detection. These findings confirm the effectiveness of thermal imaging for detecting the chlorosis stage.

Table 6. Comparisons of classification accuracy among different data sources.

Evaluating Indicators	TIR	Composited RGB	RGB&TIR	MSI	MSI&TRI
Healthy stage (%)	68.17	90.32	90.62	87.27	100.00
Chlorosis Stage (%)	45.78	56.98	71.83	69.27	92.73
Red Stage (%)	49.72	65.86	83.45	99.23	91.97
Gray Stage (%)	52.12	84.12	89.33	95.05	97.38
OA (%)	53.22	72.54	81.64	84.79	94.26
Kappa	0.50	0.70	0.74	0.79	0.92

Furthermore, the MSI-based dataset had a much higher classification accuracy than the RGB-based datasets. This provides additional evidence for the utility of red-edge and NIR bands in forest disease identification.

4.4. PWD-infected trees detection and spatial resolution response

Because the combination of the MSI&TIR and RF classification algorithms had the highest accuracy in detecting different PWD infection stages (OA = 94.26% and Kappa = 0.92), the MSI&TIR dataset was used as the standard data source and the RF classifier as the standard classification algorithm. The number of simple decision trees was set to 100 for consistency.

(I) PWD infected individual tree detection results. Figure 10 shows the resulting map. It can be seen that infected trees are well detected using the MSI&TIR dataset with the RF classification algorithm, and that the results are highly consistent with the survey results (only one tree was not detected). The OA and Kappa were 95.83% and 0.89, respectively.

Figure 10. Classification results and infected tree crown delineation maps based on the MSI&Tir dataset.

(II) Results of PWD infection-stage classification using different spatial resolution images. Table 7 shows the results of the PWD infection-stage classification using images with different spatial resolution. Previous results of the MSI&TIR dataset with a 0.062 m spatial resolution were used as the reference. It can be seen that when spatial resolution lowers, the classification accuracy decreases to a certain extent. Once the spatial resolution was less than 1.0 m, the classification accuracy of almost each different infection-stage was less than 75%.

Table 7. Comparisons of classification accuracy among different spatial resolution datasets.

Evaluating Indicators	Original 0.053 m	Resampled 0.3m	Resampled 0.8m	Resampled 1.5m
Healthy stage (%)	100	95.12	89.32	80.32
Chlorosis Stage (%)	92.73	90.21	74.78	59.33
Red Stage (%)	91.97	90.89	84.93	72.18
Gray Stage (%)	97.38	97.22	86.64	74.54
OA (%)	94.26	92.83	86.98	69.93
Kappa	0.92	0.90	0.86	0.65

5. Discussion

5.1. THe feasibility of detecting trees affected by the PWN using UAV-based remote sensing

In this study, the UAV platform was equipped with both multispectral and hyperspectral sensors and flew at an altitude of 120 m to acquire data sources. The spatial resolution of HSI, MSI and thermal band was 0.14 m, 0.053 m, and 0.062 m, respectively. On the one hand, most of the classification results exceeded 80% in terms of the overall classification accuracy of different classification algorithms under different datasets (Tables 3 and 4), and it is clear from Figure 10 that the positioning information of a individual infected tree is accurate. On the other hand, the crown size of pine trees in this study was approximately 3 m or less, and the regions were both medium-density forested, necessitating the use of a UAV platform. Furthermore, the results of the comparative experiments (I) and (II) highlight the need to obtain high spatial resolution images for detecting PWD.

These results are consistent with the Joint Research Center technical report titled “The feasibility of detecting trees affected by the Pine Wood Nematode using remote sensing” in 2015, which pointed out that “the prevalence of pine trees with projected crown diameters of 2 m or less and this phenomenon dictates the necessity of very high-resolution imagery for tree-level forest monitoring in the PWN threat.” Although the actual implementation of UAVs for forest management and investigation still needs to consider a number of challenges, such as endurance, position signals, and the personnel, the applicability of UAV-based remote sensing in both spectral, spatial, and temporal resolution aspects cannot be ignored for PWD infection-stage detection. Thus, according to the analysis presented above, using UAVs to identify PWD is technically possible and has been enacted in forest surveys; however, it serves as a supplemental method. It is anticipated that this will strengthen their position in future practical applications through hardware and service staffing.

5.2. The performance of different optical imaging in different PWD infection- stage detection

According to the analysis of the physiological change process of the host pine tree, after PWN infects the host vegetation, it will gradually lead to the obstruction of water transport, followed by a drop in transpiration rate and further water loss until needles wilt, and can result in plant death. When water stress exceeds a specific threshold, the lamellar structure of chloroplasts in needles is damaged, and the chlorophyll content decreases. However, at the beginning of stress, carotenoids perform a defensive role, and then trigger the plants’ own defense mechanism. Canopy water temporarily increases at this time (Xu, Luo, and Zhang 2012). Thus, theoretically, the spectral continuity of HSI data can make it easier to detect, monitor, and identify subtle changes in the target vegetation. However, as shown in Figure 7, the classification accuracy of the HSI dataset is significantly lower than that of the MSI&TIR data set regardless of the method used, and the number of correct classifications for each infection stage is also not particularly high. In face, as the number of bands in hyperspectral images increase, the amount of data noise also increase. Moreover, despite the high spatial resolution of UAV images, the presence of mixed pixels in complex landscapes is still significant. This not only increases the amount of data stored and transmitted but also has a significant impact on analysis accuracy. More importantly, when the number of training samples is insufficient, a more serious Hughes phenomenon exists in the classification of hyperspectral images. Additionally, we must consider the cost of the hyperspectral sensor as well as the time required for data analysis.

Throughout the study, the classification results of the MSI&TIR dataset were the most prominent. The number of correct classifications for all stages was extremely high. And, as demonstrated in Table 6, the thermal band has a significant performance for detecting chlorosis stage trees based on the classification accuracy of both the TIR and RGB&TIR datasets. Similarly, according to an analysis of physiological changes in host vegetation, because of the sharp decrease in water content in the early stage, both the transpiration rate and stomatal conductance decreased to a certain extent, scattered needles withered, and the canopy temperature changed. Corresponding to the three infection stages and classification accuracy, the MSI&TIR dataset is the most advantageous for early PWD infection-stage detection. Then, as the chlorophyll content decreased and the needles’ color changed from green to red and brown (scatter yellow), all three datasets could efficiently recognize trees in the red infection stage, and classification accuracy achieved 90%.

To summarize, the scalability of HSI is not strong, regardless of the quantitative analysis accuracy. the cost, time-consuming and operability. In terms of quantitative classification accuracy, the MSI&TIR dataset are the best choice for detecting each infection stage. Although it only adds one thermal infrared band to MSI, the chlorosis stage classification accuracy improves by nearly 22% (PMSI and PTSI). This increase is critical in preventing the spread of PWD during the chlorosis stage. And even without taking the infection stages into account, MSI&TIR still has excellent classification accuracy. However, when considering both the portability of data acquisition and the ease of data processing, temperature calibration of the thermal band remains a challenge. Currently, no single method for providing an easy calibration similar to the calibration plate and separate Downward Light Sensor for thermal imaging. In this case, an MSI data source with red-edge and NIR bands is also a good choice. Thus, dependent on different practical requirements and by comparing the results of Tables 4 and 7, we believe that MSI&TIR data are superior for early PWD identification, while the simpler MSI dataset is more suitable for mid- to late-stage PWD infection detection in trees.

5.3. Different detection methods for different needs

The analysis of spectral changes with the physiological change process of the host pine tree revealed that particular spectral information was strongly related to each infection stage. In most cases, spectral analysis is associated with parametric modeling techniques, and optimal sensitive band and spectral index combinations are frequently employed. In this study, we constructed three spectral indices based on three datasets and used threshold segmentation to classify PWD infection stages. From Figure 8 and Tables 3 and 4, while the classification results of PMSI and PTSI can meet the needs of practical applications, the recognition accuracy of each stage is still lower than that of classical machine learning classification algorithms, especially PHSI. Notably, due to the spatial variability of spectral and spatial information, it is difficult to construct an effective spectral index that can be applied to all types of remote sensing data. This directly resulted in poor transferability and usage limitations (Tables 3 and 4). Therefore, this method is unsuitable for different areas and sensors. Unlike the spectral analysis approach, machine learning algorithms are freely distributed and can manage noisy data, missing data values, both numerical and categorical data, and incorporate over sources of auxiliary spatial data (Stone and Mohammed 2017). Both SVM and RF classifiers were used to classify different PWD infection stages. Aside from the chlorosis stage classification results, accuracy was reasonably constant across datasets. As shown in Tables 3 and 4, machine learning algorithms are effective and reliable for the classification and recognition of forest diseases. Furthermore, a large amount of commercial software provides ideal and simple process platforms that can support the basic data analysis. However, the machine learning classifiers often have low susceptibility to overfitting, and some decisive parameters are determined subjectively (such as the kernel parameters in SVM and tree numbers in RF). As a result, when the trial-and-error method is used to determine the optimal parameters, the time cost is considerable. Further, we also used 2D- and 3D-CNN to classify the PWD infection stages. It is obvious that the classification accuracy of deep learning algorithms is not outstanding compared to that of machine learning algorithms. This could be due to our network selection and training sample size. However, it is worth noting that the hardware requirements and effective data volume for model training are very high, and this process is time-consuming.

In summary, if the flight altitude is low enough, the canopy density is suitable for obtaining pure pixels of all different infection stages, and there is sufficient corresponding ground survey data (pigments and water content at the minimum), the spectral analysis method is the best choice, especially for the red and gray stages. This method is relatively simple and can provide a clear explanation of the whole disease infection process through physiological changes. If there is a lack of corresponding ground-based information, the deep learning algorithm may be the best in terms of its high accuracy and lack of need to consider the selection of classification features. Specifically, when selecting a classification algorithm, it is important to evaluate not only the efficiency and accuracy of an algorithm but also the data source types and actual application environments. For PWN damage mechanism research, the highest accuracy (both spectral and data itself, as well as the result accuracy) may be the first consideration, but for actual forest disease investigation, easy and effective system integration is most significant. Consequently, when combined with available data sources, the classic machine learning approach, especially the RF classifier is arguably the best choice for multispectral data. The choice of optimal data analysis algorithm strongly depends on the specific problem. Therefore, it is not possible to provide general recommendations, but some criteria and simple results obtained in this study could help to identify applicable algorithms.

5.4. Limitations and summary

Manual forest disease surveying is time-consuming, labor-intensive, and prone to errors and omissions, particularly during the early stages of identification. Using low-altitude remote sensing and multi-modal data, the images and techniques used in this study showed positive results in identifying various PWD infection stages. However, this research has certain limitations.

On the one hand, the study’s research site was a pure pine forest that only suffered PWD damage. Although we tested the results in a coniferous-broad-leaf mixed forest (Site B), we eliminated the scattered broad-leaf trees in advance, and the classification effect was not as good as that of Site A (the influence of understory vegetation occupies a certain percentage). However, a greater percentage of forests are theropencedrymion, or multi-species mixed coniferous forests, and these forests are affected by numerous pests and diseases. Further investigation is necessary to effectively distinguish between PWD damage and the damage caused by other pests and diseases. Considering the aforementioned issues, using time-series imaging data that can satisfy the biological and ecological characteristics of pests and diseases or the phenological characteristics of vegetation, it is possible to distinguish between various tree species as well as various diseases and insect pests. Effective combination or fusion methods of both images and point cloud data (UAV-based, ground-based, or ultra-high resolution image-based data) can be presented and used to distinguish between coniferous and broad-leaf trees based on structural and spectral features.

On the other hand, the infected tree canopies were manually delineated. However, manual delineate approach cannot be used for large-scale surveys, even the forest farm scale, which takes a lot of labor and material resources. Nowadays, there are a few well established techniques for automated forest canopy extraction. These techniques can be combined with infection-stages detection algorithms, allowing for full automation of PWD infection trees detection. In future research, we can begin with weakly supervised or unsupervised classification algorithms to achieve more accurate and efficient individual infected-tree identification under the premise of automated crown extraction.

6. Conclusion

In this research, one hyperspectral dataset (HSI) and two multispectral datasets (MSI and MSI&TIR) were compared to detect PWD infection stages at the single tree level. The validity of these three UAV-based optical image datasets and the applicability of different classification algorithms for each dataset were analyzed. Importantly, the efficiency of the thermal dataset for the early PWD detection was verified.

The MSI&TIR dataset performs well in classifying the infection stages, especially the chlorosis stage. On the one hand, the combination usage of both red-edge and NIR bands is the most significant for PWD infection-stage classification or other forest pest and disease detection, which encompasses discolor changes. Furthermore, it is obvious that multispectral sensor imagery is less expensive than hyperspectral images in actual investigation and is frequently available at a fine spatial resolution, making MSI&TIR an excellent tool for classifying, detecting, and mapping the PWD, particularly at the single tree level. On the other hand, when compared to the common classification algorithms using different datasets, the machine learning and deep learning algorithms performed consistently better than spectral analysis for each infected stage, especially the chlorosis stage classification using the MSI&TIR dataset. However, when the infected trees entered the gray stage, the accuracy varies by no more than 10% between the various algorithms (SVM, RF, 2D- and 3D-CNN algorithms). Thus, we should consider not only the method itself but also basic properties such as the number of features and training samples, representatives of samples, and classification systems. Thus, when the MSI&TIR data are chosen as data sources for PWD infection-stage detection, RF classification techniques may be considered the best option.

Author contributions

N.Z. proposed the main idea, designed the methodology, and wrote the manuscript; X.C. guided the algorithms and writing of the manuscript; N.L. supervised the field surveys and conducted the photogrammetric processing; J.Z detailed all of the steps of the UAV-based data acquirement; T.S. revised the manuscript.

Acknowledgments

We would like to thank Precision Forestry Key Laboratory of Beijing of Beijing Forestry University for the help of PWD field investigating and Editage [www.editage.cn] for English language editing.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The data that support the findings of this study are available from the first author, [Ning Zhang, [email protected]], upon reasonable request.

Additional information

Funding

This work was supported by the [National Natural Science Foundation of China] under Grant [number 31901240, 31870534, and 31971792]; [Central Public-interest Scientific Institution Basal Research Fund] under Grant [number Y2022QC17]; and [DRAGON 5 COOPERATION] under Grant [number 59257].