Using time-series imagery and 3DLSTM model to classify individual tree species

ABSTRACT

Classification of individual tree species (ITS) is critical for fine-scale forest surveys. However, it is difficult to obtain the complete and high-precision data needed for ITS classification in large areas. Lower spatial resolution time-series imagery is more accessible than other types of imagery and contains rich phenological information. In this study, after delineating individual tree crowns using a 0.2-m unmanned aerial vehicle (UAV) image, we used 3-m time-series imagery and a new 3DLSTM model to identify ITS at the Gaofeng Forest Farm in Guangxi Province, China. The 3DLSTM ITS classification model combines three-dimensional convolutional neural network (3D CNN) and long short-term memory (LSTM) models; thus, spatial, multiband, and time-series information can be extracted simultaneously to identify ITS more accurately. In this study, when only 3-m Planet time-series imagery was used for classification, the 3DLSTM model offered an ITS classification accuracy of 92.68%, outperforming two ITS classifiers (DenseNet or AlexNet model) based on one individual image. Moreover, the 3DLSTM model was better at identifying broad-leaved tree species than other deep-learning models. The experimental results proved that ITS classification could be significantly improved using only 3DLSTM and time-series images, offering the possibility of classifying large-scale ITS at a low cost.

KEYWORDS:

• Remote sensing
• individual tree species classification
• time-series imagery
• deep learning

1. Introduction

Forests are the mainstay of terrestrial ecosystems and the largest carbon reservoirs and sinks on land (Hansen et al. 2013). Correct classification of individual tree species (ITS) is important for forest resource management (Chemura, van Duren, and van Leeuwen 2015; Yin and Wang 2016). With the development of remote sensing technology, it has become possible to classify ITS on a large scale. Many studies have used remote sensing imagery to classify ITS (Dalponte et al. 2015; Nguyen, Demir, and Dalponte 2019; Puttonen, Litkey, and Hyyppa 2010; Zhang and Hu 2012). For instance, Dalponte et al. (2014) used support vector machines, the majority voting rule, and hyperspectral and airborne laser scanning (ALS) data to classify pine, spruce, and broadleaf trees in the municipality of Aurskog-Høland, southeastern Norway, with an overall classification accuracy of 93.5%. Qin et al. (2022) used a random forest classifier, unmanned aerial vehicle (UAV)-based LiDAR, hyperspectral, and ultrahigh-resolution red-green–blue (RGB) data to classify eighteen tree species in Julongshan Park, a subtropical broadleaf forest situated in Shenzhen City in southern China, with an overall classification accuracy of 91.8%.

The data currently used for ITS classification are high-spatial-resolution multispectral, hyperspectral imagery, and LiDAR data. (Deng et al. 2016; Lee et al. 2016; Shen and Cao 2017; Terryn et al. 2020; Xu et al. 2020). High spatial resolution remote sensing imagery is expensive and not sufficiently rich in spectral information; hyperspectral remote sensing imagery and LiDAR data are frequently unavailable or unsuitable for large-scale classification studies (Bunting et al. 2010; Fassnacht et al. 2016; Hartling, Sagan, and Maimaitijiang 2021; Zhao et al. 2020). In addition, when remote sensing imagery is used for ITS classification, only the spectral, textural, or structural features of monotemporal remote sensing images are generally used, without considering the phenological information of ITS in time-series imagery (Wu et al. 2022). Lower spatial resolution remote sensing imagery is cheaper and has a richer database inventory; therefore, time-series imagery with lower spatial resolution in large study areas is more readily available than ultrahigh-resolution imagery (Zheng and Zhu 2017). Different species of broadleaf trees have different growth cycles and characteristics of budding, maximum depression, and defoliation; therefore, phenological information on tree species present in time-series imagery can be used to help classifiers complete classification more quickly and accurately (Belcore et al. 2021; Bjerreskov, Nord-Larsen, and Fensholt 2021; Fassnacht et al. 2016; Hoscilo and Lewandowska 2019; Immitzer et al. 2019; Lechner et al. 2022; Michez et al. 2016; Miyoshi et al. 2020). The use of lower-resolution time-series imagery for ITS classification with high accuracy and methods to accomplish ITS classification more efficiently have become challenges that need to be met.

To increase the classification accuracy of the ITS, the current classification model can be improved besides changing the data used. Deep learning, as an efficient image classification method, has been used to classify ITS in remote sensing imagery. (Hamraz et al. 2019; Miyoshi et al. 2020; Sothe et al. 2020; Sun et al. 2019). For instance, Xi et al. (2020) used 13 classifiers, including PointNet++, ResNet, and terrestrial laser scanning (TLS) point cloud data, to classify nine dominant tree species in Canada and Finland, with PointNet++ having the best performance and a mean intersection over union (mIoU) accuracy of over 90%. Onishi and Ise (2021) used RGB imagery captured by a UAV and an object-based convolutional neural network (CNN) to classify seven tree species at the Kamigamo Experimental Station of Kyoto University, with an overall classification accuracy of over 90%.

Most of the currently used deep learning models are two-dimensional convolutional neural networks (2D CNNs), which can extract flat features well (Fromm et al. 2019; Kattenborn et al. 2020; Milioto, Lottes, and Stachniss 2017; Neupane, Horanont, and Hung 2019; Wagner et al. 2020; Weinstein et al. 2019). However, the output of a 2D CNN consists of 2D feature maps, which rely on the image of each frame to extract features, but do not include interframe movement information in the band dimension (Kattenborn et al. 2021). A three-dimensional convolutional neural network (3D CNN) proposed by Ji et al. (2013) can better capture the interframe motion information in the band dimension than a 2D CNN, and its output retains 3D feature maps. Furthermore, 3D CNN can better capture band and spatial information in images (Ayrey and Hayes 2018; Barbosa et al. 2020; Jin et al. 2020; Lottes et al. 2018; Zhong, Hu, and Zhou 2019). For example, Nezami et al. (2020) used 3D CNN and RGB imagery combined with a Fabry-Pérot interferometer (FPI) orthomosaic to classify three tree species in the Vesijako research forest area in the municipality of Padasjoki, southern Finland, with an overall classification accuracy of over 95%. Mayra et al. (2021) used 3D CNN and hyperspectral LiDAR data to classify four tree species in the Evo forest area in Hämeenlinna, southern Finland, with an overall classification accuracy of 87%.

When considering multiband and multitemporal remote sensing images, spatial, multiband, and time-series information should be extracted simultaneously, if possible, to classify the ITS more accurately. A 3D CNN can effectively work with 3D information; however, for time-series information (information in the fourth dimension), it is necessary to find a suitable model for extraction. The long short-term memory (LSTM) model exhibits good performance in time-series information processing (Deng, Huang, and Xu 2022; Li, Shen, and Yang 2021; Wang et al. 2019; Xi et al. 2021). LSTM model is a temporal recurrent neural network proposed by Hochreiter and Schmidhuber (1997) and has been improved and generalized by many researchers in subsequent studies (He et al. 2021; Sayah et al. 2021; Xi et al. 2021; Yang et al. 2018; Zhang et al. 2020).

Before the input data are processed in the regular LSTM unit, they must be spread into a one-dimensional vector, which results in the loss of spatial context information. To solve this problem, Shi et al. (2015) proposed the ConvLSTM structure, in which the input data and intermediate results are tensors rather than vectors. Therefore, the ConvLSTM structure can provide results corresponding to each pixel of the input data to facilitate the processing of spatial information. However, for multiband multitemporal remote sensing imagery, the spatial information of each multiband image should be fully utilized, in addition to extracting time-series information using the LSTM model. Therefore, ConvLSTM must be extended to Conv3DLSTM (Wang et al. 2019) to fully utilize the spatial, band, and temporal information in the imagery. There have been few studies on ITS classification using multitemporal imagery and the Conv3DLSTM module, so determining how to make full use of spatial, multiband, and multitemporal information in time-series imagery to classify ITS accurately has become another challenge to be solved.

In this study, we combined 3D CNN and LSTM models to construct a 3DLSTM model for ITS classification based on lower-resolution multitemporal images (3 m) in the study area of the Gaofeng Forest Farm, Guangxi Province, with the aim of obtaining a more efficient high-precision ITS classification model for providing the ability to classify large-scale ITS at low cost.

2. Study area and experimental data

2.1. Study area

The study area was located in the central area of the Gaofeng Forest Farm in Yulin, Guangxi Zhuang Autonomous Region (Figure 1(a)). The terrain of Guangxi Province is high in the northwest and low in the southeast, tilting from northwest to southeast. Gaofeng Forest Farm is located in the central-southern area of Guangxi Province, and the landform mainly consists of hills and mountains, with an average elevation of 200–500 m (Figure 1(b)). The Gaofeng Forest Farm is located in a subtropical monsoon climate zone dominated by secondary plantation forests (Figure 1(c)). The area is a typical southern Chinese forest with rich tree species and 87.5% forest cover (Wan et al. 2021).

Figure 1. Map of the study area. (a) Province in which the study area is located. (b) Digital Elevation Model of study area, with the red area indicating the study area. (c) Land use map of Guangxi Province. (d) UAV image of the study area with sampling points.

The specific spatial extent of the study area is 108°21′36″E to 108°23′38″E and 22°57′43″N to 22°58′59″N, covering an area of approximately 7.67 square kilometers (Figure 1(d)). According to a field survey, the tree species in the study area included Eucalyptus urophylla S.T. Blake (Eu.s), Cunninghamia lanceolata (Lamb.) Hook. (Cl.h), Pinus massoniana Lamb (Pm.l), Eucalyptus grandis x urophylla (Eg.x), Illicium verum Hook. f (Iv.h), Castanopsis hystrix A. DC (Ch.a), Michelia odora (Chun) Nooteboom & B. L. Chen (Mo.n), and Manglietia glauca Blume (Mg.b).

2.2. Overview of dominant tree species

Field surveys indicated that the dominant tree species in the study area were Eucalyptus urophylla S.T. Blake, Cunninghamia lanceolata (Lamb.) Hook., Pinus massoniana Lamb, Eucalyptus grandis x urophylla, Illicium verum Hook. f, Castanopsis hystrix A. DC, Michelia odora (Chun) Nooteboom & B. L. Chen, and Manglietia glauca Blume. The growth patterns of the dominant tree species in the study area are shown in Figure 2.

Figure 2. Growth cycle of major tree species.

2.3. Experimental data

In this study, the data used were RGB UAV, and Planet time-series imagery. The UAV image was used to delineate individual tree crowns accurately and to investigate the improvement of texture information on ITS classification. Planet time-series imagery was used for classifying ITS using time-series information. The spatial resolution, number of bands, and imaging times of the different images are listed in Table 1.

Table 1. The parameters of different images.

Image	Spatial Resolution	Band	Imaging Time
UAV image	0.2 m	blue	October 9, 2016
		green
		red
Planet time-series imagery	3 m	blue (420–530 nm)	September 15, 2016; October 14, 2016; February 13 and 15, 2017; March 2, 2017; April 13 and 15, 2017; August 22, 2017
		green (500–590 nm)
		red (610–700 nm)
		near-infrared (760–860 nm)

2.3.1. UAV image

One UAV image with an acquisition date of October 9, 2016, was selected for this study. The high-resolution charge-coupled device (CCD) camera used for acquiring UAV image was a Hasselblad H4D-60 with 60% and 30% overlap in the heading and lateral directions, respectively. A medium-format airborne digital camera system (DigiCAM-60) was used as the CCD sensor. DigiCAM-60 has 60 million pixels (8956 × 6708), a 1.6 s image repetition rate, and a 16 bit recording depth. The focal lens was 50 mm thick. This camera provided a spatial resolution of 12 cm with a flying altitude of 1000 m. This image has red, green and blue bands with a spatial resolution of 0.2 m, as shown in Figure 1(d).

2.3.2. Time-series imagery

After investigating the growth cycles of the dominant tree species in the forest (Figure 2), we selected Planet imagery containing the three main seasons of autumn, spring, and summer in September and October 2016 and February, March, April, and August 2017 as the time-series data source (Figure 1 in Supplemental material).

3. Method

3.1. Image preprocessing

After completing the radiometric calibration using ENVI software, orthorectification of the UAV image was performed using a local digital elevation model (DEM). The preprocessed UAV image was resampled to 3 m, and the resampled UAV image was used as the base map to align the Planet images of different periods. The interalignment accuracy of the Planet images and UAV image was approximately 0.1 pixels to ensure the accuracy of the results. Subsequently, composite images of 24 (six time-series images) or 27 (six time-series images + 3-band UAV image) bands were superimposed chronologically as the study images. To simplify the description, we refer to the combination of 24 images as Planet and the combination of 27 images as Planet-UAV. Because an individual tree occupies only one or two pixels at 3-m resolution, to obtain more pixels during deep learning training, we resampled the composite image to spatial resolution of 0.2 m before investigating the effect of ITS time-series information on classification.

3.2. Sample set construction and enhancement

3.2.1. Individual tree crown delineation

The construction of the ITS remote sensing imagery sample set requires identifying and cropping out the individual crowns of the entire remote sensing image and labeling the cropped images of the individual crowns with species categories.

In this study, crown slices from the imagery (CSI) algorithm proposed by Jing et al. (Jing et al. 2013) were used for the multiscale segmentation of remote sensing imagery and automatic delineation of crowns.

By comparing the delineation results of CSI algorithm and manual identification in pure forests of eight tree species (Table 1 of the Supplemental material), we obtained the accuracy of the CSI algorithm in the study area (Table 2), where the evaluation metrics refer to Qiu (Qiu et al. 2020).

Table 2. The accuracy of the CSI algorithm.

Tree species	Total Segments	Matched	Nearly Matched	Merged	Missing	Spilt
Eu.s	173	153	16	4	0	0
Cl.h	150	148	2	0	0	0
Pm.l	155	147	8	0	0	0
Eg.x	171	166	4	1	0	0
Iv.h	151	151	0	0	0	0
Ch.a	182	181	1	0	0	0
Mo.n	139	138	1	0	0	0
Mg.b	128	126	2	0	0	0
Total	1249	1210	34	5	0	0

According to Table 2, the CSI algorithm's delineation is faithful to the manual delineation. Most of the crown delineation results match or nearly match the manual delineation results, and only a few crowns of Eu.s and Eg.x appeared merged. However, this error is acceptable due to the dense branching characteristics of broadleaf species, which makes it difficult to completely delineate the crowns from the images manually.

3.2.2. Sample set construction and enhancement

The steps for constructing the ITS sample set in this study are illustrated in Figure 3. (1) A field survey of the study area was conducted to collect ITS samples. During the field survey, to ensure accurate sampling points, the sampling points were only selected at locations where all trees were of the same species within 30 m around the center of the point. Field sampling indicated that the main tree species were Eucalyptus urophylla S.T. Blake, Cunninghamia lanceolata (Lamb.) Hook., Pinus massoniana Lamb, Eucalyptus grandis x urophylla, Illicium verum Hook. f, Castanopsis hystrix A. DC, Michelia odora (Chun) Nooteboom & B. L. Chen, and Manglietia glauca Blume. (2) After automatically delineating the individual tree crowns in the UAV imagery of the study area using the CSI algorithm, the accurately delineated crowns were manually selected. (3) Combining the geographic location and species type of the collected samples, the results of delineating the individual tree crowns, and the results of the manual visual interpretation of remote sensing imagery, individual tree crowns that were spectrally and texturally consistent with the surrounding crowns were labeled with the tree species category. (4) Only the individual tree crowns linked to a sampling point were retained, and then the calibrated time-series imagery of the minimum bounding rectangles of these labeled crowns was output to obtain individual crown images of different tree species. (5) ITS images were categorized according to different tree species to construct the ITS sample set in the study area.

Figure 3. The steps for building the ITS sample set.

After rotating or flipping the constructed ITS sample set, it was enhanced to six times the original set.

Each class in the ITS sample set was divided at a ratio of 3:1:1 to obtain the training, validation, and test sample sets (Table 2 of the Supplemental material), to participate in the subsequent training and testing processes.

3.3. Model structure

In this study, AlexNet (Krizhevsky, Sutskever, and Hinton 2017) and DenseNet (Huang et al. 2017) were used as comparison models, and a combination of 3D CNN and LSTM was used to construct the 3DLSTM model to process multiband and multitemporal remote sensing imagery.

3.3.1. Conv3DLSTM module

Three-dimensional convolution is achieved by convolving a three-dimensional kernel into a cube formed by stacking multiple consecutive frames. With this structure, the feature maps in the convolutional layer are connected to multiple consecutive frames in the previous layer, which can capture the interband information of multiband remote sensing imagery for classification (Ji et al. 2013). The LSTM model is composed of multiple chained memory units and is effective in handling time-series information (Shi et al. 2015) (Ge et al. 2022).

The 3DLSTM module is obtained by combining the three-dimensional CNN model and the LSTM model, which can effectively utilize multiband and multitemporal data in time-series imagery simultaneously. The specific structure of the Conv3DLSTM module is shown in Figure 4, where the red arrows indicate short-term information flow. The blue arrows represent the attentive memory flow, which potentially enables the model to capture long-term relations. The cubes indicate higher-dimensional hidden and memory states. The cylinders denote higher-dimensional gates. ⨀ is the Hadamard product. ⊗ is the matrix product after reshaping matrices into appropriate 2-dimensional forms. $H_{t-1}^k$ denotes the hidden states from the previous time stamp, $C_{t-1}^k$ represents the memory states from the previous time stamp, and $M_t^{k-1}$ denotes the previous spatiotemporal memory states described earlier.

Figure 4. The structure of the Conv3DLSTM module (Wang et al. 2019b).

3.3.2. 3DLSTM model

The overall structure of the 3DLSTM model that is suitable for ITS classification is shown in Figure 5. The 3DLSTM model is composed of a Conv3DLSTM layer, six 3D convolutional layers, a fully connected layer, two dense layers, and a softmax layer. The Conv3DLSTM layer was mainly used in the Conv3DLSTM module to extract temporal information, the 3D convolutional layers were used to further extract the spatial and spectral information, the fully connected and dense layers were used for the intermediate processing of the features, and the softmax layer was used to obtain the ITS category information. In the 3DLSTM model, when time-series remote sensing imagery is input to the deep learning network, the temporal information is first extracted by the Conv3DLSTM layer and then processed by a series of 3D convolutional layers to fully extract the spatial and spectral band information. After processing the fully connected layer and dense layer, the softmax classifier was used to classify the ITS.

Figure 5. The 3DLSTM model structure.

4. Experimental settings

The study was performed using Python 3.9.7, the front-end deep learning environment was Keras 2.8.0, and the backend environment was TensorFlow 2.8.0. In this study, the ReLU function was chosen as the neuronal activation function, the loss function was categorical_ cross-entropy, and the iterative optimizer was Adam. Considering the limited number of sample sets for each tree species, the number of batches was set to 25 and the number of iterations was 500 to improve the training runs. To speed up the training, the tolerated number of times that the accuracy of the network did not improve was set to five. After five iterations without any improvement, the learning rate was reduced by 0.005. The lower limit of the learning rate was 0.5e⁻⁶, and an accuracy improvement greater than 0.001 was considered an improvement. Test results were obtained for the validation set at the end of each epoch. As the number of epochs increased, if the test error increased on the validation set or the accuracy improvement was less than 0.001 for more than 10 epochs, the network stopped training and saved the model with the highest classification accuracy on the validation set.

In the experimental process, we reasonably set the parameters based on the sample number and training accuracy of the model. For example, we need to reasonably adjust parameters such as batch size, iteration number, and learning rate. If the batch size is too large, the direction of the gradient descent will be inaccurate during the training of the model, whereas if the batch size is too small, it will result in fewer features being learned by the model each time, which will lead to low classification accuracy. Similarly, an iteration number that is too large or too small of a learning rate will result in too long a training time, whereas an iteration number that is too small or too large of a learning rate will result in insufficient training.

5. Results

5.1. Classification results

5.1.1. The training and validation accuracies of the models

The AlexNet, DenseNet, and 3DLSTM models were trained using the divided Planet or Planet-UAV ITS training sample set, and the model accuracy was verified using the validation sample set during the training process. The training accuracy (TA) and validation accuracy (VA) of the three models after multiple training iterations are presented in Table 3.

Table 3. The training and validation accuracies of the model.

Model	UAV		Planet		Planet-UAV
	TA/%	VA/%	TA/%	VA/%	TA/%	VA/%
AlexNet	93.67	93.58	98.92	98.79	99.66	99.57
DenseNet	93.76	93.50	100	100	100	100
3DLSTM	–	–	99.87	99.83	100	100

The following findings were obtained by comparing the classification accuracies of the three models using the Planet or Planet-UAV ITS sample set. There was a significant increase in the classification accuracy of ITS after the introduction of time-series information. For example, the training accuracy of the AlexNet model increased from 93.67% to 99.66%, the validation accuracy increased from 93.58% to 99.57%, the training accuracy of the DenseNet model increased from 93.76% to 100%, and the validation accuracy increased from 93.50% to 100%. The increase in model accuracy proves that time-series information contributes to improving the classification accuracy. In addition, by comparing the classification accuracy of the 3DLSTM and AlexNet models using the Planet-UAV ITS training sample set, it can be seen that the 3DLSTM model leads to higher classification accuracy.

5.1.2. The spectral mean values of tree species

To compare the separability of the different tree species, we plotted the spectral mean values of the eight species at different times in different bands, as shown in Figure 6.

Figure 6. The spectral mean values of eight tree species at different bands. (a) The spectral mean values of eight tree species in blue band. (b) The spectral mean values of eight tree species in green band. (c) The spectral mean values of eight tree species in red band. (d) The spectral mean values of eight tree species in near-infrared band.

As shown in Figure 6, there was almost no difference among the eight tree species in the blue band. In the green and red bands, slight differences were observed between the eight tree species in October 2016 and February 2017. In the near-infrared band, there were significant differences among the eight tree species during the different periods. However, Cunninghamia lanceolata (Lamb.) Hook. had overlapping spectral values with other tree species in different periods, which was similar to that of Illicium verum Hook. f, which also had no clearly distinguishable features during different periods. Similarly, Table 4 shows that Cunninghamia lanceolata (Lamb.) Hook. and Illicium verum Hook. F had a low classification accuracy.

Table 4. The classification accuracies of different models.

Model	Evaluation Indicators	Tree species
		Eu.s	Cl.h	Pm.l	Eg.x	Iv.h	Ch.a	Mo.n	Mg.b
AlexNet (UAV)	PA /%	89.47	84.26	72.28	86.71	48.96	88.89	83.33	88.33
	UA /%	94.44	78.31	72.28	90.16	64.09	81.16	67.11	82.17
	OA /%	78.08
	Kappa coefficient	0.7336
DenseNet (UAV)	PA /%	100	88.15	77.19	86.49	67.71	94.44	92.50	95.00
	UA /%	87.02	87.02	80.59	94.35	68.66	93.70	75.51	85.71
	OA /%	84.11
	Kappa coefficient	0.8078
AlexNet (Planet)	PA /%	100	80.93	88.07	88.29	72.57	75.40	95.00	100
	UA /%	89.76	78.88	89.17	98.00	75.72	70.37	86.36	88.89
	OA /%	85.40
	Kappa coefficient	0.8232
DenseNet (Planet)	PA /%	100	89.81	95.26	91.89	81.25	99.21	100.00	100
	UA /%	99.13	89.15	93.62	98.55	92.86	90.58	85.11	86.96
	OA /%	92.55
	Kappa coefficient	0.9097
3DLSTM (Planet)	PA /%	100	94.07	95.09	95.05	78.13	84.92	95.00	100
	UA /%	98.28	85.81	96.79	96.13	90.36	93.86	88.37	97.56
	OA /%	92.68
	Kappa coefficient	0.9109
AlexNet (Planet-UAV)	PA /%	100	82.78	94.21	90.09	76.39	84.13	99.17	100
	UA /%	95.00	82.78	85.78	98.52	90.16	86.18	85.00	97.56
	OA /%	88.85
	Kappa coefficient	0.8643
DenseNet (Planet-UAV)	PA /%	100	89.26	91.75	94.59	87.50	95.24	100	100
	UA /%	100	85.31	90.96	98.59	100	95.24	90.91	90.91
	OA /%	92.64
	Kappa coefficient	0.9105
3DLSTM (Planet-UAV)	PA /%	100	95.56	92.46	95.27	88.89	96.83	99.17	100
	UA /%	99.13	87.90	95.99	99.06	96.97	97.60	88.15	100
	OA /%	94.62
	Kappa coefficient	0.9346

5.1.3. The classification accuracies of different models

To further illustrate the strengths and weaknesses of the 3DLSTM model with the AlexNet and DenseNet models for different ITS classification predictions, we used the test sample set to calculate the confusion matrices of the AlexNet, DenseNet, and 3DLSTM models to obtain the producer accuracy (PA), user accuracy (UA), OA, and Kappa coefficient (shown in Table 4), as well as the true positive rate (TPR) and false positive rate (FPR) (shown in Figure 7), to compare the applicability of the different models.

Figure 7. TPR and FPR of different models.

In Figure 7, the ROC space was formed by taking the FPR as the horizontal coordinate and the TPR as the vertical coordinate. In the ROC space, points closer to the top left indicate that the model was better classified. Figure 7 shows that the best classification results were obtained using the Planet-UAV sample set and 3DLSTM model, followed by using the Planet sample set and 3DLSTM model, and subsequently using the Planet-UAV sample set and DenseNet model. The worst results were obtained using the UAV sample set and the AlexNet model.

The classification results in Table 4 indicate that the classification accuracy of both the DenseNet and 3DLSTM models exceeded 90% when only the Planet ITS sample set was used. This shows that ITS can be effectively distinguished using time-series information, providing the possibility of efficiently classifying a large range of ITS. Moreover, compared with classifying ITS using only UAV image, classifying ITS using Planet imagery exhibited higher classification accuracy. For example, the classification accuracy of the AlexNet model increased from 78.08% to 85.40%, and that of the DenseNet model increased from 84.11% to 92.55%. Thus, time-series information is more effective than texture information in distinguishing ITS.

In addition, the 3DLSTM model had a higher classification accuracy than the DenseNet model when classifying ITS using the Planet sample set, demonstrating that the 3DLSTM model can better extract the time-series information of ITS.

Compared with the other two sample sets, the accuracy was higher when using the Planet-UAV sample set for ITS classification, indicating that the combination of time-series and texture information helps classify ITS with higher accuracy. The classification accuracy of the 3DLSTM was 94.62%, which was higher than those of AlexNet (88.85%) and DenseNet (92.64%). This proves that the combination of time-series and texture information and the introduction of the 3DLSTM model can lead to more accurate ITS classification.

For the Planet-UAV sample set, 3DLSTM was used to classify four trees (Cunninghamia lanceolata (Lamb.) Hook., Eucalyptus grandis x urophylla, Castanopsis hystrix A. DC, and Manglietia glauca Blume), the producer's accuracy and user's accuracy were higher than those of the AlexNet and DenseNet models. For example, the producer accuracy in Cunninghamia lanceolata (Lamb.) Hook. increased from 82.78% (AlexNet) and 89.26% (DenseNet) to 95.56%, and the user's accuracy increased from 82.78% (AlexNet) and 85.31% (DenseNet) to 87.90%. These accuracy improvements demonstrate that the 3DLSTM model is better able to utilize the spectral, texture, and timing information of Planet-UAV imagery. In addition, Eucalyptus grandis x urophylla, Castanopsis hystrix A. DC, and Manglietia glauca Blume are broad-leaved trees that are often difficult to identify when classifying ITS. Using 3DLSTM and phenological information can improve the identification accuracy of these species, which can greatly contribute to ITS identification of broad-leaved trees in the future.

5.2. Classification map

To better understand the classification results of the three models, pure forests with eight tree species (Eucalyptus urophylla S.T. Blake, Cunninghamia lanceolata (Lamb.) Hook., Pinus massoniana Lamb, Eucalyptus grandis x urophylla, Illicium verum Hook. f, Castanopsis hystrix A. DC, Michelia odora (Chun) Nooteboom & B. L. Chen, and Manglietia glauca Blume) were selected from the study area for ITS classification (Figure 8).

Figure 8. Classification map of the study area. (a) The imagery of the study area; a-h in the imagery are pure forests of Pinus massoniana Lamb, Illicium verum Hook. f, Cunninghamia lanceolata (Lamb.) Hook., Castanopsis hystrix A. DC, Eucalyptus grandis x urophylla, Michelia odora (Chun) Nooteboom & B. L. Chen, Manglietia glauca Blume, and Eucalyptus urophylla S.T. Blake; (b1)-(i4) the classification maps of the reference, AlexNet, DenseNet, and 3DLSTM of study areas a-h, respectively.

Figure 8 indicates that the classification accuracy of the three models was higher for Michelia odora (Chun) Nooteboom & B. L. Chen, Eucalyptus urophylla S.T. Blake pure forest, and the other six pure forests, with 100% classification accuracy for Eucalyptus urophylla S.T.Blake. In addition, the classification accuracy of the 3DLSTM model for Michelia odora (Chun) Nooteboom & B. L. Chen was higher than that of the DenseNet model, and only one tree crown was classified incorrectly. AlexNet's classification accuracy in the pure Pinus massoniana Lamb forest was slightly higher than that of the other two models. DenseNet classification accuracy in Illicium verum Hook. f and Manglietia glauca Blume were slightly higher than those of the other two models, but the classification result of the 3DLSTM model was close to that of the DenseNet model. Classification accuracy of the 3DLSTM model for Cunninghamia lanceolata (Lamb.) Hook., Castanopsis hystrix A. DC, and Eucalyptus grandis x urophylla pure forests were significantly higher than those of the other two models, and the misclassification of tree species for individual tree crowns was significantly reduced. Overall, the 3DLSTM model has higher ITS classification accuracy and fewer misclassified crown species, and can more adequately classify ITS with time-series information of individual trees compared with the other two deep learning models.

6. Discussion

The experimental results showed that the 3DLSTM model and time-series imagery can be used to accurately classify ITS and achieve superior classification accuracy compared to that of a single high-spatial-resolution image and DenseNet or AlexNet models, providing the possibility of inexpensive classification of large-scale ITS. To further reduce the image cost for ITS classification, it is necessary to select images with a lower resolution, and fewer numbers. Therefore, we considered a combination of UAV image with different spatial resolutions and time-series images, and a combination of different time-series images to select lower-resolution, and fewer time-series images for ITS classification, which will provide the basis for large-scale ITS classification at a lower cost.

6.1. Superior spatial resolution of UAV imagery

Remote sensing imagery with high spatial resolution can help locate individual trees accurately, and it also contains rich information that can help classify ITS with high accuracy. However, as the spatial resolution of imagery increases, the research cost increases, and imagery with excessively high spatial resolution causes redundancy in the classification information. Finding a suitable resolution for remote-sensing imagery for ITS classification is an area that should be explored further.

We resampled the calibrated UAV image and Planet time-series imagery to 0.3, 0.4 and 0.5 m spatial resolutions to create ITS sample sets with different spatial resolutions. The 3DLSTM model was trained and validated using those ITS sample sets. The training and validation accuracies of the 3DLSTM model after several training iterations are shown in Figure 9.

Figure 9. Classification accuracy of Planet-UAV imagery at different spatial resolutions.

We compared the ITS sample sets with different spatial resolutions and the results indicated that the ITS overall classification accuracy gradually decreased with decreasing spatial resolution within 0.2 m to 0.5 m. Therefore, we should select imagery with higher spatial resolution when classifying ITS using time-series imagery to ensure the accuracy of ITS classification. The classification accuracy at a 0.3 m spatial resolution was slightly inferior to that at a 0.4 m spatial resolution, which may be due to the misalignment of individual tree crown pixels when 0.2 m was resampled to 0.3 m. However, the classification accuracies of the ITS sample sets with 0.2 m to 0.5 m spatial resolution were all above 90%, demonstrating the effectiveness of using time-series information to classify ITS. Moreover, high classification accuracy could still be achieved when using remote sensing imagery with a 0.5 m spatial resolution to classify ITS. Remote sensing imagery with a suitable spatial resolution can be chosen as the data source for ITS classification depending on the needed classification accuracy.

6.2. Superior time-series combination

The imagery of different time series contains different phenological information, which can be used to effectively classify the ITS. However, an excessive number of images can render preprocessing tedious and redundant, thereby reducing the efficiency of ITS classification. Determining how to obtain accurate classification results with fewer time-series images is another problem that requires further exploration.

The existing time-series imagery was combined in different ways, such as the Planet imagery combination, including spring-summer, spring-autumn, summer-autumn, spring-summer-autumn, and other combinations, with the specific time-series combination shown in Table 5.

Table 5. The abbreviations of different time-series combinations.

Time-series combination	Abbreviation
March and August in 2017	38
April and August in 2017	48
September in 2016, March in 2017	39
September in 2016, April in 2017	49
October in 2016, March in 2017	310
October in 2016, April in 2017	410
September in 2016, August in 2017	89
October in 2016, August in 2017	810
September in 2016, March and April in 2017	349
October in 2016, March and April in 2017	3410
September in 2016, March and August in 2017	389
October in 2016, March and August in 2017	3810
October in 2016, April and August in 2017	4810
September and October in 2016, March in 2017.	3910
September in 2016, April and August in 2017	489
September and October in 2016, April in 2017.	4910
October in 2016, February, April and August in 2017	24810
October in 2016, March, April, and August in 2017	34810
September and October in 2016, March and April in 2017	34910
October in 2016, February, March, April and August in 2017	234810
September and October in 2016, February, March, and April in 2017	234910
September and October in 2016, March, April and August in 2017	348910
September and October in 2016, February, March, April and August in 2017	2348910

According to the above time-series imagery combinations, sample sets of ITS with different time-series combinations were created separately, and the 3DLSTM model was used to classify ITS. The classification accuracy of the 3DLSTM model after iterative training is shown in Figure 10.

Figure 10. The classification accuracy with different time-series combinations.

According to Figure 10, there is a large difference in the classification accuracy of the ITS sample set with different time-series combinations. When combining two time-series images, the highest classification accuracy was obtained using 810 images (summer and autumn). During these two periods, Pinus massoniana Lamb cones grew rapidly and Illicium verum Hook. f flowers, Eucalyptus urophylla S.T. Blake flowers, and Cunninghamia lanceolata (Lamb.) Hook. seedpods turn from blue-green to yellow-brown. When three time series were combined, the highest classification accuracy was achieved using 4810 images (spring, summer, and fall seasons), with information on Pinus massoniana Lamb young cone formation and Illicium verum Hook. f young fruit formation was added to the 810 periods. When four images were combined, the highest classification accuracy was achieved using 34810 images, with information about Illicium verum Hook. f fruit maturation, Cunninghamia lanceolata (Lamb.) Hook. flowers, and Illicium verum Hook. f spreading leaves added to previous ones. When five images were combined, the highest classification accuracy was obtained using the combined image of 348910, adding information about Pinus massoniana Lamb cone ripening to the previous image. When a combination of 2348910 images was used to classify the ITS, the accuracy reached 92.68%. Overall, for the classification of the eight tree species, images 8, 10, 4, and 3 were more informative for classification, and the trees were classified more effectively when they were flowering and fruiting. In addition, using time-series imagery, the 3DLSTM model achieved 92.68% ITS classification accuracy, demonstrating that higher classification accuracy can be achieved using time-series information.

7. Conclusion

In this study, we used the 3DLSTM model, which is more suitable for extracting information from multiband and multitemporal imagery, and 3-m Planet time-series imagery to classify ITS. We also employed the 3DLSTM model to investigate the effects of different combinations of spatial resolution, and time series on ITS classification at the Guangxi Gaofeng Forest Farm.

The experimental results showed that after delineating individual tree crowns using high-precision UAV image, ITS could be effectively classified using the 3DLSTM model and 3m-resolution time-series imagery, which outperformed the two deep-learning networks, providing the possibility of large-scale ITS classification at a low cost.

In addition, the 3DLSTM model and time-series imagery could better classify broad-leaved species that were previously difficult to identify. Furthermore, adding texture information to time-series imagery can help classify ITS. Among the different temporal image combinations, the images of trees at flowering and fruiting contained the most significant classification information.

In future work, we will further explore ITS classification accuracy using other time-series imagery with low or medium spatial resolution to reduce the cost of large-scale ITS classification.

Author contributions

C.C. designed and completed the experiment; L.J. provided comments on the method; and L.J., H.L., Y.T., and F.C. revised and provided feedback on the manuscript. B.T. provided UAV images and field survey data from Gaofeng Forest Farm. All authors have read and agreed to the published version of the manuscript.

Acknowledgments

We thank Mr. Guang Ouyang, Mr. Haoming Wan, and Mr. Xianfei Guo, who graduated from this research group, for their contribution to this research.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the National Key R&D Program of China under Grant [number 2021YFB3900503]; the National Natural Science Foundation of China under Grant [number 41972308]; the Jiangxi Provincial Technology Innovation Guidance Program (National Science and Technology Award Reserve Project Cultivation Program) under Grant [number 20212AEI91006]; and the Second Tibetan Plateau Scientific Expedition and Research under Grant [number 2019QZKK0806].