Comparison of high-resolution NAIP and unmanned aerial vehicle (UAV) imagery for natural vegetation communities classification using machine learning approaches

ABSTRACT

To map and manage forest vegetation including wetland communities, remote sensing technology has been shown to be a valid and widely employed technology. In this paper, two ecologically different study areas were evaluated using free and widely available high-resolution multispectral National Agriculture Imagery Program (NAIP) and ultra-high-resolution multispectral unmanned aerial vehicle (UAV) imagery located in the Upper Great Lakes Laurentian Mixed Forest. Three different machine learning algorithms, random forest (RF), support vector machine (SVM), and averaged neural network (avNNet), were evaluated to classify complex natural habitat communities as defined by the Michigan Natural Features Inventory. Accurate training sets were developed using both spectral enhancement and transformation techniques, field collected data, soil data, texture, spectral indices, and expert knowledge. The utility of the various ancillary datasets significantly improved classification results. Using the RF classifier, overall accuracies (OA) between 83.8% and 87.7% with kappa (k) values between 0.79 and 0.85 for the NAIP imagery and between 87.3% and 93.7% OA with k values between 0.83 and 0.92 for the UAV dataset were achieved. Based on the results, we concluded RF to be a robust choice for classifying complex forest vegetation including surrounding wetland communities.

KEYWORDS:

• Natural community habitats
• NAIP
• UAV
• image classification
• machine learning

1. Introduction

A key goal of forest management is to maintain and preserve natural biodiversity and protect the pristine landscape (Lindenmayer and Franklin 2002) by using efficient, affordable science-based practices. In order to maintain a balance between biological diversity and societal needs, it is important to develop management plans capable of achieving these goals (Bettinger et al. 2016). Within the natural resource community, there are numerous techniques used for management, monitoring and conservation planning. Field surveys and aerial photography are traditional techniques for obtaining information about forest conditions, identifying tree species or evaluating habitats. However, they are expensive, time-consuming, and have limitations (Ruiliang and Landry 2012). Aerial photography and satellite imagery interpretations provide areal coverage but are constrained by temporal, spectral, and spatial resolutions. Free or low-cost multispectral imagery commonly has spatial resolutions between 10 and 30 m. This information is employed for vegetation monitoring, predicting species abundance in association with environmental changes, and mapping habitats for species distribution modeling (Taylor et al. 2000; Buchanan et al. 2005; Prasad, Iverson, and Liaw 2006; Bradter et al. 2011; Monahan et al. 2022).

Within the remote sensing community, forest land use/cover classification using satellite imagery, piloted aircraft imagery and/or unmanned aerial vehicle (UAV) data is well documented (Homer et al. 2004; Hansen et al. 2010; Hayes, Miller, and Murphy 2014; Yang et al. 2018; Maxwell et al. 2019; Bhatt et al. 2022b). Land use/cover classification schemes typically use well defined, non-overlapping categories (Anderson 1976; Vogelmann et al. 2001; Bailey 2004, 2009) to classify forests, wetlands, grasslands, etc. These frameworks are uniform in the characterization of land use/cover and are based on canopy cover as detected by airborne sensors and include human activities.

By contrast, natural community habitats have a much broader definition and are based not only on the canopy but also understory vegetation, soils, and landform (Bradter et al. 2011; Cohen et al. 2014). “Protecting, managing, and restoring these communities is critical to biodiversity conservation, since native organisms are best adapted, to environmental and biotic forces with they have survived and evolved over the millennia” (Cohen et al. 2014). Natural community descriptions are derived from “physiography, hydrology, soils, natural processes and vegetation and do not include modern anthropogenic disturbances” (Cohen et al. 2014). The classification is organized hierarchically by ecological class, group, and type. Each community differs in its physical environment and species composition and the same plant species (canopy cover) often occurs in more than one community (Cohen et al. 2014; Cohen 2020). This increases class complexity and classification challenges due to a lack of distinct identifying spectral signatures (Jensen 2015; Lillesand et al. 2015). Delineating and mapping these communities require high spatial resolution imagery for improved feature delineation, image enhancements, and transformations as well as the incorporation of biogeophysical data.

Along with high spatial resolution imagery, classification accuracy depends on selecting the correct classification algorithm. Use of machine learning (ML) classification approaches has exponentially increased in the last decade (Schulz et al. 2018) and applied to a variety of environmental and natural resource applications (Mahesh and Mather 2003; Mountrakis et al. 2011; Rodriguez-Galiano et al. 2012b; Hayes, Miller, and Murphy 2014; Maxwell, Warner, and Fang 2018; Yang et al. 2018). The algorithms (MLAs) use a nonparametric approach to model and classify data and do not require normally distributed data. Numerous land use/cover classification studies highlight the advantages of using MLAs such as random forest (RF) and support vector machine (SVM) (Gunn 1998; Huang et al. 2002; Hayes, Miller, and Murphy 2014). AvNNet (Burton 1993) has been used in studies for predicting soil organic carbon, groundwater quality index, and land use/cover classifications (Taghizadeh-Mehrjardi et al. 2020; Kavhu et al. 2021; Ahn et al. 2022). MLAs were utilized in the classification of the 2001 National Land Cover Database (NLCD) [16]. They have also been used with NAIP imagery for accurate land cover classification (Kulkarni and Lowe 2016; Maxwell et al. 2019).

Useful ancillary data are equally important given the depth and breadth of the natural community class definitions. These datasets help overcome spectral limitations of the imagery and provide information beyond the bird’s eye view of the canopy. Researchers have used various environmental and geomorphological variables to improve classification results (Anderson 1976; Corcoran et al. 2013; Hayes, Miller, and Murphy 2014; Juel et al. 2015; Berhane et al. 2018; Kumar et al. 2020). Therefore, it is important to understand the contribution each ancillary dataset provides to the classification. Widely available feature selection methods evaluate ancillary datasets’ importance (Guyon and Elisseeff 2003) by reducing data complexity (Hughes 1968) and improving computational times (Maxwell, Warner, and Fang 2018). The robustness of the approach used in this study expanded on a previous study done by the authors (Bhatt et al. 2022b), which included the use of NAIP coupled with biogeophysical variables and widely used machine learning algorithms like RF and SVM.

Researchers have used NAIP and UAV datasets for delineating and mapping land cover, identifying tree species, delineating wetlands, habitat mapping, and invasive species mapping (Maxwell et al. 2017; Bhatt 2018, 2022; Hogland et al. 2018; Bhatt et al. 2022a; Monahan et al. 2022, 2022 2022). Each has advantages and disadvantages in terms of spatial, spectral, and temporal resolution. To date, a direct comparison between the two sets of imagery for natural habitat community classification has not been completed. The key objective of this research is to compare and contrast the utility of ultra-high spatial resolution UAV imagery versus high spatial resolution NAIP imagery to delineate and map complex natural community habitats.

2. Materials

2.1. Study areas

Upper Midwest forests are classified as Laurentian Mixed Forest (LMF) which is made up of complex geomorphology, climate, soils, fauna, and vegetation due to the extensive glaciation which occurred over thousands of years. Natural community boundaries may be sharply defined or change gradually. The area has a climatic tension zone, and sites along the Great Lakes shoreline support vegetation with northern and southern affinities which pose classification challenges. Dividing this landscape into natural communities provides guidance to better describe, understand, and restore the native community diversity (Cohen et al. 2014; Cohen 2020).

The study areas are within the Hiawatha National Forest and fall under the International Union for Conservation of Nature (IUCN) category IV and contain diverse upland and lowland ecosystems, including extensive pristine coastal forests and wetlands. Two study sites, Point aux Chenes (PAC) Bay (Figure 1) and Carp River Mouth (CRM) (Figure 2), were selected as there are numerous natural community habitats which are unique in vegetation, soil, and landform within relatively small geographic areas (Cohen et al. 2014). Several of the communities (Interdunal Wetlands, Open Dunes and Wooded Dune and Swale Complex) are considered imperiled due to rarity and/or vulnerability (Cohen et al. 2014). The PAC site encompasses 420 ha (1,038 ac) and 790 ha (1,952 ac) for CRM. The study areas are located within glacial lake and outwash plain landforms, respectively (Jerome 2006). Current threats to these areas include unauthorized off-road vehicle use, poorly designed or degraded road and stream crossing structures which create physical barriers to hydrologic function, roads that parallel coastlines with inadequate drainage structures, and the presence and/or expansion of non-native invasive species.

Figure 1. Pointe aux Chenes Bay study area. The shoreline is adjacent to Lake Michigan. NAIP imagery was collected in August 2018. UAV imagery was collected in August 2019.

Figure 2. Carp River Mouth study area. The shoreline is adjacent to Lake Huron. NAIP imagery was acquired September 2018. UAV imagery was collected in August 2019.

2.1.1. Datasets and software

High-resolution multispectral NAIP and ultra-high-resolution UAV imagery were used for the study. NAIP imagery has four bands (Blue (420–492 nm), Green (533–587 nm), Red (604–664 nm), and Near-Infrared (683–920 nm)) (USDA 2022) and was acquired 4,877 m (16,000 feet) above ground level (AGL) with a Leica ADS100 airborne digital sensor. The imagery has 8-bit radiometric resolution with 0.6 m spatial resolution. Imagery tiles dated 11 August 2018 and 6 September 2018 were downloaded from USGS Earth Explorer for PAC and CRM, respectively.

UAV data were collected in August 2019 using a fixed-wing Trimble U×5-AG aircraft. The U×5-AG has a 1 m wingspan with 2.5 kg weight and is capable of flying up to 45 minutes with a cruise speed of 80 km/h. Imagery with 80% overlap was acquired using a five-band (Blue (475 ± 20 nm), Green (560 ± 20 nm), Red (668 ± 10 nm), Red Edge (717 ± 10 nm), and Near-Infrared (840 ± 40 nm)) MicaSense camera mounted onboard. Flying height was between 104 and 134 m (341 ft–440 ft) with a 7 cm spatial resolution for the PAC and 9 cm for the CRM. The spatial resolution varied due to the flying height and differences in terrain geometry. Days with optimum sunlight conditions and minimum clouds were selected for flying to minimize illumination and shadowing inconsistencies. Using the onboard high-accuracy Global Navigation Satellite System (GNSS) positioning data, the UAV images were processed and mosaicked with Agisoft Metashape 1.5.3 software using the standard workflow procedure provided by the USFS UAV office (Sloan 2017).

ERDAS IMAGINE (Hexagon Geospatial) was used to generate Principal Component Analysis (PCA), spectral indices, and Gray-Level Co-Occurrence Matrix (GLCM) texture layers for both sets of imagery. Random training points were generated using ArcPro software. Machine learning algorithms were implemented using the “caret” (Kuhn et al. 2020) package within R (Team 2013) programming language.

3. Methods

Spectral variability and similarity within and between the vegetative components of the natural community habitats created classification challenges and was documented by Bhatt et al. (2022a). However, the high spatial resolution NAIP and UAV imagery combined with ML permitted utilization of the variability (Maxwell et al. 2017). All NAIP and UAV spectral bands were utilized for training set generation. An integrated classification approach incorporating ancillary data (image transformation and enhancement techniques), field data, and expert knowledge was developed (Whittaker 1962; Adam, Mutanga, and Rugege 2009; Corcoran et al. 2013; Cohen et al. 2014; Lane et al. 2014; Berhane et al. 2018; Congalton and Green 2019). Accurately delineated training area polygons were critical for optimal performance of Machine Learning Algorithms (MLAs) (Maxwell, Warner, and Fang 2018).

3.1. Image transformation techniques

PCA is commonly used for various classification applications and is one of the most widely used transformation techniques and generates uncorrelated components (Dunteman 1989; Jensen 2015). It has been used by natural resource managers to delineate vegetation, map change detection, and observe vegetation distribution (Almeida and Souza Filho 2004; Munyati 2004; Lasaponara 2006; Dronova et al. 2015). By contrast, Independent Component Analysis (ICA) uses higher-order statistics and considers each component to be non-Gaussian (Shah et al. 2002). The transformation highlights minute details in the imagery even when the feature occupies a small area (Hyvärinen and Oja 2000). However, it has been used minimally to map vegetation and for land use/cover classification (Shah et al. 2007a, 2007b; Fangfang and Xiao 2011). Components from both transformations were visually assessed for edge detection within and between the natural habitat communities to generate valid training sets. With the multispectral data, the UAV imagery was preprocessed and mosaicked using Metashape and the NAIP imagery was preprocessed by the contractor after acquisition.

3.2. Topographic data

Land surface characteristics for the study areas are greatly influenced by extensive glaciation and influence elevation and soil characteristics such as drainage and pH which influence natural community development. A high-resolution (1 m) LiDAR Digital Elevation Model (DEM) was used to identify topographic details of the natural community habitats for the NAIP imagery. Additionally, an ultra-high-resolution DEM was generated in Metashape from the point cloud data to use with the UAV imagery.

3.3. Texture

Similar spectral signatures occur between the natural habitat communities and increase the difficulty of accurate separation during training set development and classification. However, the communities do display various types of texture traditionally used in manual interpretation. Different texture statistics can detect unique information and spatial patterns for features which are hard to separate using only spectral information (Haralick, Shanmugam, and Dinstein 1973; Maillard 2003; Lane et al. 2014; Hall-Beyer 2017). In the past, texture-based variables have been incorporated by researchers into species detection, for fine-scale wetland classification, and in land use/cover classification (Rodriguez-Galiano et al. 2012a; Feng et al. 2015; Berhane et al. 2018; Franklin and Ahmed 2018; Tassi and Vizzari 2020). GLCM texture measures were calculated from the first and second PCA components and created two uncorrelated texture datasets. For both the NAIP and the UAV imagery, first (55.38% to 67.72%) and second (27.02% to 38.32%) principal components contributed the highest to explaining the data variability (Dunteman 1989). Four GLCM texture measures (contrast, entropy, standard deviation, dissimilarity) were calculated. Contrast measures the local variations present in the image, entropy measures the randomness within the data, standard deviation looks at its frequency of occurrence with reference and neighboring pixel values, and dissimilarity measures the differences in elements of the GLCM from each other (Haralick, Shanmugam, and Dinstein 1973; Hall-Beyer 2017). Data were generated with a 32-bit grayscale level and two Euclidean geometry offsets (2, 2 and 2, −2). Window sizes of 3 × 3, 5 × 5, 7 × 7, and 9 × 9 were evaluated.

3.4. Spectral indices

Spectral indices have been used extensively to map and monitor vegetation (Bannari et al. 1995; Berhane et al. 2018; Bhatt et al. 2022b). The normalized vegetation index (NDVI) is widely used by researchers to look at vegetation growth, phenology extraction, and landcover classification (Rouse et al. 1974; Tucker 1979; Shuang et al. 2021) and was employed in this study to classify the natural communities. NDVI calculation takes ratio between the red (R) and near-infrared (NIR), while the two modified water indices based on the WorldView water index (WV-WI) were developed by Wolf (Wolf 2012). The water index for the NAIP imagery (WINAIP) using its blue and near-IR bands, and a water index for the UAV imagery (WIUAV) using the blue and near-IR bands of the Micasense camera created customized indices.

3.5. Evaluation of ancillary datasets

When classifying natural community habitats, inputting multispectral imagery alone was not adequate for accurate data classification. With manual interpretations, ancillary data such as soil maps are traditionally used for improved boundary delineation and vegetation classification. It made sense to provide the ML classifiers with this type of information as well. DEMs, GLCM-textures (contrast, entropy, standard deviation, dissimilarity), and spectral indices (NDVI, WINAIP, WIUAV) were calculated. The next step was to understand each ancillary dataset’s contribution to classification improvement as using all of them does not guarantee the best result. Ancillary input dataset selection approaches have been used in many remote sensing applications (i.e. data mining, natural language processing, bioinformatics, image processing, crop classification, crop yield prediction, and mineral mapping) (Guyon and Elisseeff 2003; Hoque et al. 2014; Zhong et al. 2019; Kumar et al. 2020; Momm, ElKadiri, and Porter 2020; Zheng et al. 2021), but have not been extensively used in natural habitat community classification (Bhatt et al. 2022a). Input data selection (also known as variable or feature selection) approaches are fast, cost-effective, and provide insight into the contribution of each ancillary dataset (Guyon and Elisseeff 2003). For this study, joint mutual information maximization (JMIM) (Bennasar et al. 2015), a filter-based method, was used.

3.6. Training set development

Training polygons, from which the ML training points are selected, were manually drawn with careful consideration given to vegetation community species, soil drainage classes and pH, elevation and landform, information highlighted in the PCA and ICA components, ground truth data, and expert knowledge of the areas. Eight natural community classes and three non-community classes were identified (Table 1).

Table 1. Natural community habitats and associated vegetation components. Communities 9–11 were developed for land uses not natural community habitats.

	Natural community	Common trees, plants, shrubs
1	Emergent Marsh	Sedges, musk grasses, common reed, common waterweed, coontail, waterlily (Cohen et al. 2014)
2	Submergent Marsh	Musk grasses, common waterweed, coontail, waterlily, pond-lilies (Cohen et al. 2014)
3	Great Lakes Marsh	Broad-leaved cat-tail, waterlily, pond-lilies, coontail, duckweed, sedges, reed grass, tag alder, green ash, paper birch (Cohen et al. 2014)
4	Northern Shrub Thicket	Sedges, grasses, ferns, tag alder, bog birch, dogwoods, winterberry, willows, black ash, tamarack, black spruce, white pine, northern white-cedar (Cohen et al. 2014)
5	Interdunal Wetlands	Sedges, rush, willow, tamarack, jack pine, northern white-cedar (Cohen et al. 2014)
6	Rich Conifer Swamp	Northern white-cedar, white pine, tamarack, spruce, red maple, black ash, tag alder, balsam fir, birch (Cohen et al. 2014)
7	Wooded Dune & Swale Complex	Pines, oaks, red maple, balsam fir, sedges, rush, ferns, tag alder, willows, black spruce, black ash, tamarack, northern white-cedar, aspen (Cohen et al. 2014)
8	Sand & Gravel Beach	Marram grass, Baltic rush, willow, pitcher’s thistle, sea rocket (Cohen et al. 2014)
9	Open Water	Lake Michigan, Lake Huron, inland lakes, rivers and ponds
10	Open Land	Open land areas including agriculture and quarries
11	Impervious Surfaces	Roads, houses, cars, other man-made structures

Accurate training sets are critically important for the classification accuracies (Maxwell, Warner, and Fang 2018). Multiple training sets created with randomly selected training points were developed across the study sites to capture spectral and spatial variability. More training polygons (812 for UAV vs 136 for NAIP) were needed to classify the UAV imagery because the higher spatial resolution provided greater detail and it was important to incorporate the spectral details of different natural community classes. Boundaries of the training set polygons (Figures 3 and 4) are verified from extensive field assessment and from existing stand maps. This approach was selected based on its successful use in previous research (Bhatt et al. 2022b).

Figure 3. Natural habitat communities training and testing polygons for CRM study area.

Figure 4. Natural habitat communities training and testing polygons for CRM study area.

3.7. Image classification

RF, SVM, and avNNet are widely used classifiers within the remote sensing community for land use/cover, agriculture crop classifications, and groundwater quality assessment (Gong et al. 1997; Huang et al. 2002; Clark, Roberts, and Clark 2005; Bandos, Bruzzone, and Camps-Valls 2009; Bradter et al. 2011; Hayes, Miller, and Murphy 2014; Berhane et al. 2018; Mahdianpari et al. 2018; Gómez et al. 2019; Ahn et al. 2022). RF was developed by Brieman (1984) as an ensemble classifier which utilizes nonparametric classification and regression tree (CART) (Breiman and Cutler 2007) rules to predict. It can work with spatially large and complex datasets which are highly correlated and can work robustly without having optimization parameters (Breiman 1999; Maxwell, Warner, and Fang 2018). SVM, a supervised machine learning algorithm which identifies an optimal hyperplane separating two classes in a feature space and was developed by Vapnik (2013). Having a higher dimensional feature space to project the non-linear and noisy data distributions can help overcome the overfitting issue and classify the real-world data better (Boser et al. 1992; Maxwell, Warner, and Fang 2018). Averaged neural network (avNNet) functions by applying averaging technique to the neural network (Ripley 2007). avNNet can be used for both regression and classification by applying the modified ordinary differential equations to the neural network (Ripley 2007). For classifying the data, model scores are averaged first and then applied it to the predicted class (Burton 1993).

In a recent study completed by Bhatt et al. (2022a) with NAIP imagery, RF was shown to be a better classifier for natural community habitats compared to SVM. Along with RF and SVM, another ML algorithm, averaged neural network (avNNet) from the caret (Kuhn et al. 2020) package was tested. Within the training polygons, 75% of the randomly selected training points were used to develop the training sets and the remaining 25% were reserved for accuracy assessment. To avoid any overfitting issues, a 10-fold cross validation was applied to the data (Kohavi 1995). All three classifiers were ran with “center” and “scale” pre-processing parameters to standardize the ancillary datasets (Kuhn 2015; Kuhn et al. 2020). Classifications were executed using the “caret” package (Kuhn et al. 2020) in the “R” programming language (Team 2013). Results for the three classifiers were compared using Overall Accuracy (OA) and kappa coefficient (k) (Congalton and Green 2019). Individual communities were evaluated employing User’s Accuracy (UA), Producer’s Accuracy (PA), and F1 Scores (Congalton and Green 2019). Between 10 and 15 ground truth observations were made for each natural habitat community class during field visits (August 2019) to each study site.

3.8. Accuracy assessment and post-classification refinement

Accuracy assessment was completed using the reserved test points plus independently collected field points. These are referred to as validation points. Evaluations were completed by comparing classification values against the validation points (Congalton and Green 2019). Using the resulting accuracy assessment matrices, UA and PA values were calculated for each habitat class.

“Salt and pepper” effects (Lillesand et al. 2015) were smoothed to create a more easily interpreted final classification map. A majority filter using a 7 × 7 moving window was run based on previous research (Munyati 2004).

4. Results

Pixel-based image classifications were run using RF, SVM, and avNNet with RF producing the best classifications for the various combinations of imagery and variables. Figures 5 and 6 show the classification results for both sets of imagery at each study site. The figures show zoomed-in classification snippets highlighting various mapped details. Visual assessment of the classifications showed that the UAV classifications delineated finer boundary detail for each community. This is due to the finer spatial resolution when compared to the NAIP derived boundaries. The NAIP imagery also presented a more generalized natural community habitat map with less “salt-and-pepper” artifacts (Lillesand et al. 2015).

Figure 5. Selected areas of the PAC classification delineated by the RF classification for NAIP and UAV imageries highlighting detail differences for the natural community classes.

Figure 6. Selected areas of the CRM study site delineated by the RF classification for NAIP and UAV imageries highlighting detail differences for the natural community classes.

The PAC NAIP-based classification achieved OAs between 86.28% and 88.33% using all input variables with RF. When the highest contributing variables based on the JMIM scores were used, the OAs ranged between 85.28% and 87.74%. Reducing the input variables from 15 to 9 did not significantly affect the OAs and kappa (k) (Table 2). However, accuracies decreased significantly using only the 4 NAIP bands and ranged from 72.49% to 77.15% (Table 2). This illustrates the important contribution made by the variables. The final classification had an OA of 87.74% and a k of 0.85. Similar results occurred with the PAC UAV imagery. Using the 16 available inputs with the UAV imagery, the OA was slightly lower (Table 3) compared to inputting the best 10 JMIM selected inputs. Once again, using just the five UAV reflectance bands decreased OA between 5% and 8% with all three classifiers (Table 3). The final UAV imagery classification using RF achieved an OA of 93.74% and 0.92 k. Overall, the UAV classifications provided better end products than the NAIP classifications by a 6% increase in OA (0.7 k) using RF (Table 3).

Table 2. Accuracy assessments of the NAIP derived classifications for PAC and CRM.

Ancillary Data/Variables (NAIP)	Classifier	PAC		CRM
		OA (%)	k	OA (%)	k
R, G, B, NIR	RF	77.15	0.72	67.26	0.57
	SVM	75.33	0.69	65.50	0.55
	avNNet	72.49	0.65	65.11	0.53
NAIP + DEM + GLCM-Texture (7×7) - PC 1, 2 (Contrast, Entropy, Standard Deviation, Dissimilarity) + NDVI + WINAIP	RF	88.33	0.86	84.80	0.80
	SVM	87.04	0.84	83.57	0.79
	avNNet	86.28	0.83	80.95	0.75
Final Classification - NAIP + DEM + GLCM-Texture (7×7) - PC 1, 2 (Contrast) + NDVI + WINAIP	RF	87.74	0.85	83.85	0.79
	SVM	86.13	0.83	83.06	0.78
	avNNet	85.28	0.81	79.60	0.74

Table 3. Accuracy assessments of the UAV derived classifications for PAC and CRM.

Ancillary data/Variables (UAV)	Classifier	PAC		CRM
		OA (%)	k	OA (%)	k
R, G, B, Red Edge, NIR	RF	87.06	0.83	75.51	0.67
	SVM	86	0.82	75.34	0.66
	avNNet	78.80	0.73	70.04	0.58
UAV (R, G, B, RE, NIR) + DEM + GLCM-Texture (7×7) - PC 1, 2 (Contrast, Entropy, Standard Deviation, Dissimilarity) + NDVI + WIUAV	RF	93.02	0.91	86.77	0.82
	SVM	89.13	0.87	84.46	0.80
	avNNet	85.14	0.81	80.52	0.73
UAV (R, G, B, NIR) + DEM + GLCM-Texture (7×7) - PC 1, 2 (Contrast) + NDVI + WIUAV (without red edge)	RF	93.39	0.92	87.31	0.83
	SVM	90.56	0.88	86.14	0.81
	avNNet	84.22	0.80	80.78	0.74
Final Classification - UAV (R, G, B, RE, NIR) + DEM + GLCM-Texture (7×7) - PC 1, 2 (Contrast) + NDVI + WIUAV	RF	93.74	0.92	87.31	0.83
	SVM	91.07	0.89	86.16	0.81
	avNNet	85.09	0.81	81.13	0.74

The NAIP and UAV classifications for the CRM study site achieved higher accuracies with RF compared to SVM and avNNet. Final NAIP-CRM classification OA was 83.85% with 0.79 k (Table 2). The UAV-CRM OA and k were 87.31% and 0.83, respectively (Table 3). Both classifications performed best using JMIM scores to select input variables compared to using all available data (Tables 2 and 3). Using only the NAIP spectral bands decreased the OA by 16% (Table 2), and with the UAV the accuracy decreased 11.8% (Table 3). These results are similar to those seen with the PAC study site and the work completed by Bhatt et al. (2022b). They indicate the robustness of the classification approach across the diverse natural communities.

Kappa values were categorized into three groups, good (k < 0.80) (Landis and Koch 1977; Altman 1990), strong (k 0.80–0.90), and almost perfect (k > 0.90) (McHugh 2012). With this study, the classifications showed strong to almost perfect relationships between “truth” and the classified results (Tables 2 and 3). Along with the OA and k, F1 scores were compared for each natural community class. Within the remote sensing community, F1 scores are widely considered when the dataset does not have a balanced accuracy. It is essentially the harmonic mean of precision (UA) and recall (PA) (Etten et al. 2018; Zhang et al. 2020). Both precision (UA) and recall (PA) should be 1 for a good classification, though this seldom occurs with complex real-world datasets. Figures 7 and 8 show the F1 scores by community class for the RF classifier. For PAC NAIP classification, lower F1 scores (Figure 7) are observed with Great Lakes Marsh (0.71), Interdunal Wetland (0.77), and SGB (0.74). These lower scores for Great Lakes Marsh and Interdunal Wetland can be attributed to the similarities in the vegetation with other natural habitat communities like Wooded Dune and Swale Complex and Emergent Marsh. With the PAC UAV classification, lower accuracies (Figure 7) were observed with Great Lakes Marsh (0.66) due to spectral similarities in natural habitat communities with Emergent Marsh and Wooded Dune and Swale Complex.

Figure 7. RF classifier F1 scores for the natural communities between NAIP and UAV imagery for Pointe aux Chenes.

Figure 8. RF classifier F1 scores for the natural communities between NAIP and UAV imagery for Carp River Mouth.

Lowest F1 Scores for CRM classification were observed with Rich Conifer Swamp natural community class for both NAIP (0.52) and UAV (0.47) imagery (Figure 8). Major confusion for Rich Conifer Swamp was observed with Wooded Dune and Swale Complex (Appendix - Tables 1 and 2) Emergent Marsh natural community showed lower F1 scores (Figure 8) for NAIP imagery, whereas for UAV it was high. Major confusion for Emergent Marsh was observed with Great Lakes Marsh, Rich Conifer Swamp and Wooded Dune and Swale Complex (Appendix Table 2). The higher Emergent Marsh F1 score for UAV could be attributed to the more precise training data collection and higher spatial resolution. Similarly, Great Lakes Marsh class had a much better F1 score for NAIP than UAV. Majority of confusion for Great Lakes Marsh in UAV classification was with Northern Shrub Thicket and Open Water classes (Appendix Table 2). Highest F1 scores for both study areas were seen with Open Water, Open Land, and Impervious Surface, due to their uniform surface reflectance (Figures 7 and 8). Overall, the F1 scores for both the study areas were reasonably well given the complexity and spectral similarities of the natural habitat communities.

PAC NAIP classification (Figure 5) was more generalized and contributed to Wooded Dune & Swale Complex and Emergent Marsh over prediction when compared to field observations. The UAV classification delineated the natural communities well except for Emergent Marsh, Interdunal Wetlands, and Great Lakes Marsh. This confusion is due to the same vegetation components and water being found in all of them (Table 1); hence spectral (Appendix Figure A1 and B1) and textural similarities between them. These communities are smaller in size and intermixed with more commonly occurring, larger area, communities and therefore have fewer validation points. Fewer points mean misclassifications have a greater impact on UA and PA and were lower compared to the other classes (natural communities) (Appendix Table A1).

The confusion matrix for NAIP CRM classification (Appendix Table A2) shows both lower UAs and PAs for Emergent Marsh, Northern Shrub Thicket, and Rich Conifer Swamp. The matrix shows confusion between Emergent Marsh and Northern Shrub Thicket. Evaluation of the training sets showed large standard deviations and these training sets were replaced with ones with less variability (smaller standard deviations). The misclassification of the Rich Conifer Swamp is primarily with Wooded Dune & Swale Complex. This confusion is the result of a high percentage of the same conifer species in both communities including northern white cedar, tamarack, white pine, and tag alder (Table 1). Landform is also a strong indicator for Wooded Dune & Swale Complex (Figure 9). However, if the swales are widely spaced between the dunes, the texture changes and contributes to the error particularly with the coarser NAIP spatial resolution (Figure 9). The UAV classification shows lower UAs and PAs for Great Lakes Marsh, Northern Shrub Thicket, and Rich Conifer Swamp. Again, the poor classification of Great Lakes Marsh and Northern Shrub Thicket is due to high variability in the training sets, and sets were retaken.

Figure 9. Landforms influence on natural communities for the PAC and CRM study sites are highlighted when draped over a multi directional oblique weighted (MDOW) hillshade. Classifications derived using Random Forest.

Ancillary data sets were evaluated using JMIM feature selection. JMIM scores range between 0 and 2 regardless of the measurement units of the input variables. This allows direct comparison between the variables in ascertaining the unique contribution each input makes to the classification. Plots of the JMIM scores (Figure 10) show that the input variables (ancillary data) maintain the same pattern of importance for both sets of imagery across the study sites. The same results were seen with work completed by Bhatt et al. (2022a). These included DEMs, GLCM-textures (contrast, entropy, standard deviation, dissimilarity), NDVIs, and modified water indexes specific to the NAIP and UAV imagery. The water index for each set of imagery differed due to the NAIP having four spectral bands (B, G, R, near-IR) while the UAV has five bands (B, G, R, red edge, and near-IR). JMIM scores were calculated for all of the spectral bands as well (Figure 10). At both study sites, the most important input features with the NAIP imagery were all NAIP spectral bands, NDVI, WINAIP, DEM and contrast texture (Figure 10). For the UAV data, the five Micasense spectral bands, NDVI, WIUAV, DEM, and contrast texture (Figure 10) showed high importance. It is important to remember that JMIM-based selection values do not guarantee a more accurate classification but provide quantitative guidance to input variable selection.

Figure 10. Ancillary dataset (variable) importance scores using JMIM feature selection method. B-blue, C1-contrast texture (PC1, 7×7 moving window), C2-contrast texture (PC2, 7×7), DEM-digital elevation model, Dissim1-Dissimilarity texture (PC1, 7×7), Dissim2-Dissimilarity texture (PC2, 7×7), Ent1-entropy texture (PC1, 7×7), Ent2-entropy texture (PC2, 7×7), G-green, NDVI-normalized difference vegetation index, NIR-near-infrared, R-red, SD1-standard deviation texture (PC1, 7×7), SD2-standard deviation texture (PC2, 7×7), WINAIP- NAIP modified water index, WIUAV-UAVmodified water index.

Both NDVI and WINAIP, two of the highest JMIM scores, provided information to accurately delineate the community habitat classes. Classes such as Wooded Dune and Swale Complex, Rich Conifer Swamp, and Northern Shrub Thicket showed higher values with NDVI compared to the remainder of the habitat community classes. Incorporation of the DEM, NDVI, WINAIP, contrast textures 1, and 2 increased the OA of the natural community classifications an average of 13.59% for NAIP and 18.22% for the UAV when compared to only using the spectral bands of each set of imagery (Tables 2 and 3). Overall, the UAV-based classification outperformed the NAIP classification due to its high-spatial resolution and greater texture details.

5. Discussion

Most studies classifying land use/cover are for specific resource management purposes and categorize the imagery into narrowly defined, non-overlapping classes. However, classifying imagery into well defined, robust natural community habitats provides a holistic approach to resource management and is more representative of field condition variability. Until recently, there were no natural community habitat classification studies of the complex Laurentian Mixed Forest in the Upper Great Lakes. Inventorying, monitoring, and preserving these pristine habitats, particularly along coastlines, are increasingly important given the impacts of climate change. Field-based monitoring alone is not able to complete these tasks in a timely, cost-effective manner. The approach presented in this study is applicable to other areas across United States and Canada (White and Host 2008; Brown et al. 2013). New Hampshire (Sperduto and Nichols 2004), North Carolina (Schafale and Weakley 1990), Wisconsin (Curtis 1959; Noss 1987), Massachusetts (Whitlatch 1977; Kearsley 1999), Indiana (Jackson 1979; Homoya et al. 1984), Minnesota (Hanson and Hargrave 1996; Aaseng et al. 2011; Wilson and Ek 2017), and (Inventory, Florida Natural Areas 1990) are a few of the states that have developed Natural Communities classification schemes. While the hierarchy of the classification varies from state to state, the foundation of the schemes is the same: group recurring assemblages of flora and fauna found in particular physical environments into a descriptive classification scheme. These maps provide information essential to conservation planning and protecting biodiversity by communicating a story to resource managers, landowners, land-use planners, and scientists (Cohen 2020).

Which imagery is best for natural community classification is unclear. Variations in reflectance values between the UAV and NAIP for the same areas are shown in Figures 11 and 12. This variation is due, in part, to the different bandwidths for the RGB bands. Imagery pair A in Figure 11 shows a gray textured area (red arrow) in the UAV imagery which is Princess Pine (Dendrolycopodium obscurum) an endangered club moss and indicates a small, but important area of Wooded Dune Swale Complex. It is not visible in the NAIP; rather, the entire area is classified as Interdunal Wetland and does not provide the spatial and spectral detail required to manage and protect this at-risk species. Pair B (Figure 11) highlights underwater seepage and drainage patterns from springs not detectable on the UAV imagery. The presence of a small Great Lakes Marsh trapped by a sand bar is visible on the UAV imagery (Pair C, Figure 11) but appears as Sand and Gravel Beach on the NAIP. These small “trapped” Great Lakes Marsh Areas support bird’s eye primrose (Primula mistassinica) and blue vervain (Verbena hastata). Both species are important food sources for native bees and need to be monitored frequently given the imperiled status of many bee species. On the UAV imagery, the bright magenta areas found in Pair D (Figure 11) are Phragmites and are not visible in the NAIP. Common reed (Phragmites australis ssp. americanus) is a native species. However, Phragmites australis ssp. australis is an invasive species and is considered to be problematic in North America. It invades marsh meadows and cattail zones and reduces the number of bird species found in these habitats (Robichaud and Rooney 2017). In Figure 12, Pair A shows individual stems of downed coarse woody debris which provides food and habitat for a wide range of organisms, slows water flow in river and streams, and recycles nutrients trapped in the wood. Pair B (Figure 12) shows dead tree in both images, but the UAV imagery permits counting of the snags and species identification. Snags are critical landscape features for numerous wildlife species.

Figure 11. Spectral reflectance differences between UAV and NAIP imagery for PAC due to differences in the red, green and blue bandwidths.

Figure 12. Spectral reflectance differences between UAV and NAIP imagery for CRM due to differences in the red, green and blue bandwidths.

For areal coverage, NAIP is more comprehensive as it is acquired by aircraft flying at a constant speed and altitude. It undergoes rigorous radiometric and geometric corrections, and mosaics are easy to create. By contrast, UAV imagery has variations in lighting conditions due to acquisition times often spaced throughout the day, and the vehicle is more susceptible to pitch, yaw, and roll distortions due to its lighter weight. Radiometric and geometric conditions are applied to the UAV imagery, but the user must be willing to accept quality variation across the mosaicked imagery. A study completed by 2022b), recommends an image overlap of 80% to help overcome geometric variability and produce acceptable quality orthoimages. The finer spatial resolution of the UAV imagery maps textural changes very well, which are important to the accurate delineation of spectrally similar natural communities such as Northern Shrub Thicket, Rich Conifer Swamp and Wood Dune & Swale Complex (Table A2).

Performing radiometric and geometric corrections on the UAV imagery is time-consuming. Current processing software has limitations in photogrammetric robustness and ease of use. The physical size of the UAV imagery is also an important consideration, and adequate computing resources and storage space are prerequisites. Processing times need to be considered. With the two study areas, it took 88 hours to preprocess the UAV imagery and generate the final orthomosaics for the two study areas.

For NAIP, temporal resolution is not ideal as it is collected every 5 years especially for catastrophic events such as fires and flooding as well as phenological timed events. UAV imagery offers advantages to these types of studies as the platform can be airborne in a short-time frame and is ideal for data collection over small geographic areas where the vehicle can be kept in the operator’s line of sight.

Accurate training sets and informative ancillary datasets are equally important to the success of mapping natural communities. Spectral signatures (Appendix Figure A1 and B1) alone do not provide enough clear separation to accurately delineate the natural communities. This study used PCA, ICA, soil data, landform, field data, and expert image interpretation to outline training set polygons for each natural habitat community. Training points for the MLAs were extracted from within the polygons (Figures 3 and 4). It is important to note that the difference between the number of training sets for NAIP and UAV is attributed to the spatial resolution. Larger training polygons within the ultra-fine UAV imagery contain too much variation (large standard deviations) and result in noisy training sets and lead to misclassification of the natural communities.

The natural habitat communities at both study sites exhibited significant amounts of local variation. The finer spatial resolution of the UAV imagery emphasizes the high degree of texture created within and between the natural communities and contributed to the accurate delineation of the natural community boundaries. Each community exhibits distinctive patterns and shapes such as the well-defined ridge and valley complex associated with the Wooded Dune and Swale Complex. Contrast texture, using a 7 × 7 window size, was the most informative variable when evaluated against entropy, standard deviation, and dissimilarity using JMIM feature selection (Figure 10). Low contrast values were observed with smooth texture classes such as Open Water, Open Land, and Impervious Surface. Highly textured community classes (i.e. Wooded Dune and Swale Complex, Rich Conifer Swamp, Northern Shrub Thicket) have a rougher texture represented by higher contrast values as illustrated in Figure 10. Due to the small area of some of the natural communities and having multiple vegetation species within each community, it is not possible to run large texture windows (i.e. 21 × 21, 33 × 33). The detailed texture image degrades dramatically as window size increases (Figure A2 – Appendix) and it would add more confusion to the classification (Cohen et al. 2014; Hall-Beyer 2017).

This discussion presents advantages and disadvantages to using each set of imagery and recommending using one to the exclusion of the other would be wrong. How will the final map be used is the question(s) which should dictate to mix of UAV imagery and NAIP. RF is recommended as the classifier of choice when working with ecologically complex natural community habitats. SVM and avNNet are less efficient compared to RF and always produce lower accuracies in this study. It may be argued that tuning the RF parameters may further improve the classifications. However, parameter tuning is time-consuming and not cost-effective given the acceptable accuracies achieved for the natural community delineation and mapping.

6. Conclusions

Natural community mapping with high and ultra-high spatial resolution imagery along with informative ancillary data is challenging and critically important for the forest management and policy. The data sets are physically large and have high computational requirements along with lengthy processing times. The development of good training sets requires time and detailed knowledge of the study area. Widely used and robust machine learning algorithms like RF overcome these intricacies. However, acceptable results are achieved in this study not only in identifying the correct natural community label but also mapping accurate and detailed boundaries between the communities.

This research has many unexplored and unanswered questions to be further investigated. In the future studies, efficacy of the red edge band and derived indices needs exploring and compared with traditional indices like NDVI. Red edge indices have been used in the past with Sentinel-2A, Rapid-Eye, WorldView-2, and UAV imagery to observe vegetation phenology, spatial variability of crop growth, leaf area index, and burn severity assessments (Hill 2013; Shang et al. 2015; Fernández-Manso et al. 2016; Zhu et al. 2017; Guo et al. 2021) but not in mapping natural habitat communities. Investigation of training data size and quality should also be explored in reference to two different imagery datasets.

Due to the increasing need for sustainable planning and management practices, mapping natural communities at finer scales is important and requires robust workflows such as one provided in this study and use of advanced remote sensing techniques. Future work should involve testing the robustness of this workflow for larger areas and other regions. Lastly, advanced methods like deep learning should be considered and compared with machine learning algorithms.

Description of authors’ responsibilities

PB collected and analyzed the data, designed the study approach, performed the experiments, did field work and wrote the original manuscript draft; AM wrote the original project proposal, did field work, advised on methodology improvements, revised and edited the manuscript.

Acknowledgments

We would like to thank the College of Forest Resources and Environmental Science, Michigan Technological University for their support. We also thank Ian Anderson (Chief Product Owner) of Hexagon Geospatial for his crucial help at the beginning of this project, Jim Ozenberger of the Hiawatha National Forest for assistance with field work and Emily Clegg of The Nature Conservancy for providing technical support. We would like to acknowledge Dr. Curtis Edson and Ben Miller who helped in UAV flight planning, and data collection.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The data that support the findings of this study are available from the corresponding author, PB, upon reasonable request.

Additional information

Funding

This research was funded by the US Forest Service, Hiawatha National Forest (Grant Number 17-PA-11091000-023), The Nature Conservancy (Grant Number R45984) and the College of Forest Resources and Environmental Science.

Appendix A

Table A1. NAIP and UAV classification accuracy assessment matrices for PAC. EM – Emergent Marsh, SM – Submergent Marsh, GLM – Great Lakes Marsh, IW – Interdunal Wetlands, WDSC – Wooded Dune & Swale Complex, SGB – Sand & Gravel Beach, OW – Open Water, IS – Impervious Surface.

RF (NAIP - PAC)	EM	SM	GLM	IW	WDSC	SGB	OW	IS	UA (%)	PA (%)
EM	518	11	0	13	21	20	1	0	88.7	89.0
SM	11	329	0	0	0	0	0	0	96.7	96.7
GLM	0	0	101	0	4	2	11	0	85.6	60.1
IW	2	0	0	285	29	87	0	1	70.5	85.0
WDSC	46	0	44	17	941	20	0	0	88.1	94.1
SGB	5	0	1	20	5	223	0	0	87.8	63.3
OW	0	0	22	0	0	0	292	0	93.0	96.0
IS	0	0	0	0	0	0	0	124	100.0	99.2

OA = 87.74%, k = 0.85

Table

RF (UAV – PAC)	EM	SM	GLM	IW	WDSC	SGB	OW	IS	UA (%)	PA (%)
EM	467	1	11	25	2	1	4	24	87.2	86.1
SM	1	223	0	3	0	0	3	0	96.9	96.9
GLM	0	0	28	1	4	0	1	1	80	53.8
IW	52	1	2	465	7	2	3	9	85.9	89.7
WDSC	1	0	4	4	673	0	0	1	98.5	97.5
SGB	3	0	0	4	0	117	0	2	92.8	95.9
OW	5	5	3	5	1	0	1137	0	98.3	98.7
IS	13	0	4	11	3	2	3	366	91.0	90.8

OA = 93.74%, k = 0.92

Table A2. NAIP and UAV classification accuracy assessment matrices for the CRM. EM – Emergent Marsh, GLM – Great Lakes Marsh, NST – Northern Shrub Thicket, RCS – Rich Conifer Swamp, WDSC – Wooded Dune & Swale Complex, OW – Open Water, OL Open Land, IS – Impervious Surface.

RF (NAIP - CRM)	EM	GLM	NST	RCS	WDSC	OW	OL	IS	UA(%)	PA(%)
EM	175	23	48	3	23	0	6	0	62.9	53.3
GLM	29	1192	39	0	2	3	1	0	94.1	89.5
NST	31	83	404	9	157	0	0	1	58.9	61.4
RCS	3	0	4	95	42	0	0	0	65.9	42.6
WDSC	87	26	162	108	1494	0	2	0	79.5	86.7
OW	0	7	0	0	0	897	0	0	99.2	99.6
OL	3	0	1	8	4	0	283	0	94.6	96.9
IS	0	0	0	0	0	0	0	119	100.0	99.1

OA = 83.58%, k = 0.79

Table

RF (UAV – CRM)	EM	GLM	NST	RCS	WDSC	OW	OL	IS	UA(%)	PA(%)
EM	206	13	9	0	0	2	3	1	88.0	85.5
GLM	21	544	122	1	9	42	19	0	71.8	65.8
NST	8	164	875	10	121	0	36	1	72.0	74.5
RCS	0	0	1	44	22	0	0	0	65.6	35.5
WDSC	0	3	121	69	871	0	1	0	81.8	85.1
OW	4	78	5	0	0	3255	0	2	97.3	98.6
OL	1	23	41	0	0	1	615	1	90.1	91.2
IS	1	2	1	0	0	1	0	119	95.9	95.9

OA = 87.31%, k = 0.83

Figure A1. Spectral reflectance signatures for PAC study area shown using NAIP (8-bit-unsigned) and UAV (16-bit-unsigned) imagery DN Values.

Figure B1. Spectral reflectance signatures for CRM study area shown using NAIP (8-bit-unsigned) and UAV (16-bit-unsigned) imagery DN Values.

Figure A2. GLCM texture (Contrast) differences in details caused by the window sizes.