Pothole detection in the woods: a deep learning approach for forest road surface monitoring with dashcams

ABSTRACT

Sustainable forest management systems require operational measures to preserve the functional design of forest roads. Frequent road data collection and analysis are essential to support target-oriented and efficient maintenance planning and operations. This study demonstrates an automated solution for monitoring forest road surface deterioration using consumer-grade optical sensors. A YOLOv5 model with StrongSORT tracking was adapted and trained to detect and track potholes in the videos captured by vehicle-mounted cameras. For model training, datasets recorded in diverse geographical regions under different weather conditions were used. The model shows a detection and tracking performance of up to a precision and recall level of 0.79 and 0.58, respectively, with 0.70 mean average precision at an intersection over union (IoU) of at least 0.5. We applied the trained model to a forest road in southern Norway, recorded with a Global Navigation Satellite System (GNSS)−fitted dashcam. GNSS-delivered geographical coordinates at 10 Hz rate were used to geolocate the detected potholes. The geolocation performance over this exemple road stretch of 1 km exhibited a root mean square deviation of about 9.7 m compared to OpenStreetMap. Finally, an exemple road deterioration map was compiled, which can be used for scheduling road maintenance operations.

KEYWORDS:

• Deterioration map
• preventative maintenance planning
• YOLOv5
• object detection and tracking
• global navigation satellite systems (GNSS)

Introduction

Forest roads are a central part of the forest infrastructure that allows for access and management of the forest (Boston 2016). These roads are commonly designed as haul roads to relocate machinery and truck timber during the harvesting phase, but also support other management activities such as stand establishment and tending, monitoring, or surveying tasks that require access for personnel and equipment. From a societal perspective, functional forest roads are important for recreational purposes, but also grant access during emergencies such as rescue or firefighting (Waga et al. 2020). To safeguard the unrestricted functionality of a forest road according to its category (e.g. permanent haul road, granting all-year access with logger trucks), periodic and preventive maintenance is essential to ensure management of a sustainable forest road network (Ezzati et al. 2021).

Maintenance requirements in terms of intervals and magnitude can differ significantly according to road standard and building material, traffic intensity, and e.g. occurring extreme weather events, with climate change−induced differing in frequency and patterns (Dodson 2021). Road maintenance costs are considerable, but negligence in maintenance eventually leads to consequences such as road failure or restricting a road’s functionality and increasing the risk of negative environmental impacts (Dietz et al. [1984] 2011; Erber et al. 2021). Consequently, forest roads need to be frequently monitored to detect wear and maintenance status. Surface deteriorations in the form of potholes are often the most obvious signs of road wear since these also affect driving comfort and travel speed. If not attended to in time, potholes can lead to greater defects and are also a good indicator of lack of crossfall and other road drainage issues (New Zealand forest road engineering manual 2020). Potholes retain water, which can then penetrate the aggregates, causing strength loss of the road base layer. A common consequence of such seep from potholes is bearing capacity loss and frost heave, which can cause complete road failure (Saarenketo and Aho 2005). To avoid cost-intensive road repairs, comprehensive and frequently updated information about the road surface to determine the kind and urgency of required actions is essential for a sustainable forest road management system.

Forest road monitoring is currently widely practiced by manual inspection, which is time consuming and expensive, and the results are often subjective and dependent on the surveyor (Susnjar et al. 2020). Attempts to formalize these procedures through specified assessment protocols and derived indices, such as the Forest Road Pavement Condition Index (Heidari et al. 2022), did not gain wide acceptance, mainly due to the continuing high labor input. Consequently, maintenance operations are often scheduled in an inefficient manner. Technological progress in sensing technologies has made its way to forestry, and offers numerous potential ways to efficiently gather and process road data to be incorporated in decision support systems (Talbot et al. 2017; Picchio et al. 2019). Implementing sensor technologies into operational traffic of, for example, timber trucks, offers time and cost-efficient alternatives to manual road surveys and support target−oriented maintenance operations.

Among highway authorities, dedicated survey systems of multiple sensors and high resolution have already been established to detect various wear-induced road anomalies of concern to traffic safety and road durability (van der Horst et al. 2019). Such approaches can be integrated into mobile platforms, such as dedicated surveying vehicles of commercial operators (e.g. Roadscanners Oy, Finland), or even can be conducted with stationary build-in sensors to create “smart roads” (Barriera et al. 2020). Yet, forest roads are low-volume roads, requiring less cost-intensive systems, with lower resolution often being sufficient. Early attempts to use a rather simple sensor platform, combined with a maintenance planning tool, date back to the early 2000s with the Canadian Opti-Grade system making use of acceleration sensors to determine road deterioration based on surface roughness (Brown et al. 2003). With acceleration and Global Navigation Satellite System (GNSS) sensors available on modern smartphones, this approach has been widely transferred to monitor rural roads through end-user devices with tailored applications (e.g. roadroid.com), including first trials of transferring these techniques into a forestry setting (Susnjar et al. 2020).

Although smartphones are economic, bias due to vehicle calibration, driving habits, and phone positioning, as well as general limitations of accelerator sensors, allow a road condition assessment at only very coarse resolution (Sattar et al. 2018). For instance, it is not possible to determine if the vibrations originate from a pothole or are caused by uprising material from the subgrade such as larger stones. But this is relevant information for target-oriented maintenance planning. On the other hand, high resolution data as, for example, that captured with mobile laser scanner systems used among highway authorities, are too costly and sophisticated for low traffic volume roads (Cao et al. 2020). Despite some first promising pilot approaches to downsize laser scanning techniques to forestry road applications (Ferenčík et al. 2019), this remains a technology associated with higher efforts. Thus, optical systems and image-based object detection and recognition algorithms can be a suitable alternative for forestry purposes, with sufficient information provision at moderate levels.

Maintenance monitoring for potholes on paved public roads with camera solutions has received considerable attention (Roberts et al. 2020; Cao et al. 2020) since it is seen as a good compromise between cost-efficiency and data resolution compared to other sensors (Kim et al. 2022). The ability to integrate such solutions into smartphones allows a comprehensive and extensive road survey through daily traffic by, for example, postal services or public vehicles, constantly updating and increasing databases, as is already widely implemented among Japanese municipalities (Maeda et al. 2018; Arya et al. 2021).

Bounding box detectors such as the popular you-only-look-once (YOLO: Redmon et al. 2016) are often the essential base of such monitoring systems, and are also increasingly used within forestry computer-vision applications (Puliti et al. 2022; Puliti and Astrup 2022). The use of convolutional neural networks (CNNs), or deep learning, is motivated by the high quality of the output especially in computer vision tasks and by the ease of deployment in real-world scenarios thanks to the real-time inference capabilities. In addition, repositories, such as Ultralytics’ YOLOv5 (Jocher et al. 2022), provide technological access, allowing a broad community to train and deploy custom models efficiently while obtaining state-of-the-art results.

The range of YOLOv5 applications covers diverse domains, such as autonomous vehicles (Benjumea et al. 2021), surveillance systems (Ali et al. 2022), and medical imaging (Mohiyuddin et al. 2022), showcasing its versatility in addressing various computer vision challenges. YOLOv5 as an open-source, fast, and efficient one-stage detector has also made its way into forest road applications. A first attempt for an automized visual inspection system for forest roads was done by Starke and Geiger (2022), who applied a YOLOv5 computer vision-based model to detect forest road surface deterioration with consumer-grade optical sensors. However, this low-resolution approach based on smartphone and tablet images was able to detect only waterlogged sections on the road. Heidari et al. (2022) also presented a method for identifying the pavement damage through the processing of smartphone images using the YOLO algorithm.

Bounding box detectors are useful for frame-by-frame detection; however, when dealing with videos one must consider the sequential order of the frames and the need to associate the detected bounding boxes to separate instances across frames. Object tracking techniques allow for tagging the bounding box instances across frames and, consequently, counting the number of unique objects and mapping them in space. Tracking-by-detection methods have traditionally been a popular choice (Bochinski et al. 2017) in the domain of online multiobject tracking (MOT); that is, algorithms allowing the real-time tracking of multiple instances. These tracking methods first leverage an object tracker to locate objects in single frames followed by the association of these detected objects across frames using geometrical, motion, and appearance information. Recent implementations of the popular Simple Online and Realtime Tracking method (SORT; Bewley et al. 2016), have been readapted to complement the objects’ geometrical information (i.e. size and dimensions of the objects) with an appearance model, a deep learning model useful for reidentification of instances across frames (StrongSORT; Du et al. 2022).

In this study, we harnessed the power of the introduced deep learning−based object detection and tracking tools to process the video recordings of a consumer-grade vehicle dashcam for detecting and geolocating potholes as indicators of forest road surface deterioration. Moreover, our developed approach endeavors to generate a deterioration map, which can serve as a valuable resource for effectively scheduling road maintenance operations. In addition, the presented method can be used to generate input variables for modeling road deterioration processes as, for example, conducted through logistic regression analysis and artificial neural networks (Heidari et al. 2018).

Materials and methods

Although not geographically restricted in its application, the developed road-monitoring approach focused on Norwegian conditions for standard secondary low-volume forest roads of class 3, according to the Norwegian standard for agricultural roads (Lanbruksdirektorat, 2016). These roads follow a geometry to permit safe and efficient driving of haul trucks and must be able to be trafficked with loads year-round with limitations only during periods of heavy rainfall or freeze and thaw cycles in spring.

Video data

Road footage used for this study primarily relied on videos captured on various forest roads in southern Norway (Viken county), using the popular GNSS-equipped 622GW dashcam (NEXTBASE, UK), hereby referred to as “dashcam data” (see Table 1 and Figure 1). These videos were recorded with the camera positioned in the upper center of the windshield, capturing the oncoming road while driving at a moderate speed with a focus on 20 km/h. For recording, the camera was set at a resolution of 1080P (1920×1080 pixels) and a recording speed of 30 frames per second (fps), with engaged image stabilization and a polarization filter on the lens. The videos were captured under varying weather conditions and during different times of the day with the aim of covering a broad range of real-world scenarios.

Figure 1. Examples of different data sources used for model training. The dashcam data was further split into validation and test sets.

Table 1. Summary of the different data sources used to train the pothole detector with corresponding number of images and potholes.

	Modeling		Test
Dataset name	Images	Potholes	Images	Potholes
Dashcam	2779	6676	1726	3788
Tablet	376	561
Nienaber	3888	9428
Total	7043	16,665	1726	3788

To increase the variability and size of the dataset, we augmented the dashcam data with video frames captured for a different purpose on Norwegian forest roads in the same region using a tablet, hereby referred to as “tablet data,” as well as an already annotated dataset from South Africa in an urban setting (Nienaber et al. 2015, 2015).

Data annotation

The video frames from Norwegian forest roads captured with the dashcam and tablet (Table 1) were manually annotated by different teams of annotators. The annotation consisted of drawing bounding boxes around each of the visible potholes. The annotations were mostly performed at 1 fps to increase the number of videos that could be annotated within the given time frame.

For the test data (Table 1), the annotation also included the instance unique identifier allowing us to track the potholes across sequential frames and, consequently, to evaluate the object tracking algorithm. To allow for smooth tracking of the potholes, the test videos were annotated with a higher sampling rate (15–30 fps) compared to the modeling dashcam data and, thus, resulted in a substantially larger number of annotated potholes. Figure 2 demonstrates some of the annotated potholes.

Figure 2. Exemplary pothole conditions represented on the test road: (a) light roadway depression (dry conditions) in the wearing course, (b) deep pothole with dispersed aggregates of the base layer, and (c) water-filled potholes.

Test data

The test data consisted of two videos captured over the same road stretch (approx. 300 m) on a lowland timber haul road in southern Norway (Våler Kommune/Viken county), in a typical spruce (Picea abies)−dominated Nordic coniferous forest. Such a road stretch was selected as it was representative of various forest stand conditions from dense to partly open, and covered different appearances of potholes from being shallow, in their early development, to deep and established, already affecting the base layer (Figure 2). In addition, to account for various weather conditions, the road was filmed twice. The first recording was conducted on a bright and sunny day, and the video was characterized by solar reflections on the windshield and dried out potholes. The second recording took place after a rainy night and was characterized by well-visible and water-filled potholes, as well as by low-light conditions due to overcast sky.

Dashcam GNSS data

The deployed dashcam allowed for integrating geolocation information to the video recordings. The geolocation information was received at a fixed rate of 10 Hz and was in the standard format of NMEA-0183 (National Marine Electronics Association). The main NMEA message used in the dashcam for delivering the geolocation information was the “GPGGA” message known as Global Positioning System (GPS) fix data. The message included different pieces of information from which we extracted and used the measurement time in UTC, latitude, longitude, number of satellites used for the calculation of the position, and horizontal dilution of precision (HDOP) as an indicator of the predicted overall quality of the position in the horizontal domain. Since in the forest environment the dashcam’s GNSS receiver can frequently lose the satellite signals and deliver erroneous positions, a data screening was required to exclude any possible outliers from the processing.

Methodological pipeline

The overall AI pipeline developed by this study (Figure 3) involves different processes that can be grouped into the following main steps:

Figure 3. The developed pipeline for monitoring forest road surface deterioration using a GNSS-augmented dashcam. The depicted processing steps were carried out using dedicated python scripts developed in this study. The flowchart was created in inkscape software.

1. Pothole detector training: a YOLOv5 pothole detector was trained using our modeling data.

2. Pothole tracking: the trained detector was used to detect potholes in the test videos and the detected potholes were tracked through frames and aggregated into single instances.

3. Pothole mapping: the tracked potholes were assigned geographic coordinates and the potholes were dinned into 20 -m road sections.

Pothole detector training and validation

A YOLOv5 model was trained using the annotated modeling data. For the training, 10% of the annotated frames from the modeling dashcam data was used for validation and selection of the best model. Given that this currently represents a server solution (i.e. not for edge computing), we opted for a large model (i.e. YOLOv5x) with an image size of 1280, thus maximizing the performance.

We trained the model on a Linux machine with a single Tesla V100 with 8GB of memory. Based on these specifications, the largest batch size was two.

The resulting detector was validated against the validation and the test dataset using common object detection metrics such as the precision (P), recall (R), and mean average precision at intersection over union (IoU) of 0.5 ([email protected]). To assess the model’s performance under varying atmospheric conditions present during the capture of the test videos, we evaluated the two videos separately in addition to reporting the overall test data performance.

Pothole tracker implementation and validation

In this study we utilized the StrongSORT OSNet (Broström 2022) tracker, given its direct integration with YOLOv5 models. The repository includes a main function to run several tracking algorithms given an input video and model weights for a pretrained YOLOv5 model. The output of the tracking consists of a MOT compliant text file (MOT Challenge 2023) containing the columns listed in Table 2.

Table 2. The standard file format for multiobject tracking (MOT) methods’ output.

Column	Header title	Description
1	frame	The frame number (1-based)
2	id	A unique identification number which is assigned for each object instance (1-based)
3	bb_left	Left edge coordinate of the object’s bounding box (pixels)
4	bb_top	Top edge coordinate of the object’s bounding box (pixels)
5	bb_width	The width of the object’s bounding box (pixels)
6	bb_height	The height of the object’s bounding box (pixels)
7	conf	Detection confidence (ranging from 0 to 1)
8	x	x-coordinate (ignored for the 2D case)
9	y	y-coordinate (ignored for the 2D case)
10	z	z-coordinate (ignored for the 2D case)

We opted for the default StrongSORT (Du et al. 2022) with the OSNet (Zhou et al. 2019) as the reidentification model (Bochinski et al. 2017). The performance of the object tracking can be summarized by Higher Order Tracking Accuracy (HOTA) metrics (Luiten et al. 2021). HOTA is proposed to address difficulties of evaluating multiobject tracking (MOT) methods by unifying three metrics that measure the detection, localization, and association performances based on IoU. The IoU score, also known as the Jaccard index (J), varies between zero and one and is calculated as the intersection of the detection and annotation areas divided by the union of the two areas (Eq. 1(1) $\mathit{Io}{U}_{\mathit{A},\mathit{D}}=\mathit{J}\left(\mathit{A},\mathit{D}\right)=\frac{\left|\mathit{A}\cap D\right|}{\left|\mathit{A}\cup D\right|}$(1) ):

(1) $\mathit{Io}{U}_{\mathit{A},\mathit{D}}=\mathit{J}\left(\mathit{A},\mathit{D}\right)=\frac{\left|\mathit{A}\cap D\right|}{\left|\mathit{A}\cup D\right|}$(1)

where A and D, respectively, refer to the areas of annotated and detected objects. The overall success rate of a method regarding the spatial alignments of prediction against ground truths are captured by localization accuracy ($\mathit{\phi }$) (Eq. 2), which is the average IoU score of all successfully detected potholes:

(2) $\mathit{\phi }=\frac{1}{{\mathit{N}}_{\mathit{TP}}}{\sum }_{\mathit{k}=1}^{{\mathit{N}}_{\mathit{TP}}}{J}_k$(2)

with $N_{\mathit{TP}}$ being the number of true positive (TP) cases (i.e. the successful detections) identified with a minimum IoU of 50% ($\mathit{i}.\mathit{e}.{\phi }_0=0.5$). Such a detection is then considered as a common object in the detection and ground truth sets. Consequently, the detection accuracy (${\alpha }_D$) (Eq. 3) is the ratio of successful detections counted to the total number of objects in the union of the two sets:

(3) ${\alpha }_D\left({\phi }_0\right)=\frac{{\mathit{N}}_{\mathit{TP}}}{N_{\mathit{TP}}+N_{\mathit{FN}}+N_{\mathit{FP}}}\mathit{\hspace{0.17em}}\mathrm{\forall \phi }\ge {\phi }_0$(3)

where FN indicates false negative (i.e. when an existing pothole is not detected by the model), and FP indicates false positive, which refers to the case when the model mistakenly reports a detection. Finally, association accuracy for an object tracker determines how well the tracker links detections over time into the same identities (IDs). The overall association accuracy (${\alpha }_A$) over all successful detections ($N_{\mathit{TP}}$) is defined as (Eq. 4):

(4) ${\alpha }_A\left({\phi }_0\right)=\frac{1}{{\mathit{N}}_{\mathit{TP}}}{\sum }_{\mathit{i}=1}^{{\mathit{N}}_{\mathit{TP}}}\frac{{\mathit{N}}_{\mathit{TPA}}}{N_{\mathit{TPA}}+N_{\mathit{FNA}}+N_{\mathit{FPA}}}\mathit{\hspace{0.17em}}\mathrm{\forall }\mathit{\phi }\ge {\phi }_0$(4)

with the letter A in the subscripts referring to the association. The readers are referred to Luiten et al. (2021) for a more detailed description of the HOTA metrics.

Geolocating potholes

The dashcam used in this study has a built-in GNSS module that can track satellite signals from two different GNSS constellations. Although this feature significantly increases the number of observations used for the coordinate calculation, the forest environment can still leave its degrading signature on the position accuracy. Therefore, an outlier detection method was applied to the geolocation information that was extracted from the dashcam videos. In the first step, gross errors (i.e. coordinates or time stamps falling outside the spatiotemporal scope of the validation dataset) were eliminated. Such errors have been found in the GNSS module’s output mainly during periods when the number of satellites has significantly dropped. In the second step, a sliding window of 10 observations accounting for 1 second worth of observations was applied to latitude, longitude, and time values to calculate moving average and standard deviation of each parameter. If the difference of the variable with respect to the corresponding moving average exceeded the threshold of three standard deviations, the coordinates and time of that epoch were flagged as an outlier and were excluded from the processing.

Besides gross errors and outliers, the Standard Positioning Service (SPS) of GNSS that is generally implemented and used by low-cost receivers can have a significant positioning error. Therefore, the dashcam trajectory coordinates were orthogonally projected to the nearest forest road retrieved from OSM (OpenStreetMap 2023) data for the mapping purpose. The projected coordinates (at a fixed rate of 10 Hz) were then used in a linear interpolation process to calculate the geolocation of video frames at the rate of 30 fps. Each pothole instance was usually tracked within several frames and was tagged with a unique identifier. Therefore, the nearest available coordinates (i.e. coordinates of the latest frame in which the pothole appeared) were used as the pothole’s geolocation.

Classification of road deterioration

For road maintenance purposes, a favorable representation of the georeferenced potholes can be derived in the form of a pothole density map. To this end, along-road distance of each pothole instance with respect to a reference point on the road was calculated. Then, the potholes were grouped and counted based on a road segmentation scheme with a predetermined length. From a road manager’s perspective, it is of interest to classify road units into urgency of required attendance to ensure the maintenance of their functional class by scheduling required work accordingly. Thus, we classified the road units into four categories as represented in Table 3.

Table 3. Exemplary classification of forest road surface deterioration based on pothole density.

Class	Potholes density	Operational need	Color code
Category I	Low	The road requires no maintenance work	Green
Category II	Moderate	The road should be considered for preventive, ongoing maintenance work (e.g. grading of road surface)	Yellow
Category III	High	The road is due for periodic maintenance (e.g. resurfacing)	Orange
Category IV	Very high	The road is in urgent need of resurfacing and limitations for the design traffic needs to be currently accounted for	Red

In this study, we used a segmentation length of 20 m and classified segments with up to two potholes as Category I, three to five potholes as Category II, six to nine potholes as Category III, and 10 potholes or more as Category IV. This classification is limited to the pothole count and does not fully picture the overall road condition, nor does it consider the different magnitudes of potholes. It also does not follow any official categorization of forest road maintenance standards, but it is a first approach to evaluate the road condition. We defined these categories purely to convey the concept of a maintenance classification system based on a pothole count, which allows high flexibility to adapt according to different road classes and local standards by decreasing or increasing the pothole number thresholds.

Results

Pothole detector validation

The training of the pothole detector was interrupted after 299 epochs, following 100 epochs without model improvement (see Figure 4). Based on the internal validation data, the best model was found for epoch 199 corresponding to a validation precision (P), recall (R), and mean average precision at IoU of 0.5 ([email protected]) of 0.62, 0.53, and 0.55, respectively.

Figure 4. Summary of the training metrics development over the epochs. The black vertical line represents the 199th epoch. The validation data correspond to the subset of the modeling data.

Pothole tracking validation

An evaluation of pothole tracking performance using HOTA metrics is reported in Table 4. The HOTA scores confirm a significantly different detection performance for the two validation cases. For the case of wet and overcast weather with water-filled potholes, the detection was approximately boosted by a factor of ~ 2.5 compared to the video recorded on a sunny day. In contrast, the metrics reporting on the association performance did not highlight any prominent supremacy of the wet and overcast weather condition, suggesting that the association algorithm is not sensitive to the weather difference. However, the overall HOTA score was clearly affected by the different detection efficiency of the two cases.

Table 4. Performance assessment report for the pothole detection approach used in this study based on the Higher Order Tracking Accuracy (HOTA) metrics as a tool for evaluating multiobject tracking (MOT) models. All the entries are in percentages.

Condition	HOTA	Localization accuracy	Detection accuracy	Detection recall	Detection precision	Association accuracy	Association recall	Association precision
Sunny day	18.60	71.21	13.51	17.82	31.06	25.75	46.30	32.57
After rain	31.39	77.23	34.33	41.54	56.79	29.02	55.86	34.54
Combined	29.76	76.77	30.98	38.03	53.71	28.92	55.48	34.57

Table 5 summarizes an important feature for overall performance assessment, which is the number of detected and mapped pothole instances compared to the ground truths (i.e. the manually annotated potholes). The table highlights that an overcast weather condition after some rainfall can significantly boost the performance of the method.

Table 5. Number of pothole instances identified by model prediction and manual annotation under two weather conditions. On the sunny day, the model correctly detected 30.2% (13 out of 43) of the ground truth potholes compared to 43.2% (35 out of 81) on the overcast day.

Sunny day		Model prediction (19 detections)		Overcast (after rain)		Model prediction (44 detections)
		Pothole (Positive)	Background (Negative)			Pothole (Positive)	Background (Negative)
Reference (43 potholes)	Pothole (Positive)	13	30	Reference (81 potholes)	Pothole (Positive)	35	46
	Background (Negative)	6			Background (Negative)	9

Pothole mapping performance

An assessment of the quality of geolocation information delivered by the dashcam’s GNSS module is summarized in Figure 5. Panel (a) of the figure shows the overlaid graphs of the original coordinates received from the GNSS module, and the projected coordinates to the corresponding forest road. The trajectory shows a good agreement with the road; however, there is a varying deviation from the road that is depicted in Figure 5(b). The deviation over the illustrated road stretch can reach to about 33 m with a root mean square (RMS) value of about 13.7 m over the road stretch, combining the uncertainties of the GNSS coordinates and the OSM road data. It should be noted that the OSM data over the selected road stretch show an RMS deviation of 1.7 m with respect to the coordinates from a high-precision geodetic GNSS receiver in another experiment. The parameters plotted on Figure 5(c) can predict the expected overall quality of the delivered coordinates.

Figure 5. (a) The original GNSS coordinates (after removing outliers) overlaid by projected points to the road, (b) the deviation of original coordinates from the projected points, and (c) horizontal dilution of precision (HDOP) and satellite count graph on separate y-axes.

In Figure 6, the plots in left panel display the number of potholes per video frame, with the horizontal axis indicating the road length relative to the first frame. Meanwhile, the right panels in Figure 6 depict road deterioration maps based on the four categories described in Table 3. These plots were generated using manually annotated potholes (ground truth data), and the detection results for two test days: a sunny day and an overcast day following rainfall.

Figure 6. Panels on the left: the pothole detection results presented in terms of the number of detections per video frame translated to relative length of the road. A pothole instance can be detected and tracked in several consecutive frames. Panels on the right: road deterioration map produced based on the classification of number of pothole instances per 20-m segment. Panels (a1)-(a2) and (b1)-(b2) are associated with the dashcam video captured on a sunny day, and the results shown on (c1)-(c2) and (d1)-(d2) are based on a video recorded during an overcast weather condition after rain.

Discussion

The analysis of the training results revealed that the F1-score was maximized at a confidence value of 0.232 and, thus, such a value was used to threshold predictions in the test data. When validated against the test data, the model showed an increase in all metrics with values for P, R, and [email protected] of 0.75, 0.57, and 0.68, respectively. Such a performance boost was mainly driven by the model performance under overcast and wet conditions (P = 0.79; R = 0.58; [email protected] = 0.70). In contrast, the performance during sunny conditions (P = 0.60; R = 0.45; [email protected] = 0.47) was poorer than what was found against the validation data. Such contrasting performances between the sunny and overcast conditions agrees with the manual image annotation experience. The experience confirms that the visibility of potholes in the footage was more obvious following rainfall, when the potholes were filled with water, and lower reflections from the windshield were experienced due to the overcast sky.

Reflecting on the resulting road deterioration map, although being similar in pattern with road sections being more worn out than others, according to the exemplary road damage classification, the overall result was different. Whereas during the sunny conditions the road was in overall good shape, with only one “hot spot” section in Category IV, and none in Category III. For road management, such a result would require just a further monitoring throughout the operational season, or at most some spot improvement work. In contrast, assessing the same road after a rainy day reveals a different situation, i.e., a maintenance status which already limits the technical functionality of the road. The detection of extended sections in maintenance Categories III and IV, classified the road due for periodic maintenance, and work such as resurfacing needed to be scheduled soon.

These two contradicting outcomes from a road manager’s perspective could lead to false planning of resources and operation scheduling. Thus, limitations of optical sensors referred to sunlight reflections and poor visibility of light surface depressions under the canopy, when potholes are not water filled, must be accounted for. Consequently, the users of such an application need to factor in the conditions under which they can retrieve the best results. Furthermore, it is important to see our study as a first step toward the operational application, where different than in our study, a single road stretch may be driven through several times, thus providing the opportunity to average multiple tracking results over time and thus reducing the uncertainty of the pothole maps.

The quality of the final maps can also be influenced by another parameter – the positioning of the potholes. Accurately geolocating detected potholes depends heavily on the performance of the GNSS unit in the dashcam. The GNSS metadata highlight a high number of tracked satellites with HDOP values mainly below 1, suggesting an overall favorable geometry for calculating the position. Nevertheless, the number of satellite signals that can uninterruptedly reach to the modest antenna of the GNSS module may not be as high as what is shown in Figure 5(c). Our visual inspection confirmed that the local minima of the deviation graph in Figure 5(b), as an indicator of GNSS coordinates error, well corresponded to more open sky views in the forest.

Future work is planned to create a seamless pipeline that starts from transferring the captured videos to a cloud infrastructure on a near real-time basis. The videos uploaded to the cloud are instantly processed to update the current road deterioration maps. Availability of such a pipeline for the forest roads with active users can provide a frequently updated observation source for constantly improving the detection model and the accuracy of the generated deterioration maps.

Conclusions

An automated approach for monitoring forest road surface deterioration in terms of potholes was demonstrated in this article. The approach leverages computer vision to develop a model for detecting potholes in the captured videos of a standard dashcam. The videos have embedded geolocation information provided by a dual-constellation GNSS module. The results of applying the developed model to the video recordings under different weather conditions highlight an overall promising performance. However, a significantly higher success rate was observed for the case of combined overcast and wet weather, which created water-filled potholes. Further conditional factors, such as degree of pothole development and how these are affected by the various lighting conditions, should be considered for future studies. An exemplary road deterioration map (limited to pothole features) was demonstrated that can be frequently updated and utilized for preventative maintenance purposes.

The study also revealed some potential improvement opportunities. In case of a need for higher positioning accuracy for the potholes, a combination of hardware and processing enhancements can be considered for future studies. A dashcam with an external GNSS antenna input could improve signal reception and reduce the number of outliers in the GNSS-delivered coordinates. Moreover, a monocular depth estimation method could be assimilated to the processing for transferring the GNSS receiver’s location to the detected potholes.

Acknowledgements

This work is part of the Center for Research-based Innovation SmartForest: Bringing Industry 4.0 to the Norwegian forest sector (NFR SFI project no. 309671, smartforest.no).

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the Norges Forskningsråd [309671].