Comparative Evaluation of LiDAR systems for transport infrastructure: case studies and performance analysis

ABSTRACT

Mobile laser scanners are vital for intelligent transport infrastructure, capturing detailed 3D road representations, but their accuracy depends on factors like sensor positioning and environment. This study compares two van-mounted Mobile Laser Scanners (MLS): the dual head Lynx Mobile Mapper and the single head VUX-1 HA, along with the terrestrial laser scanner Faro Focus XX30. Using point cloud reference data from Faro Focus XX30 and GNSS data from Trimble R8, performance is assessed in road, urban, and semi-urban environments. Accuracy is measured by the difference between Trimble GNSS and MLS coordinates. Geometric features of each LiDAR are compared, and mapping tasks in road and urban areas are performed using a machine learning classifier. Results show the MLS-single head scanner achieves satisfactory accuracy in roads and semi-urban areas, while Faro performs better in urban settings for classification. MLS-single head excels in road environments, while Faro is superior in urban ones. This analysis aids researchers and professionals in selecting the appropriate mobile laser scanner for mapping transport infrastructure, providing valuable insights into MLS systems’ comparative performance across different environments.

KEYWORDS:

• LiDAR
• random forest
• terrestrial laser scanning
• geometric features
• GNSS

Introduction

In recent years, Mobile Laser Scanning (MLS) has received considerable attention in the application of digital infrastructure. For instance, MLS technology stands out in the detailed survey of civil engineering, road surveying (Balado et al., 2019; Soilán et al., 2020), and 3D city modelling (Wysocki et al., 2021; Zhou et al., 2018), which is particularly beneficial for smart cities, Building information modelling (BIM), and digital twins (Niță, 2021; Sofia et al., 2020). MLS systems could be mounted on a different moving platform such as a backpack (Gong et al., 2021; Hyyppä et al., 2020), van (Balado et al., 2021; Sánchez-Rodríguez et al., 2018), railway (Niina et al., 2018), or a boat (Schneider & Blaskow, 2021; Vaaja et al., 2013) and it incorporates Global navigation satellite system (GNSS) and Inertial measurement unit (IMU), which are the fundamental components of the MLS for the navigation (Qiu et al., 2023).

The mobile laser scanner operates by conducting a two-dimensional scanning perpendicular to the vehicle’s travel direction, effectively creating a three-dimensional scanning system by utilizing the vehicle’s motion dimension as the scanning axis (Xu et al., 2015). This is achieved by utilizing the vehicle’s motion dimension as the scanning axis. Simultaneously, the Global Navigation Satellite System (GNSS) provides precise location information of the vehicle, while the Inertial Measurement Unit (IMU) offers spatial attitude data. Through the fusion of laser scanners, GNSS, and IMU, comprehensive three-dimensional point cloud data of the scanned points is obtained.

There are various MLS in the market today with different specifications. Consequently, the question of how to assess the quality of these systems is receiving more attention. The performance of MLS data is fundamentally dependent upon density and this density can be influenced by multiple factors, including the speed of the van the angular resolution of the scanner, and the scanning frequency of the scanner (Shi et al., 2021). For example, scanline intervals increase with higher rotational frequency, lower pulse repetition frequency, or increased vehicle speed (Takahashi & Masuda, 2020). Moreover, point density decreases with greater distance from the laser scanner and varies based on the angles between laser beams and surfaces. It should be noted that not all MLS sensors have the same degree of accuracy and precision (Rashdi et al., 2022). Each sensor is influenced by factors such as environmental effects, processing time, the geometry of the MLS system, etc. For this reason, the evaluation of accuracy, precision, and point density of different MLS point clouds are the fundamental requirements for the quality of the 3D models (Niță, 2021; Sofia et al., 2020). To determine the expected quality of MLS, it is crucial to understand the systematic errors of individual sensors (Kalenjuk & Lienhart, 2022). However, such errors and their impact on the accuracy of the obtained point clouds are impossible to predict.

Hence, it is essential to have reference data to evaluate and determine the quality parameters that will allow its use for the intended purposes. The comparison of MLS point clouds with Terrestrial Laser Scanning (TLS) point clouds can be used to assess the accuracy of MLS systems (Tucci et al., 2018). TLS is a static laser scanner that involves emitting a laser beam towards a predefined scanning area by adjusting the deflection angle vertically and horizontally (Wu et al., 2022). When the laser beam encounters a reflective surface, it is reflected back to the receiver. Through various range measurement techniques, the distance between the scanner and the object can be accurately calculated. Because TLS does not experience low density and allows for a much better assessment of the local error in the tested point clouds, it is typically employed as ground truth data. Different approaches have been developed to compare and validate both MLS and TLS data (Hunčaga et al., 2020).

Numerous authors have investigated different ways to evaluate and enhance the accuracy of MLS data. Kaartinen et al. (2012) tested five MLS systems on a road environment at 1700 m. They used artificial point features such as building corners, poles, and curb corners to evaluate horizontal and vertical accuracy. Fryskowska-Skibniewska and Wróblewski (2018) used TLS as reference data to analyse the accuracy of an MMS. They performed their experiments using 395 checkpoints and the lengths of the building features. Bauwens et al. (2016) evaluated and compared a hand-held MLS (HMLS) with two TLS techniques (single scan: SS and multi-scan: MS) to estimate various forest characteristics such as canopy height model (CHM) and DBH. From their experiments, they concluded that SS is the fastest, but many trees were missing. HMLS, on the other hand, produces superior results for identifying different tree characteristics and is quicker than MS-TLS. Since a limited range of ZEB1 has been used in their study, the canopy is inadequately depicted, which makes HMLS less useful at heights above 15-20 m. Moreover, Hunčaga et al. (2020) tested their experiments on three-point cloud datasets acquired from TLS, handheld mobile laser scanning (HMLS), and close-range photogrammetry (CRP). The accuracy of stem curve models was evaluated and compared. When considering the height level of the tree stem, differences in diameter predictions and their errors associated with various height sections were examined to show the applicability of the employed device for mapping a forest stand. The study showed that the lowest RMSE was attained by TLS and CRP at a height of 1.3 m, whilst HMLS was at a height of 8 m. Toschi et al. (2015) proposed a methodology to assess the accuracy of MLS-single head VMX-450 MMS using non-parametric-statistical-models-for the robust estimation of error dispersion. The reference data was collected with TLS and photogrammetry.

Kalvoda et al. (2020) reported the assessment of MLS point cloud accuracy based on the influence of control points(CPs) numbers and configuration. They quantified system accuracy by comparing the MLS data with two ground truth datasets: test point field (TPF) and TLS. Point clouds were separated into facade and road data subsets for the evaluation of horizontal and vertical accuracy respectively. Al-Durgham et al. (2021) registered each slice of point clouds with TLS data to assess the quality of MMS. Kalenjuk and Lienhart (2022) offered a practical method, emphasizing the quality criterion of geometric accuracy of point clouds. The study is evaluated on three challenging datasets of MLS. TLS data with high accuracy surveyed Ground control points (GCPs) were used as reference data to analyse the accuracy of the point clouds. However, the analysis failed to minimize the systematic error because of distortions and other influencing factors associated with laser scanners.

Recent trends in mobile applications on iPad Pro or iPhone appear to be favourable, making data acquisition faster and cheaper. Bobrowski et al. (2023) tested iPad Pro for trunk parameter estimation in forestry applications and compared their data with a measuring tape and TLS point clouds. Furthermore, Balado et al. (2022) compared three LiDAR systems: Faro X330, Zeb-Go, and Apple iPad Pro in a cultural heritage application. They concluded that a single TLS scan offers better accuracy, however, if multiple scans are registered, the error caused by registration may cause a loss in precision as compared to HMLS. Additionally, Annok et al. (2021) evaluated MLS elevation accuracy on two polygons with distinct conditions. The first dataset had tall trees along both sides of the roads, while the second dataset featured high trees primarily in the southeast direction. In the northeast direction of the second dataset, there was a mix of open space along with high trees and a nearby building a few meters away from the research point. Habib et al. (2021) examined the use of mobile LiDAR mapping system (MLMS) for mapping roadside ditches to analyse their slope and drainage. Various MLMS units were tested, including an unmanned ground vehicle, unmanned aerial vehicle, portable backpack system (both handheld and vehicle-mounted), medium-grade wheel-based system, and high-grade wheel-based system. The accuracy of the point cloud generated by these units was compared to an RTK GNSS survey. The point clouds from all MLMS units were consistent vertically, with a maximum difference of ±3 cm for solid surfaces and ±7 cm for vegetated surfaces.

It is worth noticing that the desirable accuracy for mapping road infrastructure with LiDAR depends on the specific application. For example, Lin et al. (2019) found that mobile LiDAR data with root mean square errors (RMSE) of 0.13 ft, 0.08 ft, and 0.09 ft along the x, y, and z-directions, respectively, were acceptable for mapping airfield infrastructure. He et al. (2018) demonstrated the potential for using a LiDAR-based mobile mapping system in road inventory extraction, with a root-mean-square-error of road mark greater than 20 cm. Teo (2018) proposed a SLAM method for vector-based road structure mapping using multi-beam LiDAR, achieving an average global accuracy of 0.466 m without the aid of GPS.

From the aforementioned literature, it is noticed that validation of MLS data is necessary to obtain more detailed information about the environment, which can further aid engineers and scientists to work on mobile mapping systems efficiently. In this regard, this study focuses on the comparison between two mobile laser scanners: MLS-single head and MLS-dual head. The accuracy of both MLS with respect to Trimble GNSS will be evaluated. Additionally, the accuracy of the point clouds will be assessed with TLS (Faro X330).

The main aim of this article is to address the challenges associated with processing point cloud data from two MLS with different configurations, providing valuable insights to assess their suitability for application in intelligent transport infrastructure.

Given the above main objective, the specific objectives are stated as follows:

1. Evaluate the absolute accuracy of MLS based on developed accuracy evaluation procedures on different scenarios (road, urban and semi-urban).

2. Investigate the geometric features and compare them with each sensor and environment.

3. Analyse different scenarios for machine learning inputs depending on the contribution of LiDAR-derived features of the classification process.

Material and methods

Case study

Two MLS, MLS-single head VUX-1 HA (RIEGL - RIEGL Laser Measurement Systems. (n.d.) and Lynx Optech (Home | Teledyne Geospatial (n.d.) are used for the data acquisition. TLS Faro Focus X330 (FARO Focus Laser Scanner| Hardware | LIGHTHOUSE, (n.d.) is used as a point cloud reference data, and Trimble R8 (Trimble Geospatial | Survey and Mapping Solutions, (n.d.).The technical characteristics of these devices are listed in Table 1. For the sake of scientific clarity, we will consistently designate the Lynx Optech system as the “MLS-dual head system”, Riegl as the “MLS-single head system” and Faro Focus as “TLS” throughout this article.

Table 1. Technical characteristics of sensors used.

	Riegl VUX-1 HA	Lynx Optech	Faro Focus X 330
Range (m)	420	200	330
Field of view	360°	360°	300 /360°
Precision (mm)	3, 1σ @30 m	8,1σ	2
PRF (pulse repetition frequency) (kHz)	300–1000	75–500	NA
Scan frequency (Hz)	10–250	80–200	NA
Wavelength (nm)	Near infrared	NA	1550 nm
Weight (kg)	3.75	NA	5.2
Maximum pts/s	1,000,000	500,000	122.000–976.000

The MLS-single head system, mounted on a moving van, was integrated with a Riegl VUX-1 HA laser scanner, Trimble Zephyr 3 Rover GNSS, and STIM300 IMU for precise positioning and georeferencing of the point cloud. The Trimble Zephyr 3 Rover, designed for precision RTK and roving applications, minimizes multipath, ensuring robust low elevation tracking with millimeter phase center repeatability. Its 50 dB signal gain enables reliable tracking in challenging environments and accommodates long cable runs.

The STIM300 IMU combines three highly accurate MEMS-based gyros, three stable accelerometers, and three inclinometers. Each axis is factory-calibrated for bias, scale factor, and temperature effects, allowing high-accuracy measurements in the −40°C to + 85°C temperature range. The laser scanner has a maximum effective measurement rate of 1000,000 points per second with a maximum scan speed of 250 scans per second. The scanner’s operating wavelength is 1550 nm, which offers information on the surface reflectivity, signal amplitude, and echo duration in addition to the 3D position of each measured point.

The MLS-dual head mobile mapping system is equipped with two laser scanners, a ladybug camera, and Applanix’s POS LV 520 system, which incorporates an IMU along with a 2-antenna heading measurement system (GAMS). This integration ensures absolute accuracies (RMS) of 0.015° in heading, 0.005° in roll and pitch, as well as 0.02 m in X and Y positions, and 0.05 m in Z position. The laser scanners allow to collect the data at 500,000 measurements per second with a 360° field of view (per scanner). It has a measuring range of more than 100 meters to a target with a 20% reflectivity.

TLS point clouds were captured by Faro Focus X330 as reference data and Trimble R8 as GNSSS reference data. Faro Focus X330 has a range systematic error of ±2 mm. The scanning range is up to 330 m and can collect 976,000 points per second. The laser scanner is perfect for applications based on surveying since it has an inbuilt GNSS receiver that allows for the post-processing correlation of individual scans. The Trimble R8 has all the characteristics of a GNSS system, providing power, accuracy, and unsurpassed performance in a durable and small package. It has very low noise GNSS carrier phase measurements in a 1 Hz bandwidth with less than 1 mm precision. The signal-to-noise ratio is expressed in Db-Hz.

In this study, the trajectory acquisition rate for GNSS/IMU data differed between the MLS-single head and MLS-dual head system. The GNSS/IMU with MLS- dual head system was operated at 200 Hz, while the GNSS/IMU with MLS-single head system was operated at a higher rate of 500 Hz. This variance in trajectory acquisition rates reflects the unique real-time tracking capabilities inherent to each system. Regarding laser scanner properties, both MLS systems showcase distinctive features. MLS-dual head scanners were captured with a scanning rate of 200 Hz and a Pulse Repetition Rate (PRR) of 250 kHz (potentially adjustable to 500 kHz), demonstrating flexibility. On the other hand, the MLS- single head was operated at 250 Hz with a PRR of 1000 kHz (typically reduced to 750 kHz for regular operation).

The test field was 23.8 km long, starting from the University of Vigo (Vigo campus) to the Navia, in Vigo (Spain), and back again along the opposing lanes of the highway as shown in Figure 1. The measurements with the MLS systems were made without making any stops with the van. The van’s speed during the data acquisition was within the normal range of 50–80 km/h in the road environment, 50 km/h in the semi urban environment and 30 km/h in the urban environment. The data acquisition took place during daylight hours, characterized by clear sky conditions. For the point cloud reference data and GNSS reference data comparison, the TLS and GNSS systems were placed in three different case studies (Figure 1). Two point clouds were acquired with the TLS system in case studies 1 and 2. With GNSS, several control points have been taken in the three case studies (detailed explanation in Section 2.3). The choice of these three case studies was in the interest of analysing the performance of the MLS systems in a:

Figure 1. Survey map. Start and back path for MLS-single head and MLS-dual head. Zoomed images in case studies measured with Faro (TLS) and Trimble GNSS for the corresponding comparisons. 1) case study 1: road environment, 2) urban environment, and 3) case study 3: semi-urban environment.

• Case study 1: road environment with the guardrail, vegetation, and road markings.

• Case study 2: urban environment with road and predominance of built-up area.

• Case study 3: semi-urban environment with a lot of vegetation, road, and little built-up area.

With these three case studies, the different possibilities to be found by MLS systems were analysed. For proximity issues in the GNSS data comparison, the point cloud acquired on the start path was used for Case studies 1 and 2, and the point cloud acquired on the back path for Case study 3. For feature comparison, only the start path was used.

GNSS comparison

The testing field for GNSS data comparison was composed of 36 known control points measured by the Trimble R8 GNSS. The reference points were mainly targets (the centre and one of its diagonals), poles (base of traffic signals, lights, and bus stops), and corners (road markings and sidewalks) as shown in Figure 2. The GNSS accuracy of the Trimble R8 was always less than 3 cm. The choice of these control points was due to their usual good visualization in point clouds given their geometric characteristics and surface properties. We imported a csv file containing control points along with MLS point clouds into CloudCompare. Using this reference data, we conducted a performance comparison. It is important to highlight that we faced limitations when it came to directly measuring the center of poles using GNSS. In practical terms, we treated our measurement as that of the south corner without attributing significant importance to this specific aspect of the study. Despite the inherent error introduced due to the discrete nature of LiDAR data, it can generally be regarded as negligible, particularly considering the high resolution of the equipment we utilized. The GNSS error was evaluated by measuring differences between Trimble and MLS coordinates in absolute terms.

Figure 2. Reference points with GNSS Trimble in each case study. 1) case study 1: road environment, 2) case study 2: urban environment, and 3) case study 3: semi-urban environment.

Feature extraction

Geometric features are highly relevant for distinguishing varying classes. They provide a spatial arrangement of a point cloud with respect to the local neighbourhood, which also takes into consideration the relative eigenvalues. A collection of eight local 3D shape features was obtained for a given 3D point i and its k = 25 nearest neighbours by using respective derived eigenvalues (${\lambda }_1,\mathit{\hspace{0.17em}}{\lambda }_2,\mathit{\hspace{0.17em}}{\lambda }_3)$. These features were calculated by adapting the method used by (Weinmann et al., 2013):

• Linearity L (Equation (1)(1) $\mathit{L}=\frac{{\lambda }_1-{\lambda }_2}{{\lambda }_1}$(1) ): Describes 1D surface such as line-like shape objects, and (poles, traffic lights).

• Planarity P (Equation (2)(2) $\mathit{P}=\frac{{\lambda }_1-{\lambda }_3}{{\lambda }_1}$(2) ): Describes 2d structure such as plane shape objects (facades, roads).

• Scattering S (Equation (3)(3) $\mathit{S}=\frac{{\lambda }_3}{{\lambda }_1}$(3) ): Describes sphericity of an object for example sphere like objects (vegetation).

• Omnivariance O (Equation (4)(4) $\mathit{O}=\sqrt[3]{{\lambda }_1{\lambda }_2{\lambda }_3}$(4) ): Describes how neighbourhood of points are distributed unevenly within a 3D volume.

• Anisotropy A (Equation (5)(5) $\mathit{A}=\frac{{\lambda }_1-{\lambda }_3}{{\lambda }_1}$(5) ): Ability to differentiate between oriented and non-oriented objects.

• Eigentropy E (Equation (6)(6) $\mathit{E}=\sum _{\mathit{i}=1}^3{e}_{i}ln\left(e_i\right)$(6) ): Describes the order or disorder of 3D point clouds inside the covariance ellipsoid (Weinmann et al., 2014).

• Change of curvature C (Equation (7)(7) $\mathit{C}=\frac{{\lambda }_3}{{\lambda }_1+{\lambda }_2+{\lambda }_3}$(7) ): Specifies the curve in the given point clouds and is calculated by ratio of minimum eigen value to sum of eigen values.

  (1) $\mathit{L}=\frac{{\lambda }_1-{\lambda }_2}{{\lambda }_1}$(1)
  (2) $\mathit{P}=\frac{{\lambda }_1-{\lambda }_3}{{\lambda }_1}$(2)
  (3) $\mathit{S}=\frac{{\lambda }_3}{{\lambda }_1}$(3)
  (4) $\mathit{O}=\sqrt[3]{{\lambda }_1{\lambda }_2{\lambda }_3}$(4)
  (5) $\mathit{A}=\frac{{\lambda }_1-{\lambda }_3}{{\lambda }_1}$(5)
  (6) $\mathit{E}=\sum _{\mathit{i}=1}^3{e}_{i}ln\left(e_i\right)$(6)
  (7) $\mathit{C}=\frac{{\lambda }_3}{{\lambda }_1+{\lambda }_2+{\lambda }_3}$(7)

Other features, such as local point density has been calculated from the neighbourhood as shown in Equation (8)(8) $\mathit{D}=\frac{\mathit{k}+1}{\frac{4}{3}\mathit{\pi }{r}_{k-NN}^3}$(8) . Where$,\mathit{\hspace{0.17em}}{r}_{k-NN}^3$is the largest radius between point x and its neighbours. The impact of calculating point cloud density is quite relevant for detecting different road objects. Since point cloud density varies depending on the speed and distance from the scanner to the relative orientation, it provides a detailed perspective of the road environment. In this experiment, the density is calculated between k = 26 nearest points with KNN algorithm. In addition, the intensity values of each sensor have been extracted and displayed in Section 3.

(8) $\mathit{D}=\frac{\mathit{k}+1}{\frac{4}{3}\mathit{\pi }{r}_{k-NN}^3}$(8)

Machine learning classifier

The main objective of the implementation of machine learning classifier is the usability evaluation of the three LiDARs through mapping tasks, one of the main uses of point clouds today. For this reason, a Random Forest (RF) algorithm was implemented for point-by-point classification of point clouds scanned with two MLS system and TLS using the features extracted in Section 2.4 and performing one training for each LiDAR and study area (six trainings in total). We employed Random Forest classifier because it excels in handling complex data with numerous dimensions and multicollinearity issues (Belgiu & Drăguţ, 2016). It is known for its speed and resilience against overfitting. In a previous study by (Rashdi et al., 2023) we conducted a comparison involving Random Forests (RF), Support Vector Machines (SVM), and Neural Networks. Our results demonstrated that RF consistently outperformed the other classifiers, which is why we chose to utilize RF in our current study.

The Random Forest classifier (Breiman, 2001) consists of a set of decision trees drawn from a randomly chosen training set. The final class of the test object receives aggregated votes from several decision trees for decision-making. Probabilities can be calculated by adding up the single tree classifier’s votes; the percentage of single trees that assigned an observation to a particular class is the probability that the observation belongs to that class. It is non-parametric where variable preselection is not required. It performs effectively when handling different features and normalization is not required. Additionally, they are not sensitive to outliers, making them suitable for classification tasks.

Since supervised classification methods such as RF require labelled data, point clouds were manually labelled using Cloud Compare. The classes were according to the road or urban environment, selecting the most relevant ones according to their usefulness and the balance of the number of points. For the road environment, the five classes selected were: road, guardrails, traffic marks, vegetation, and others (which included bus stops, cars, and traffic signs). In the urban environment, six classes were selected: road, traffic marks, roadside (curb, sidewalk, and median strip), buildings, cars, and others (pole-like objects, waste containers, vegetation, and pedestrians). The vegetation class includes both shrubs and trees. The building class has six floors in total. However, only three floors have been shown in the segment.

To reduce the number of points in the scene and balance the classes, since the road concentrates the highest percentage of the points, a spatial point density reduction of 1 cm was applied. Figures 3 and 4 show the number of points per class after the reduction process for road and urban environments respectively. For training, 1000 samples per class were randomly selected. Testing was performed on the remaining set of points. The training was performed with a cross-validation of five sets.

Figure 3. Number of points per class in road environment case study.

Figure 4. Number of points per class in urban environment case study.

To evaluate the effectiveness of the proposed method, four accuracy metrics were used which include Precision, Recall, Fscore, and IoU as shown in Equations 9(9) $\mathit{Precision}=\mathit{\hspace{0.17em}}\frac{\mathit{TP}}{\mathit{TP}+\mathit{FP}}$(9) –12(12) $\mathit{Io}{U}_n=\frac{T{P}_n}{\mathit{T}{P}_n+\mathit{F}{P}_n+\mathit{F}{N}_n}$(12) , with information of true positives (TP), true negatives (TN), false positives (FP) and false negatives (FN). These scores indicate how well the model performs for each class.

(9) $\mathit{Precision}=\mathit{\hspace{0.17em}}\frac{\mathit{TP}}{\mathit{TP}+\mathit{FP}}$(9)

(10) $\mathit{Recall}=\mathit{\hspace{0.17em}}\frac{\mathit{TP}}{\mathit{TP}+\mathit{FN}}$(10)

(11) $\mathit{Fscore}=2\ast \frac{\mathit{Precsion}\ast \mathit{Recall}}{\mathit{Precision}+\mathit{Recall}}$(11)

(12) $\mathit{Io}{U}_n=\frac{T{P}_n}{\mathit{T}{P}_n+\mathit{F}{P}_n+\mathit{F}{N}_n}$(12)

Results

GNSS test

The results of errors for each Case study are depicted in Figure 5. Mean absolute errors were also computed for both MLS data, for each Case study as depicted in Table 2. The absolute coordinate difference serves as an additional metric to evaluate performance of two MLS systems. Analyzing these differences can provide methodological insights into the systematic variations present in the measurements of each system.

Figure 5. Absolute error analysis of the control points in each coordinate system (X,Y & Z) of both MLS-dual head and MLS-single head. t: targets, c: corners, and p: poles.

Table 2. Mean absolute error of each coordinate system of the MLS system and the comparison between different case studies using control points. The bold values refer to maximum errors.

Mean absolute error (m)
	Xdual head	Ydualhead	Zdualhead	Xsinglehead	Ysinglehead	Zsinglehead
Case study 1	0.273	0.148	0.335	0.034	0.058	0.043
Case study 2	0.550	0.413	0.622	0.409	0.599	0.690
Case study 3	0.163	0.265	0.262	0.035	0.076	0.055

It is noted that Case study 1 shows a pretty good figure in terms of absolute accuracy. The lowest accuracy error was found with Riegl data with a mean absolute error of 0.034 m in X coordinates, 0.058 m in Y coordinates, and 0.043 m in Z coordinates.

As expected, results from Case study 2 reveal how GNSS accuracy is affected in urban areas. This is because satellite visibility in urban areas is interfered with by vegetation, buildings, vehicles, and other urban furniture. The maximum error observed in Case study 2 is 0.55 m for the MLS- dual head data in correspondence with the X coordinate and 0.599 m and 0.69 m for the MLS- single head data in correspondence with Y and Z coordinates, respectively. The enhanced precision of the GNSS/IMU system installed with the MLS-dual head system is attributed to its superior performance in capturing both horizontal and vertical accuracy. This superiority is further underscored by the comprehensive integration of auxiliary sensors, IMU, and dual-frequency GNSS antennas within the Applanix POS LV 520. The cohesive approach of the Applanix POS LV 520 plays a crucial role in elevating the system’s accuracy levels. This, in turn, positions it as more precise and of higher quality when compared to the GNSS/IMU of MLS-single head system. This distinction holds particular significance in scenarios involving urban asset detection, where the extraction of linear elements, especially roads, poses a primary challenge. Additionally, MLS- dual head system has two LiDAR sensor heads, which provide a wider field of view from different angles. These factors have enabled MLS-dual head data to effectively alleviate occlusions in urban areas, thereby contributing to its superior performance over MLS – single head data.

In Case study 3, the highest accuracy error was found in MLS-dual head data, giving a maximum error of 0.163 m in X coordinates and 0.265 m and 0.262 m in Y and Z coordinates, respectively. It should be noted that some of the control points (targets and corners) were missing as they were surrounded by vegetation and therefore an intuitive approximation had to be made with reference to the TLS point clouds. However, MLS-single provided better accuracy with mean absolute error of 0.035 m, 0.076 m in X and Y coordinate, and 0.055 m in Z coordinate.

Figure 5 provides an additional illustration of error analysis using the |ΔX|, |ΔY| and |ΔZ| for both MLS-dual head and MLS-single head equipment, to have a more direct visualization of the error values of the different control points. The control points in the horizontal axis are distinguished with letters “t” for targets, “c” for corners, and “p” for poles.

Analysing Figure 5, the elevation difference along the control points is higher in MLS-dual head data, giving the maximum error of 1.32 m in control point “8t”. From 6c to 23c control points, the errors are generally higher than 0.5 m, and the remaining control points are lower than 0.5 m, with little variation between them. With this regard, it is proved that the best GNSS accuracy is achieved in case studies 1 and 3. Various factors influence LiDAR data quality, including environmental effects, point density, and vehicle speed. Case studies 1 and 3, focusing on road and semi-urban environments, exhibited fewer occlusions and irregularities in natural landscapes. Conversely, case study 2, centered on an urban area, showed some point clouds with sparse data and occlusions, leading to perceived lower accuracy. Maintaining constant speed in the urban environment is challenging, due to the presence of man-made objects and pedestrians in the surroundings. One solution could be fine-tuning MLS settings. If there is a requirement to capture highly accurate details such as road cracks, we can increase the pulse repetition rate while maintaining constant power and minimizing the scan rate. This modification enables us to collect a greater number of data points within the same time frame, providing better results.

Feature test

Figure 6 compares the distribution of all datasets as boxplots. As the figure shows, significant differences were noted in the intensity values of each sensor in both environments. The achieved point density for Faro is quite similar in both environments and is higher than MLS.

Figure 6. Box plots representation of MLS 2(MLS-dual head), MLS 1(MLS-single head) and TLS in different environments (road and urban). Where: red points represent outlier values & red lines represent median values.

Regarding eigen-related features, for linearity, performance across MLS-dual head Road, TLS Road, MLS-dual head Urban, and TLS Urban did not vary, with the exception of MLS-single head Road, which resulted in a large quartile range. This conclusion indicates that the data was scattered with large fluctuations ranging from 0.0002 to 0.9996 in MLS-single head road. Likewise, for the planarity, MLS-single head Road data largely differs from other datasets. Interestingly, the median value of MLS-dual head Road is somehow close to MLS-dual head Urban, but the quartile range is different. Scattering results demonstrate variations not only between the sensors but also between environments.

Features comparison of each LiDAR in the road and urban environments is displayed in Figures 7 and 8. The results of the intensity of each sensor are noticeably different. MLS-single head achieves the highest intensity values in both urban and road environments. However, the intensity distribution in the MLS-dual head point cloud is the best to differentiate road objects. The point density is distributed according to the movement of the laser scanners. While the TLS point cloud has a very high density in the immediate vicinity of the scan position, in the MLS point cloud, the density is higher in the rolling zone of the trajectory.

Figure 7. Behaviour of features in road environment: intensity, density, & linearity. The upper row indicates MLS-dual head data, middle row for MLS-single head and last row for TLS. The triangle (right) shows the color bar for linearity (l) in red, planarity (p) in green, & scattering (s), respectively.

Figure 8. Behaviour of features in urban environment: intensity, density, & linearity. The upper row indicates MLS-dual head data, middle row for MLS-single head and last row for TLS. The triangle (right) shows the color bar for linearity (l) in red, planarity (p) in green, & scattering (s) in blue, respectively.

The point cloud from MLS-dual head provides a good analysis of linearity, planarity, and scattering features following the expectations. On the contrary, MLS-single head has shown some parts of the road as linear (red color). This might be due to the high point density close to the sensor, making the k nearest neighbours distributed along one single scanline. From Figure 8 it is evident that some of the points of MLS-single head and TLS are represented as planar (green) in the poles of the urban environment. Also, most of the vegetation points in the TLS are detected as linear (red) instead of scattering (blue) in the road and urban environment.

Machine learning classifier test

As seen from Figure 9, point clouds classified from MLS-dual head, MLS-single head, and TLS, had very similar success rates. In most cases, the difference was less than 5%. However, in the classes traffic marks and others, the MLS-single head f-scores were substantially better than those of MLS-dual head and/or TLS. In general, and despite the small differences, the MLS-single head point clouds were better segmented with the RF algorithm.

Figure 9. Metric results of point cloud semantic segmentation of road and urban environment.

The urban environment metrics as shown in Figure 9 were not as favorable to the MLS-single head sensor. An f-score analysis indicated that there were significant changes between the segmentation of the three sensors. The segmentation of road, roadside, and buildings was better with the TLS; traffic signs and cars with the MLS-dual head; and the class others were better segmented with the MLS-single head. Overall, the best performance was achieved with the TLS point cloud. In this survey, both MLS followed a trajectory at a road junction that involved a 90º turn near the TLS scan position, so three point clouds from all sensors were very similar from a point density and perspective point of view.

Figure 10 depicts the predicted classification results of each sensor in the road and urban environment. In the road environment, roads and vegetation were well classified. MLS-dual head data had made a minor classification error, misclassifying the road near the bus stop as others. Additionally, some points of class vegetation were mistaken as others. Roads in an urban environment were mislabelled as others. This is not surprising since roads and road markings have the same geometric characteristics. Nevertheless, TLS appeared to be effectively classified with respect to roads and road markings. Moreover, certain points of the class buildings were incorrectly classified as cars.

Figure 10. Predicted classes of each sensor: reference, MLS-dual head, MLS-single head, & TLS. Road environment (upper) and urban environment (lower).

Discussion

As the results demonstrate in the GNSS comparison (Section 3.1), MLS-single head data were significantly more accurate than MLS-dual head, especially in Case studies 1 and 3. This was because the targets were not easily identifiable in MLS-dual head data. It should be noted, however, occlusions due to vehicles, vegetation, and building were the main problem for MLS data acquisition and it affected the accuracy in urban areas. In addition, the human visual limitation during the selection of MLS points also affected the accuracy calculation.

Case study 1 (road environment) using MLS-single head had the smallest errors in both horizontal (0.034–0.058 m) and vertical (0.043 m) accuracy, which are consistent with previous studies. For example, Kaartinen et al. (2012) reported 0.025 m vertical and 0.01–0.02 m horizontal accuracy and Fryskowska-Skibniewska and Wróblewski (2018) reported 0.042 m vertical and 0.06 m horizontal accuracy.

Also, MLS-single head in Case study 3 attained the second best accuracy, which was three units higher than the limit but is acceptable. Whereas, in Z coordinates, the mean absolute error achieved was (0.04–0.05 m) similar to the state of the art. MLS-dual head data, however, could not deliver desirable accuracy, and this discrepancy may be related to the target’s lack of visibility and assumption based on TLS data. In general, achieving positional accuracy at the millimeter level is challenging, even with advanced LiDAR equipment, primarily due to problems with GNSS reception and system calibration. Additionally, accurately identifying corresponding points in point clouds from different scans is not a simple task, as it is difficult to determine which points correspond to each other.

Collectively, these papers imply that the desirable accuracy for LiDAR-based road infrastructure mapping varies depending on the specific application and the level of detail required. Comparing the results, it is evident that horizontal and vertical errors (Figures 11 and 12) obtained in Case Study 1, for MLS-single head (0.10 m and 0.05 m, respectively), are consistent with the corresponding RMSE values reported by (Veneziano et al., 2002). Thus, it can be inferred that the elevation and position accuracy attained in Case study 1 using the MLS-single head LiDAR system meet the established standards. The vertical error observed in Case Study 3 also closely aligns with the methodology described in (Veneziano et al., 2002). However, Case study 2 exhibits the poorest performance, with significantly higher errors and a lack of correlation with the standard accuracy compared to Case study 1 and 3.

Figure 11. Horizontal RMSE comparison between MLS-dual head and MLS-single head.

Figure 12. Vertical RMSE comparison between MLS-dual head and MLS-single head.

The provided results in feature comparison (Section 3.2) (Figures 7 and 8) revealed a huge difference in intensity values between MLS-dual head, MLS-single head, and TLS since each sensor has different wavelengths which leads to heterogeneous intensity values. As seen from the box plots (Figure 6), there was a big difference found in MLS-single head data (road and urban environment). Therefore, density information might be a major factor influencing the estimation of features. It was found from the density representation that TLS point clouds were denser than those of MLS-dual head and MLS-single head. In a road environment, MLS-single head and MLS-dual head data showed the most variance in densities, while the TLS provided figures that appeared to be somehow constant, except for the points close to the sensor. In addition, it was apparent in Figure 7 that the very distinct road to the TLS is considered as a linear surface (red) as the point clouds are quite sparse.

From the classification results (Section 3.3), it is shown that MLS-single head had higher overall accuracy in the road environment whereas the TLS achieved high overall accuracy in the Urban environment. The best classified classes were road, vegetation, and buildings, which were also the ones with the highest number of points. The classes that were highly affected were roadsides and traffic markings.

The main limitations encountered in our study were within an urban environment due to several factors, namely:

Invisibility of targets: In the urban environment, certain targets or reference points necessary for accuracy assessment were not readily visible or accessible. The presence of tall buildings, dense vegetation, or other obstructing elements hindered the direct observation or placement of targets.

Approximations leading to increased errors: Due to the aforementioned challenges in target visibility, approximations had to be made in estimating the true measurements of certain features. These approximations introduced inherent errors, which subsequently affected the overall accuracy assessment of our study.

Blockage of GPS signals: The presence of buildings and the movement of vehicles within the urban environment caused significant blockage and interference with GPS signals. This obstruction led to difficulties in precisely determining and recording location information.

All the studies reviewed so far, however, suffer from the fact that mapping in urban areas with MLS is still the biggest challenge.

Conclusions

In this paper, three sensors; MLS single-head system, MLS double-head, and TLS were evaluated and compared in urban, semi-urban, and road environments. The case studies confirm the importance of each sensor in different scenarios. Conducted analysis from GNSS comparison confirms that MLS-single head in road environment delivers desirable accuracy that is adequate for mapping road infrastructure. According to the classification experiments, MLS-single head performs efficiently in the road environment whereas TLS performs well in an urban environment (both MLS overall results were similar in an urban environment). For roads and vegetation mapping, MLS-single head could be a better option. In contrast, for building inspection TLS is an appropriate choice. Nevertheless, the fusion of MLS and TLS can open doors to solve this problem. The findings of this study have several important implications for future practice. Processing point clouds collected at different times and with varying point densities using diverse mobile laser scanners poses challenges. To ensure a consistent approach applicable to various applications, like detecting road markings and traffic signs, it is essential to normalize the point cloud data. Future research will focus on point cloud normalization based on intensity and geometry can be applied to different datasets in order to have uniform characteristics, which will help in asset detection of road assets in transport infrastructure.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

All research data supporting this publication are directly available within the publication.

Additional information

Funding

This project received funding from the European Union’s Horizon 2020 research and innovation programme under Marie Sklodowska-Curie grant agreement no. 860370. The authors would like to thank the Government of Spain through projects RYC2022-038100-I funded by MCIN/AEI/10.13039/501100011033 and FSE+. PID2019-108816RB-I00 funded by MCIN/AEI/10.13039/501100011033.