A geometric modelling approach to estimate apple fruit size by means of LiDAR 3D point clouds

ABSTRACT

Remote sensing in agriculture aims to search new methods to monitor fruit at the tree, thus improving the estimation of yield-related variables. Light detection and range (LiDAR) scanning was introduced to obtain geometric and radiometric information from fruit surfaces by means of 3D point clouds. A geometric model to estimate apple size by means of segmented 3D point cloud of fruit is proposed in the present study. The model consists in the approximation of 3D point clouds to a reference shape given by 2D Fourier series expansion. Each point cloud was approximated to its reference shape using an iterative error minimization routine. The geometric model was applied to laboratory and field data of spheres and apples during fruit growth, ranging from 60 to 151 days after full bloom (DAFB). An overall $\mathit{RMSE}$ between measured and predicted fruit radius of 20.1, 76.8, and 119.1% was found for the geometric model, mean, and maximum Euclidean distance approaches, respectively, including all studied growth periods in field conditions. Moreover, the linear regression on measured and predicted values showed considerably improved coefficient of determination ($R^2$) of the geometric model in comparison to Euclidean distance calculations with $R^2$ values of 0.76 and 0.49 for laboratory and field scanned apples, respectively. The data processing method enables fruit monitoring and their application of terrestrial LiDAR sensing in precise orchard management.

KEYWORDS:

• 3D point clouds
• apple
• centroid approximation
• fourier series
• LiDAR
• ToF sensor

Introduction

Research in precision horticulture has been contributing in the pursuit of new strategies to build up resilience of food production to climate change. Particularly, the recent technological advances in the fields of robotics, optical sensing methods, and modeling capabilities for crop monitoring at the tree and in postharvest processes can provide tools to guide harvest and postharvest managers.^[1] Thereof, diverse methods for 3D spatially resolved fruit identification and size estimation had been studied, including RGB-D and thermal imaging, together with RGB-based structure from motion approaches.^[2–4] However, such imaging technologies are often influenced by environmental conditions such as lighting and temperature, in detriment of image quality, temporal availability, and prediction performance.^[5,6] Additional complexity in the incorporation of such approaches is given by the mathematical processing time in analyzing large sets of data, making models time consuming for most commercial computational capabilities.

Among newly studied fruit recognition and analysis methods, light detection and ranging (LiDAR) has been proven reliable in the acquisition of geometric and radiometric information in orchards by means of 3D point clouds.^[7–9] By having its own light source, LiDAR systems are able to function in different instant and within canopy lighting conditions, adding flexibility of use considering the time of the day and feasibility in orchard applications. Crop managing-related variables such as soil variability, crop load, plant protection, and water needs had been studied by means of LiDAR technology.^[10,11] Furthermore, mathematical transformation of point clouds was investigated for the identification of apple point clouds in canopies. Using a fruit segmentation algorithm based on geometric features of point clouds and return signal strength intensity, apples were segmented and distinguished from leaves, and woody parts in foliated and defoliated apples trees.^[12,13] The thresholds to distinguish fruit and other tree parts were described in similar ranges depending of raw data^[3] or calibrated return signal strength intensity pre- and postharvest.^[13,14] The feasibility of the method in comparison to RGB and RGB-D fruit segmentation approaches was pointed out.^[3,15]

Despite the advances using 3D point clouds, fruit size estimation by means of LiDAR 3D point clouds remains a difficult task, mostly due to occlusion by other tree parts and varying geometric and radiometric attributes of specific horticultural produce. Typically scanned fruit data capture only a certain percentage of the produce surface, making it difficult to obtain a full apple geometry. Recent methods to estimate apple size consist in fitting a sphere of known size, or using an assumed complete fruit geometry as a template to which a given point cloud is fitted.^[13] The most common approach is to estimate apple size by means of the largest distance between points within a point cloud, namely largest segment approach, which consider the Euclidean distance between extremely located points within a fruit point cloud.^[15] However, such approach ignores the specific shape of fruit, which is subject to change during fruit growth.

The description of detailed fruit geometry for digital process simulation in the supply chain was studied for diverse horticultural produce.^[16] Among the most promising approaches to model shape of spherical produce is the Fourier series expansion, where shape features are modeled by means of a set of Fourier descriptors.^[17] First approaches on Fourier series considered the modeling of 2D contours of fruit and other spherical produce by converting 2D Cartesian contours to 1D polar functions – resembling a radial signal as a function of a phase angle.^[18] Additionally, variations of Fourier series expansion had been developed for the description of 2D contours using elliptical approaches.^[19] More recently, complex geometrical features have been described by incorporating a 2D Fourier series expansion, where the complete geometry of apples and pears was characterized as 2D surface in the spherical plane, thus aiding a dimensional reduction with enhanced mathematical and computational efficiency.^[20] Such approach appears promising in apple size estimation based on fruit 3D point clouds.

The objectives of this study were to develop and validate a geometric modeling approach to estimate apple size by means of LiDAR derived 3D point clouds. Thereof, an automated algorithm to approximate a segmented fruit point cloud to a reference shape geometry based on a Fourier series expansion is proposed. The efficiency of the proposed model was evaluated in terms of precision on apple size estimation and solving time. The proposed model was compared to standard Euclidean distance calculation.

Material and methods

Data acquisition and pre-processing

3D point cloud acquisition

All point clouds were obtained by means of a mobile 2D LiDAR sensor system (LMS-511, Sick AG, Waldkirch, Germany) emitting at wavelength of 905 nm with angular resolution of 0.1667° and 25 Hz scanning frequency. This sensor system captured geometric and radiometric information of the scanned environment. Point clouds were analyzed considering ideal (spheres) and complex (apples) spherical objects, each at laboratory and field conditions. For the monitoring in the laboratory, the LiDAR sensor was mounted on a rigid linear tooth belt conveyor of 2 m length (Module 115/42, IEF Werner, Germany) on an aluminum frame at 1.2 m height. The linear conveyor was adjusted to move the sensor horizontally at a speed of 20 mm s⁻¹, with distance between LiDAR and object of 1.0 m (Figure 1a). Each point cloud was manually segmented using CloudCompare (2.10, GPL software, Paris, France) from the raw point cloud of the LiDAR scan (Figure 1b).

Figure 1. a) Experimental set-up for scanning apples in indoor conditions, b) Raw point cloud of scanned apples in indoor conditions.

For the monitoring in field conditions, the measurements took place in the experimental apple orchard (Field Lab for Digital Agriculture) located in Potsdam-Marquardt, Germany, at the Leibniz Institute for Agricultural Engineering and Bioeconomy (ATB). The LiDAR sensor system was mounted on a circular conveyor that moves the sensors around a tree row (Figure 3a). The circular conveyor was made of a stainless-steel chain system with mechanical suspension according to the plant sensors configuration, and connected to an electrical engine with a working frequency of 50 Hz (DRN71, SEW Eurodrive, Germany). The moving speed was 10 mm s⁻¹ and apparent distance between the LiDAR and the middle of trees row was 2.0 m.

As reference, the size of spheres and apples was measured manually by averaging the diameter at two perpendicular points along the equatorial length. This was done either in the laboratory or directly in the field with an electronic caliper (Type 1108, INSIZE, Suzhou, China).

Point cloud processing of LiDAR scan in laboratory conditions

The size estimation based on 3D point clouds was assessed in laboratory conditions providing same lighting conditions, temperature and relative humidity during the measurements. Ideal shaped spheres of 60 and 80 mm size (n = 3) were scanned individually. Additionally, apples were scanned at 60, 74, 90, 104, 132, and 151 days after full bloom (DAFB) (n = 10) (Figure 2).

Figure 2. Randomly selected apple point cloud for each measuring date, recorded in the laboratory.

Point cloud processing of LiDAR scanning in field conditions

Twenty spheres of 60 and 80 mm size evenly distributed on a metallic frame were located within the tree rows in the experimental field. In addition to the spheres, trees were scanned on their right and left side. The effect of apple location in relation to the position of LiDAR sensor was assessed by studying apples grown at three tree heights with respect to the central moving line of the LiDAR sensor (Figure 3b). Apples were scanned consistently at five dates during fruit development: 67, 81, 117, 132 and 136 DAFB. As a result, 72 apples were measured, identified, and segmented for this study.

Figure 3. a) Field experimental sensor frame set-up, b) Outline of one tree with apples marked (red) and tree simulator with spheres of small (green) and large (orange) size. Heights of in-field scanned objects are marked.

General description of geometric model

Each point cloud was expressed in the Cartesian space as $\mathit{P}\left(x_i,y_i,z_i\right)$. Thereof, the LiDAR sensor measures as it moves at constant speed in front to the scanned object – along the $\mathit{x}$ direction – while it captures at each step a 2D slice of points of depth and height of the object in $\mathit{y}$ and $\mathit{z}$ direction, respectively (Figure 4). For each point cloud $\mathit{P}$ centered on the origin, a transformation into spherical coordinates was performed. Each point was thus described as a vector with an angular position in the spherical plane as $\mathit{R}\left({\theta }_i,{\varphi }_i\right)$, where $\mathit{\theta }$ and $\mathit{\varphi }$ are the angles of azimuth and elevation, respectively. The elevation angle $\mathit{\varphi }$ starts either from the top of a sphere or stem in the case of apples to its calix, from $\mathit{\varphi }=0$ to $\mathit{\varphi }=\mathit{\pi }$, respectively. The angle of azimuth $\mathit{\theta }$ has a period of $2\mathit{\pi }$, starting and ending at the back of the point cloud, thus locating the measured face of the objects in the middle of the spherical plane, where $\mathit{\theta }=\mathit{\pi }$.

Figure 4. Diagram of an ideal sphere and its corresponding localization in Cartesian and spherical space in relation to the LiDAR position and measuring configuration.

The concept of minimizing the error between the point cloud ($\mathit{R}$) and a reference shape ($R^{\mathit{ref}}$) is given by a projection of the point cloud using spherical coordinates ($R^{\mathit{fourier}}$). Thereof, the direction in which each point cloud is projected is given by an Eigen analysis, in which a given eigenvector captures most of the object curvature given by its variance ($max{\frac{n}{e}}_{\mathit{y}}$). However, typical LiDAR derived point clouds capture geometrical outliers due to reflected beams at curved edges at the objects surface. Moreover, the presence of outliers within the point cloud lowers the sphericity of the local centroid of the point cloud, thus obscuring the quality of the eigenvector ${\frac{n}{e}}_{\mathit{y}}$, in detriment of an accurate approximation procedure.^[21] Therefore, an iterative Gaussian filter was applied on the initial point cloud in the spherical space ($\mathit{R}$) using the reference Fourier shape ($R^{\mathit{fourier}}$). By using an interquartile range approach, all points laying above the upper tail of the error of all points with respect to the Fourier reference shape were deleted.

Each point cloud was centered on the origin and the Gaussian filter performed again in order to look for newly detected outliers after centering, until no new outlier was detected. Subsequently, by applying an iterative step algorithm based on the direction of the eigenvector of the Fourier shape projection ${\frac{n}{e}}_{\mathit{y}}$, a given point cloud is therefore approximated to its reference shape ($R^{\mathit{ref}}$) – towards the space location where the error between the point cloud and its reference shape is minimum ($R^{\mathit{opt}}$) (Figure 5). Finally, each approximated point cloud has an approximated radius with respect to its centroid, aiding fruit size estimation by means of a projected radius.

Figure 5. Flow diagram of the geometric model algorithm.

Fourier projection using spherical reference shape

Firstly, a 2D reference contour in Cartesian coordinates was estimated for each point cloud to obtain a reference 2D surface shape of the object in spherical coordinates. An ideal 3D point cloud of a sphere was generated in Blender (Blender Foundation, 2017), which represents the reference shape of spheres of 60 and 80 mm diameter in the present work. As for the apples, the 2D Cartesian contour for each point cloud in laboratory conditions was used as a reference to calculate an individual 2D Fourier reference shape ($R^{\mathit{fourier}}$) for each measuring date (DAFB). An Alphashape algorithm was used to obtain the outer contour of each point cloud,^[22] where an alpha value of 0.5 was arbitrary selected after image exploration (Figure 6a). All obtained edge contours were transformed to polar coordinates of the form $\mathit{R}\left({\varphi }^{\prime }\right)$, where ${\varphi }^{\prime }$ is the rotation angle with a period of $2\mathit{\pi }$ starting at the top of each point cloud (Figure 6b and equation 1(1) $\mathit{R}\left({\varphi }^{\prime }\right)=A_0+\sum _{\mathit{i}=1}^k\left[A_{k}cos\left(\mathit{k}{\varphi }^{\prime }\right)+B_{k}sin\left(\mathit{k}{\varphi }^{\prime }\right)\right]$(1) ). The approximation of $\mathit{R}\left({\varphi }^{\prime }\right)$ was performed using a 1D Fourier series expansion (equation 1(1) $\mathit{R}\left({\varphi }^{\prime }\right)=A_0+\sum _{\mathit{i}=1}^k\left[A_{k}cos\left(\mathit{k}{\varphi }^{\prime }\right)+B_{k}sin\left(\mathit{k}{\varphi }^{\prime }\right)\right]$(1) ).

Figure 6. Fourier reference shape calculation of an example apple. a) 2D Cartesian reference point cloud, b) 1D Fourier series expansion of angular transformation of reference contour, and c) Interpolated 2D reference shape of this apple in spherical space.

(1) $\mathit{R}\left({\varphi }^{\prime }\right)=A_0+\sum _{\mathit{i}=1}^k\left[A_{k}cos\left(\mathit{k}{\varphi }^{\prime }\right)+B_{k}sin\left(\mathit{k}{\varphi }^{\prime }\right)\right]$(1)

where the Fourier coefficients $A_0$, $A_k$, and $B_k$ are characteristical of the size and shape of object as a 1D period within the 0 –$2\mathit{\pi }$ range. As sin and cosine functions are orthogonal in the linear space, the Fourier coefficients were calculated (equations 2(2) $A_0=\frac{1}{2\mathit{\pi }}\underset{0}{\overset{2\mathit{\pi }}{\int }}\mathit{R}\left(\mathit{\varphi }{ }^{\mathrm{\prime }}\right)\mathit{d\varphi }{ }^{\mathrm{\prime }}$(2) –4(4) $B_k=\frac{1}{\mathit{\pi }}\underset{0}{\overset{2\mathit{\pi }}{\int }}\mathit{R}\left(\mathit{\varphi }{ }^{\mathrm{\prime }}\right)sin\left(\mathit{k\varphi }{ }^{\mathrm{\prime }}\right)\mathit{d\varphi }{ }^{\mathrm{\prime }}$(4) ). The resulting set of Fourier coefficients were obtained for the k number of harmonics of the Fourier expansion. For apples at all measuring dates, an arbitrary number of harmonics were used (k = 7) for all studied point clouds.

(2) $A_0=\frac{1}{2\mathit{\pi }}\underset{0}{\overset{2\mathit{\pi }}{\int }}\mathit{R}\left(\mathit{\varphi }{ }^{\mathrm{\prime }}\right)\mathit{d\varphi }{ }^{\mathrm{\prime }}$(2)

(3) $A_k=\frac{1}{\mathit{\pi }}\underset{0}{\overset{2\mathit{\pi }}{\int }}\mathit{R}\left(\mathit{\varphi }{ }^{\mathrm{\prime }}\right)cos\left(\mathit{k\varphi }{ }^{\mathrm{\prime }}\right)\mathit{d\varphi }{ }^{\mathrm{\prime }}$(3)

(4) $B_k=\frac{1}{\mathit{\pi }}\underset{0}{\overset{2\mathit{\pi }}{\int }}\mathit{R}\left(\mathit{\varphi }{ }^{\mathrm{\prime }}\right)sin\left(\mathit{k\varphi }{ }^{\mathrm{\prime }}\right)\mathit{d\varphi }{ }^{\mathrm{\prime }}$(4)

As this contour line represents two lines within the spherical domain (at $\mathit{\theta }=\mathit{\pi }/2$ and $\mathit{\theta }=3\mathit{\pi }/2$) two segments of the 1D Fourier shape were obtained, for $\mathit{\varphi }{ }^{\mathrm{\prime }}\le \mathit{\pi }$ and $\mathit{\varphi }{ }^{\mathrm{\prime }}>\mathit{\pi }$, respectively (Figure 6b). Additionally, an interpolation at $\mathit{\theta }=\mathit{\pi }$ and $\mathit{\theta }=0$ was performed, allowing for a signature shape that is quasi-symmetric considering its reference contour. Finally, a linear interpolation algorithm was used to obtain a 2D reference shape of the form $R^{\mathit{ref}}\left(\mathit{\theta },\mathit{\varphi }\right)$ (Figure 6c).

Typically, a Fourier series expansion is applied over a function that is continuous at the surface boundary. Any discontinuity is reflected in large signal peaks at the boundary, also known as Gibbs phenomena.^[23] Thus, the interpolated 2D surface must relate to the offset of a typical 1D Fourier series, rather to an individual 2D shape. Moreover, a given 2D Fourier surface is then described by a 1D (reference contour) and 2D (offset respect to the 1D contour) Fourier series. A 1D contour at $\mathit{\theta }=0$ was used as reference for all studied point clouds. The offset shape was described as 2D Fourier series expansion (equation 5(5) $\mathit{R}\left(\mathit{\theta },\mathit{\varphi }\right)=\sum _{\mathit{n}=1}^{\mathrm{\infty }}\sum _{\mathit{m}=1}^{\mathrm{\infty }}{C}_{\mathit{n},\mathit{m}}sin\left(\frac{\mathit{n\pi \theta }}{2\mathit{\pi }}\right)sin\left(\frac{\mathit{m\pi \varphi }}{\pi }\right)$(5) ).

(5) $\mathit{R}\left(\mathit{\theta },\mathit{\varphi }\right)=\sum _{\mathit{n}=1}^{\mathrm{\infty }}\sum _{\mathit{m}=1}^{\mathrm{\infty }}{C}_{\mathit{n},\mathit{m}}sin\left(\frac{\mathit{n\pi \theta }}{2\mathit{\pi }}\right)sin\left(\frac{\mathit{m\pi \varphi }}{\pi }\right)$(5)

Similarly by orthogonality, the Fourier coefficients were calculated (equation 6(6) $C_{\mathit{n},\mathit{m}}=\frac{4}{2{\pi }^2}\underset{0}{\overset{\mathit{\pi }2}{\int }}\underset{0}{\overset{\mathit{\pi }}{\int }}\mathit{R}\left(\mathit{\theta },\mathit{\varphi }\right)sin\left(\frac{\mathit{n\pi \theta }}{2\mathit{\pi }}\right)sin\left(\frac{\mathit{m\pi \varphi }}{\pi }\right)\mathit{d\theta d\varphi }$(6) ).

(6) $C_{\mathit{n},\mathit{m}}=\frac{4}{2{\pi }^2}\underset{0}{\overset{\mathit{\pi }2}{\int }}\underset{0}{\overset{\mathit{\pi }}{\int }}\mathit{R}\left(\mathit{\theta },\mathit{\varphi }\right)sin\left(\frac{\mathit{n\pi \theta }}{2\mathit{\pi }}\right)sin\left(\frac{\mathit{m\pi \varphi }}{\pi }\right)\mathit{d\theta d\varphi }$(6)

Where $C_{\mathit{n},\mathit{m}}$ are the Fourier descriptors for the harmonics $\mathit{n}$ and $\mathit{m}$. An arbitrary number of 10 harmonics were chosen for this analysis.

By having a Fourier signature shape ($R^{\mathit{fourier}})$ determined by a set of 1D and 2D Fourier coefficients within a continuous domain, a projection in the spherical space for each point cloud of spheres and apples was performed. The error associated to the reference Fourier projection was calculated (equation 7(7) $\mathit{Erro}{r}_{\mathit{ref}}=R^{\mathit{cloud}}\left({\theta }_i,{\varphi }_j\right)-R^{\mathit{fourier}}\left({\theta }_i,{\varphi }_j\right)$(7) ).

(7) $\mathit{Erro}{r}_{\mathit{ref}}=R^{\mathit{cloud}}\left({\theta }_i,{\varphi }_j\right)-R^{\mathit{fourier}}\left({\theta }_i,{\varphi }_j\right)$(7)

Thus, this error describes the difference between a point cloud and its own reference projection using 1D and 2D Fourier series expansion. Ultimately, such reference function serves as the spatial location, which an actual point cloud should fit. Furthermore, by minimizing this error function, an estimation of a reference ideal object in space can be somewhat mimicked, and the apple point clouds were approximated without losing the specific shape features at each measuring day during fruit development.

Eigen analysis and Fourier approximation algorithm

Once $R^{\mathit{ref}}$ was calculated, in order to approximate the individual point cloud, it is necessary to determine the direction onto which the point cloud shall be projected. Thereof, a singular value decomposition (SVD) algorithm was performed on the covariance tensor of the projection of each point cloud in the Cartesian space ($P^{\mathit{fourier}})$. As previously indicated (Figure 4), the LiDAR sensor moves along the $\mathit{x}$ direction, capturing at each step geometric information on the width and height (given by the $\mathit{z}$ direction) of the scanned object. Thus, each point cloud was projected towards its depth, given by the eigenvector that contains the major absolute magnitude in the $\mathit{y}$ axis, ${{\frac{n}{e}}_{\mathit{y}}}^{\mathit{fourier}}$ (Figure 7). Having a projected direction of approximation, each point cloud of fruit was translated towards its projected shape given by its Fourier series expansion.

Figure 7. Eigenvector ${{\frac{n}{e}}_{\mathit{y}}}^{\mathit{fourier}}$ of the Fourier projection (blue) of a point cloud (grey) for a) random sphere of 80 mm size, and b) random apple at 151 DAFB.

An iterative error minimization algorithm was performed over the cloud projection, defined by a step function (equation 8(8) $P_{\mathit{adj}}=\mathit{P}\pm {{\frac{n}{e}}_{\mathit{y}}}^{\mathit{fourier}}\mathit{\delta }$(8) ).

(8) $P_{\mathit{adj}}=\mathit{P}\pm {{\frac{n}{e}}_{\mathit{y}}}^{\mathit{fourier}}\mathit{\delta }$(8)

where at each step, a given point cloud $\mathit{P}$ is approximated by a step coefficient $\mathit{\delta }$ into the direction given by ${{\frac{n}{e}}_{\mathit{y}}}^{\mathit{fourier}}$. At each iteration, the Fourier projection is recalculated, thus obtaining a new value of ${{\frac{n}{e}}_{\mathit{y}}}^{\mathit{fourier}}$, accounting for the previous iteration and new position of the point cloud in space (Figure 5). Ultimately, the overall point cloud error reflected by the root mean squared error with respect to $R^{\mathit{fourier}}$ ($\mathit{RMS}{E}_{\mathit{ref}}$) can thus be minimized. Moreover, when the iteration steps go beyond the minimum root found in the iteration, the $\mathit{RMS}{E}_{\mathit{ref}}$ of point cloud would increase as the point cloud is translated beyond its reference shape (Figure 8).

Figure 8. Diagram of a point cloud optimization results considering a randomly selected apple measured at 151 DAFB in the laboratory: a) $\mathit{RMS}{E}_{\mathit{ref}}$ curve, b) initital point cloud underestimating the curvature, c) optimal point cloud, d) increased, over-estimating error as a result of projecting the point cloud beyond its reference Fourier shape. Three marked dots in a.) are related to point cloud appearance shown in b), c) and d).

Model performance, comparison and validation

The computational efficiency of the model was assessed by comparing the processing time of the approximation algorithm for each processed point cloud considering the point cloud density. The model was entirely coded in Python (64-bit version 3.9.7) using a processor Intel(R) Core(TM) i3–2100 CPU @ 3.10 GHz with 8 GB RAM.

Each point cloud was evaluated by merits of its root mean square error compared to its reference Fourier shape ($\mathit{RMS}{E}_{\mathit{ref}}$) and the empirically measured object radius ($\mathit{RMS}{E}_{\mathit{exp}}$). The RMSE of each point cloud to its reference 2D Fourier signature was calculated (equation 9(9) $\mathit{RMS}{E}_{\mathit{ref}}=\sqrt{\frac{{\sum }_{i=1}^n{\left(R_i-R_i^{f\mathrm{ourier}}\right)}^2}{\mathit{n}}}$(9) ).

(9) $\mathit{RMS}{E}_{\mathit{ref}}=\sqrt{\frac{{\sum }_{i=1}^n{\left(R_i-R_i^{f\mathrm{ourier}}\right)}^2}{\mathit{n}}}$(9)

where the radius of each point $R_i$ (mm) is contrasted to its own projection using the Fourier shape signature $R_i^{fo\mathrm{urier}}$ (mm) at the same location $\mathit{\theta }$ and $\mathit{\varphi }$, for all total amount of points of the point cloud ($\mathit{n}$). Considering the optimized point cloud, the assessment of the model performance considering the measured size of apple was performed by calculating an averaged radius of the projected apple using its 3D point cloud. For the laboratory scanned apples, the $\mathit{n}$ points located within the boundary of $\mathit{\varphi }=\mathit{\pi }/2$ $\pm 5\mathrm{\%}$ were considered for apple radius estimation. As for the field scanned apples, $R_i$ (mm) of all points of the point clouds ($\mathit{n}$) were considered. The $\mathit{RMS}{E}_{\mathit{exp}}$ with respect to the measured apple radius was calculated (equation 10(10) $\mathit{RMS}{E}_{\mathit{exp}}=\sqrt{\frac{{\sum }_{i=1}^n{\left(R_i-R^{\mathit{exp}}\right)}^2}{\mathit{n}}}$(10) ).

(10) $\mathit{RMS}{E}_{\mathit{exp}}=\sqrt{\frac{{\sum }_{i=1}^n{\left(R_i-R^{\mathit{exp}}\right)}^2}{\mathit{n}}}$(10)

where the radius of all $\mathit{n}$ points of the point cloud $R_i$ (mm) and the manually recorded radius of the apple $R^{\mathit{exp}}$ (mm) were considered. The shape of apple changed during fruit development and as the measuring dates in the laboratory and field readings were different, a spline interpolation algorithm was performed on all set of Fourier coefficients obtained from the laboratory data, and subsequently, the interpolated coefficients for each growing period studied in the field were estimated. Thereof, the geometric model was applied on collected field data using indoor obtained Fourier reference shapes. By using the indoor Fourier coefficients as reference shape, it was ensured to have close to real geometry of the scanned object, avoiding the presence of occlusions due to other tree parts or angular inclinations due to stem location in the tree structure.

Considering the field data, all fruit data are available open access already segmented as fruit point clouds. However, as a source of measuring uncertainty, the occlusion problem in the canopy can affect the quality of the segmented point clouds, where fragments of points belonging to curve-shaped leaves and branches can be included within the occluded apple, in detriment of the segmented point cloud and model performance. Therefore, a filter was applied to the population of segmented apple point clouds, where an Eigen decomposition of the covariance tensor was performed and their geometric features such as linearity (Ln) and curvature (C) were calculated.^[13,24] Prior to filtering, all data of the geometric features from all measured growing periods were standardized by setting their mean value at zero and subsequently scaling to unit variance. Thus, point clouds that presented the conditions $\mathit{Ln}<1$ and $\mathit{C}<-1$ were eliminated.

The geometric model performance was assessed by comparison to two standard 2D approaches; the mean and maximum Euclidean distances of each point cloud along the $\mathit{x}$ and $\mathit{z}$ axis – width and height, respectively. Moreover, the performance of each model was evaluated over the entire time series of apples after filtering, accounting for predicted and expected, measured values. The model comparison was performed by means of a linear regression between predicted and measured data – for both laboratory and field conditions. Values of coefficient of determination ($R^2$), root mean squared error ($\mathit{RMS}{E}_{\mathit{exp}}$) and mean bias error ($\mathit{MB}{E}_{\mathit{exp}}$) were calculated for each model regression using the sklearn Python library.

Results and discussion

Computational performance

The model performance by means of computational efficiency was estimated for each analyzed point cloud (Figure 9). The solving time to reach its minimum $\mathit{RMS}{E}_{\mathit{ref}}$ was found to be directly linked to the density of each point cloud. Thereof, as the angular distance between the LiDAR and the scanned objects is minor in the laboratory conditions, the corresponding point clouds were consistently denser. As a consequence, larger solving times were found in the laboratory, ranging from 10 to >100 seconds. On the contrary, the objects scanned in the fields were solved at a considerably lesser time, ranging from 6·10⁻³ up to 6 seconds. At larger distance between the LiDAR and the scanned objects, the angular distance between the beams of the LiDAR increase, thus diminishing the point cloud density of the segmented object. Additionally, the occlusion by other tree parts diminished the scanned region in the case of apples. The decision on where to place the LiDAR in an integrated scanning system is thus directly linked to the speed of the solving algorithm. However, such decision must be considered with regard to the overall performance of the model with respect to the accuracy of size estimation of the horticultural produce.

Figure 9. Computational efficiency of the geometric algorithm for the scanned spheres in laboratory (n = 6) and field conditions (n = 20), and scanned apples in laboratory (n = 60) and field conditions (n = 283).

Centroid approximation of LiDAR point clouds in laboratory conditions

The approximated optimal point clouds were calculated in Spherical and Cartesian coordinates (Figure 11). Changes in the 2D surface shape of each point cloud before and after approximation were observed for all scanned spheres (Figure 10), where each point approached to its reference shape at minimum$\mathit{RMS}{E}_{\mathit{ref}}$. Additionally, estimated scanned region of the spheres in the Cartesian space were identified (Figure 11). Resulting average $\mathit{RMS}{E}_{\mathit{ref}}$ values of 6.09 and 3.77 mm were found for the spheres of 60 and 80 mm, respectively. Minor overestimation of the approximated point clouds with respect to the Fourier shape of $\mathit{MB}{E}_{\mathit{ref}}$ = 1.27 and 0.56 mm were obtained, respectively (Table 1). Moreover, the model showed to be sensible to potential undetected outliers, where each point within the point cloud is accounted for within the model, increasing its sensibility to error given by location in space. Thereof, areas of error beyond 15 mm were observed (Figure 11c), thus increasing the overall error. Additionally to the model performance, differences in relative $\mathit{RMS}{E}_{\mathit{exp}}$ up to 1.96 and 5.27% were found when comparing the geometric model to the mean and maximum Euclidean distance approaches, respectively.

Figure 10. Optimal point cloud of indoor-measured spheres of 60 (a-c) and 80 (d-f) mm in spherical coordinates. Point clouds before approximation are shown in grey, while error is given in false colour scale.

Figure 11. Optimal point cloud of spheres of 60 (a-c) and 80 (d-f) mm in Cartesian coordinates, where error is provided in false colour scale.

Table 1. Averaged values of root mean squared error (RMSE) and mean bias error (MBE) of point clouds measured in laboratory conditions.

	Geometric model				Mean Euclidean distance		Maximum Euclidean distance
	$\mathit{MB}{E}_{\mathit{ref}}\left(\mathit{mm}\right)$	$\mathit{RMS}{E}_{\mathit{ref}}\left(\mathit{mm}\right)$	$\mathit{MB}{E}_{\mathit{exp}}\left(\mathrm{\%}\right)$	$\mathit{RMS}{E}_{\mathit{exp}}\left(\mathrm{\%}\right)$	$\mathit{MBE}\left(\mathrm{\%}\right)$	$\mathit{RMSE}\left(\mathrm{\%}\right)$	$\mathit{MBE}\left(\mathrm{\%}\right)$	$\mathit{RMSE}\left(\mathrm{\%}\right)$
Spheres 60 mm	1.27	6.09	4.24	20.28	10.51	22.24	12.26	23.85
Spheres 80 mm	0.56	3.77	1.41	9.43	6.46	8.08	14.67	14.70
$\mathit{DAF}{B}_{60}$	−0.54	7.33	34.64	43.51	99.30	99.30	153.16	153.16
$\mathit{DAF}{B}_{74}$	−0.13	4.22	−0.07	16.21	17.45	18.75	29.48	29.48
$\mathit{DAF}{B}_{90}$	−0.91	4.67	9.94	18.73	31.22	31.22	38.74	38.74
$\mathit{DAF}{B}_{104}$	−0.55	4.75	−5.49	15.04	14.17	14.61	22.68	22.68
$\mathit{DAF}{B}_{132}$	−0.80	4.78	−9.09	13.97	4.05	5.84	8.56	8.56
$\mathit{DAF}{B}_{151}$	−0.73	4.99	−16.51	19.63	−5.07	9.83	−1.46	10.67

The overall performance of all size estimation approaches by means of their $\mathit{RMS}{E}_{\mathit{exp}}$ was found higher for the spheres of 60 mm, in comparison to the spheres of 80 mm size. Additionally, average values of $\mathit{MB}{E}_{\mathit{exp}}$ of 4.24 and 1.41 % were found for the spheres of 60 and 80 mm, respectively. The Euclidean distance methods showed a $\mathit{MB}{E}_{\mathit{exp}}$ up to 10.51 and 14.57 % for mean and maximum Euclidean distances, respectively. As the laser hits an object, it scatters its beams according to the optical and geometrical properties of the object – such as size – thus influencing the overall error magnitude and bias of the standard approaches. The sensor configuration used in this study showed a high uncertainty considering the depth data as well as along the scanning trajectory. Resulting variation of sensor speed and distance to objectives can lead to low-quality point cloud density, in detriment of accurate size estimation.^[12,25]

The model performance and validation were benchmarked against mean and maximum Euclidean distance methods assessed for each measuring date during fruit growth of apples (Table 1). Values of $\mathit{RMS}{E}_{\mathit{ref}}$ of the geometric model ranged from 7.33 to 4.22 mm for all studied time series of apples. Additionally, average $\mathit{MB}{E}_{\mathit{ref}}$ values of −0.61 mm and $\mathit{RMS}{E}_{\mathit{ref}}$ of 5.12 mm were found between the approximated point clouds compared to their Fourier reference shapes, assessing the accuracy of the geometric model approximation procedure during fruit development. Highest $\mathit{RMS}{E}_{\mathit{ref}}$ values were observed at $\mathit{DAF}{B}_{60}$. Largest source of error was generally located near the stem region of the apples, where an accurate estimation of a Fourier reference is difficult due to potential apple angle of inclination and resulting asymmetry in the hanged apples. Furthermore, as the Fourier reference shape is based on the outer 2D contour extraction, it does not account for the actual shape variability of the point cloud as the apple is rotated.

In comparison, the geometric model accuracy was higher than the Euclidian distances methods. Average $\mathit{RMS}{E}_{\mathit{exp}}$ values of 21.2, 29.8 and 43.9 % were obtained for the geometric model, and the mean and maximum Euclidean distance, respectively when predicting the size of apples. Overall, all approaches had their lowest performance at the earliest measured growing stage ($\mathit{DAF}{B}_{60}$), due to a high point cloud density of scattered beams, affecting the model accuracy on small objects – shown also in the $\mathit{RMS}{E}_{\mathit{ref}}$ values of the 60 mm spheres (Table 1). However, in these conditions the Fourier-based approach showed tremendously lower errors, being able to account for the shape of the fruit at different growing periods (Figure 12). Similar approaches for apple size estimation in laboratory conditions were observed earlier,^[3] where $\mathit{RMSE}$ values of 2.47 and 2.74 mm were found when fitting a sphere of known size and a reconstructed apple shape template using structure from motion approach (SfM), respectively.

Figure 12. Averaged reference Fourier shapes for all monitored dates during fruit development, a) 1D Fourier shape in Cartesian coordinates, b) Interpolated 2D Fourier reference shape in spherical coordinates.

Centroid approximation of LiDAR point clouds in field conditions

The interpolation of previously obtained Fourier coefficients successfully aided the geometric model computation on field obtained data, as indicated by their $\mathit{RMS}{E}_{\mathit{ref}}$, for all measuring dates during fruit development (Table 2). Additionally, the error of each point cloud with respect to its Fourier reference shape before and after approximation was assessed for all studied fruit growth stages and vertical locations within the tree canopy (Figure 13). Average $\mathit{RMS}{E}_{\mathit{ref}}$ of 6.43, 6.35 and 7.41 mm were found for the apples located within the regions $\mathrm{\Delta }{H}_1$, $\mathrm{\Delta }{H}_2$ and $\mathrm{\Delta }{H}_3$, respectively for all the measured dates during fruit development. Higher average values of $\mathit{RMS}{E}_{\mathit{ref}}$ were found at $\mathit{DAF}{B}_{67}$, in comparison with the rest of the monitored growing stages. All other measured dates after $\mathit{DAF}{B}_{67}$ showed increased error at enhanced angular distance between the LiDAR and the location of the apples ($\mathrm{\Delta }{H}_1$ and $\mathrm{\Delta }{H}_3$). Moreover, at these locations the appearance of outliers notably increased in comparison with the apples located at $\mathrm{\Delta }{H}_2$. Additionally, values of $\mathit{MB}{E}_{\mathit{ref}}$ decreased from minor over estimation to under estimation between $\mathit{DAF}{B}_{67}$ and $\mathit{DAF}{B}_{136}$. Similarly, previous work on apple size estimation found the largest RMSE values ranging from 7.7 to 14.5 % at the most distant locations considering the LiDAR position in defoliated trees, where point clouds appear less dense due to the angular beam trajectory.^[13]

Figure 13. Performance of model approximation and error associated to the Euclidean distance calculation capturing all fruit from three canopy heights with respect to the LiDAR. Boxplots represent the point clouds before (red) and after approximation (blue). Maximum (green) and mean (blue) Euclidean distance calculations are represented by scattered points for: a) metal tree with spheres, b) 67 DAFB, c) 81 DAFB, d) 117 DAFB, e) 132 DAFB, and f) 136 DAFB.

Table 2. Averaged reference values of root mean squared error ($\mathit{RMS}{E}_{\mathit{ref}}$), mean bias error ($\mathit{MB}{E}_{\mathit{ref}}$) and number of samples ($\mathit{n}$) at all vertical locations within scanned trees.

	$\mathit{n}$		$\mathit{DAF}{B}_{67}$	$\mathit{DAF}{B}_{81}$	$\mathit{DAF}{B}_{117}$	$\mathit{DAF}{B}_{132}$	$\mathit{DAF}{B}_{136}$
$\mathrm{\Delta }{H}_1$	90	$\mathit{RMS}{E}_{\mathit{ref}}\left(\mathit{mm}\right)$	7.00	7.28	4.91	6.44	6.55
		$\mathit{MB}{E}_{\mathit{ref}}\left(\mathit{mm}\right)$	3.54	2.15	−0.58	−1.23	−1.10
$\mathrm{\Delta }{H}_2$	81	$\mathit{RMS}{E}_{\mathit{ref}}\left(\mathit{mm}\right)$	8.28	5.11	5.00	7.40	5.97
		$\mathit{MB}{E}_{\mathit{ref}}\left(\mathit{mm}\right)$	1.40	−0.42	−0.48	0.30	−1.35
$\mathrm{\Delta }{H}_3$	112	$\mathit{RMS}{E}_{\mathit{ref}}\left(\mathit{mm}\right)$	7.89	5.70	8.34	7.75	7.41
		$\mathit{MB}{E}_{\mathit{ref}}\left(\mathit{mm}\right)$	0.68	−0.73	1.91	0.36	−0.48

The geometric model validation and comparison on field collected data were calculated for the metal tree and for each time series separately (Table 3). For the metal tree, differences up to 2.97 % in $\mathit{RMS}{E}_{\mathit{exp}}$ were found between all the studied approaches. Furthermore, the geometric model presented considerable minor values of $\mathit{MB}{E}_{\mathit{exp}}$, in comparison with the standard Euclidean distance approaches. Additionally, the spheres mounted on the metal tree showed minor error in comparison to the apples. The known size of spheres, the uniform spherical shape and the absence of leaves increased the overall accuracy of the geometric model, compared to the apples with more complex shapes and occurrence of occlusion within the canopy, affecting the capture of geometric features by the LiDAR sensor. Thereof,^[3] obtained coefficients of determination up to 0.81, 0.91 and 0.95 for apples with visibility > 20, > 40 and > 70 %, respectively using a RGB-D camera.

Table 3. Field application of the geometrical model and comparison by means of $\mathit{RMS}{E}_{\mathit{exp}}$ (%) and $\mathit{MB}{E}_{\mathit{exp}}$ (%) for spheres (n = 20) and apple measuring dates (n = 283).

		Spheres	$\mathit{DAF}{B}_{67}$	$\mathit{DAF}{B}_{81}$	$\mathit{DAF}{B}_{117}$	$\mathit{DAF}{B}_{132}$	$\mathit{DAF}{B}_{136}$
Geometric model	$\mathit{RMS}{E}_{\mathit{exp}}\left(\mathrm{\%}\right)$	15.06	33.18	15.50	19.44	17.69	14.55
	$\mathit{MBE}\left(\mathrm{\%}\right)$	6.4·10^−3	33.00	14.24	18.94	16.94	13.24
Mean Euclidean distance	$\mathit{RMS}{E}_{\mathit{exp}}\left(\mathrm{\%}\right)$	15.05	27.84	25.23	168.60	29.57	132.81
	$\mathit{MBE}\left(\mathrm{\%}\right)$	−7.53	17.03	0.56	136.95	−10.52	95.66
Max Euclidean distance	$\mathit{RMS}{E}_{\mathit{exp}}\left(\mathrm{\%}\right)$	18.02	43.42	36.06	270.70	36.10	209.04
	$\mathit{MBE}\left(\mathrm{\%}\right)$	2.82	37.50	20.24	247.47	4.24	182.52

For the apples monitored at different dates during fruit development, the geometric model performed consistently better than the Euclidean distance methods. Average $\mathit{RMS}{E}_{\mathit{exp}}$ values of 20.1, 76.8 and 119.1 % were found for the model, the mean and maximum Euclidean distances, respectively. Similarly, averaged $\mathit{MB}{E}_{\mathit{exp}}$ values of 19.3, 47.9 and 98.4 % were found for the model, the mean and maximum Euclidean distances, respectively. Additionally, $\mathit{RMS}{E}_{\mathit{exp}}$ and $\mathit{MB}{E}_{\mathit{exp}}$ of 33.2 and 33.0 % were found at $\mathit{DAF}{B}_{67}$, respectively for the geometric model. Generally, large errors were reported using the mean and maximum Euclidean distances approach, where both $\mathit{RMS}{E}_{\mathit{exp}}$ and $\mathit{MB}{E}_{\mathit{exp}}$ values surpassed the 100 % at various growing stages (Table 3). Given that these methods are based on distances between extreme points in a reference plane, the potential occlusion of the apples – which is typically the case in field conditions – affected the size prediction. Thereof, the geometric model uses previously obtained Fourier coefficients, which partially accounts for the occlusion problem by giving a more accurate reference shape in which the point clouds are projected. However, the error of the geometric model increased at the earliest monitored date, where occluded apples are hard to approximate due to a more complex shape towards the stem of the produce (Figure 12). On the other hand,^[2] achieved coefficients of determination from 0.68 to 0.70 using thermal imaging, and^[26] obtained coefficient of determination of 0.95 using RGB-D imaging. However, these are limited to specific temperature and lighting conditions, as well as being typically a 2D approach, neglecting any in depth shape features.

Time series regression and model performance

Overall, all approaches performed considerably better for laboratory scanned data, in comparison to the field obtained data (Figure 14). The geometric model presented a better coefficient of determination for both laboratory and field conditions, in comparison to the Euclidean distance approaches (Table 4). However, the performance of the model in field conditions, given by its coefficient of determination reached only 0.49. The occlusion of the scanned apples due to the presence of leaves affected the model performance. Additionally, poor point cloud density is an indicator of scarcely visible regions where curvature is not entirely captured. Moreover, the linear behavior of all the studied approaches was indicated by the overall lower values of $\mathit{MBE}$ ($<1\cdot {10}^{-14}$ mm).

Figure 14. Linear regression between calculated and measured data for all growing periods and for all the studied models for: a) Laboratory and b) Field collected data.

Table 4. Goodness of fit comparison between studied approaches using a linear regression for laboratory and field data.

		$R^2$	$\mathit{RMSE}\left(\mathit{mm}\right)$	$\mathit{MBE}\left(\mathit{mm}\right)$
Laboratory	Model	0.76	2.29	2.66·10^−15
	Mean Euclidean Distance	0.44	6.58	−9.47·10^−16
	Maximum Euclidean Distance	0.02	10.36	6.04·10^−15
Field	Model	0.49	4.02	1.80·10^−15
	Mean Euclidean Distance	0.21·10^−2	113.22	−4.01·10^−15
	Maximum Euclidean Distance	0.11·10^−2	192.68	3.22·10^−15

Differences up to 77.9 and 97.1 mm on $\mathit{RMSE}$ between the geometric model and the Euclidean distances approach were found for laboratory and field conditions, respectively. The Euclidean distance methods showed consistently lower performance respect to other proposed models that do not consider largest segments approaches. Similarly, mean absolute percentage error of 23.6 % was found in previous study^[3] when comparing a large segment method – Euclidean-like approach, showing the typically poor performance of such methods in comparison to some more complex direct models.

Future work for the robust application of the approach presented, would be the development of a database for increasing the number of indoor scanned fruit. A larger amount of Fourier coefficients should accumulate different fruit populations at different growing season, capturing fruit shape variability in real-world scenario. In field conditions, more work on data quality should capture various sensors as well as oriented measurements using mobile platform such as short distance linear conveyors for tree scanning.

Conclusion

A geometric modeling approach to predict apple size by means of LiDAR 3D point clouds was developed in python environment. The proposed model presented solving times up to 5.9 and 0.7 seconds for scanned spheres and apples in field conditions, respectively. The applicability of the geometric model in laboratory and field conditions was assessed for scanned spheres of known size and apples at different growing stages during fruit development. Averaged $\mathit{RMS}{E}_{\mathit{exp}}$ of 21.2, 30.0 and 43.9 % were found for the geometric model, mean, and maximum Euclidean distances, respectively. Additionally, the geometric model approximation algorithm performed on 283 segmented apples and 20 spheres in field conditions. Averaged $\mathit{RMS}{E}_{\mathit{exp}}$ of 15.1 and 20.1 % was found for the scanned spheres and apples, respectively. The large error difference between the spheres and the apples was found due to the large presence of occluded apples, which affected the geometric model performance. Moreover, $\mathit{RMS}{E}_{\mathit{exp}}$ of 20.1, 76.8 and 119.1 % were obtained for the geometric model, the mean and maximum Euclidean distance approaches, respectively. Finally, a linear regression between predicted and measured apple radius showed $R^2$ values of 0.76 and 0.49 for the laboratory and field scanned apples, respectively. Overall, the model performed better than the Euclidean distance approach, where $R^2$ values of 0.49, 0.2·10⁻² and 0.1·10⁻² were found for the geometric model, mean and maximum Euclidean distance methods.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The work was supported by the Bangladesh Agricultural Research Institute Bundesministerium für Ernährung und Landwirtschaft. The project SHEET is part of the ERA-NET Cofund ICT-AGRI-FOOD, with funding provided by national sources [BLE] and co-funding by the European Union Horizon 2020 research and innovation program, Grant Agreement number 862665.