3D Modeling of Façade Elements Using Multi-View Images from Mobile Scanning Systems

Abstract

There is a growing demand for detailed building façade models (Level-of-Detail 3: LoD 3) in a variety of applications. Despite the increasing number of papers addressing this issue in the literature, occlusions are still a significant problem when processing building façade elements. Conversely, depending on the view angle of the images, the detected elements might not be projected to their accurate locations causing uncertainties in their 3D locations. In this paper, we address the aforementioned issues utilizing multi-view images. Using a building footprint layer, we first locate the points belonging to buildings. We then detect the 2D windows and doors in images by combining Faster R-CNN and Segment Anything (SAM) deep learning models. The 2D borders are projected into the 3D object space using a pinhole camera model and collinearity equations. Utilizing the multi-view capabilities of mobile scanning systems, this method effectively mitigates uncertainties associated with occlusion and exterior orientation parameters (EOP). This study provides a comprehensive evaluation of 3D spatial accuracy, achieving an average of 84% Intersection over Union (IoU) accuracy for 12 different single-sided façades over 750 multi-view images for 312 windows and doors of various sizes with rectangular and curved shapes.

RÉSUMÉ

Il y a une demande croissante de modèles détaillés de façades de bâtiments (Niveau de détails 3: LoD 3) pour diverses applications. Malgré la croissance d‘articles traitant de ce problème dans la littérature, les occultations posent toujours un gros problème lorsqu’il s‘agit de traiter les éléments des façades des bâtiments. D'autre part, en fonction de l‘angle de vue des images, les éléments détectés peuvent ne pas être projetés à leurs emplacements précis, entraînant des incertitudes quant à leurs emplacements en 3D. Dans cet article, nous abordons les problèmes susmentionnés en utilisant des images multi-vues. En utilisant la couche d’empreinte au sol des bâtiments, nous localisons d‘abord les points appartenant aux bâtiments. Ensuite, nous détectons les fenêtres et les portes en 2D sur les images en combinant les modèles d‘apprentissage en profondeur Faster R-CNN et Segment Anything (SAM). Les bordures en 2D sont projetées dans l‘espace objet en 3D à l‘aide d‘un modèle de caméra sténopé et d‘équations de colinéarité. En utilisant les capacités multi-vues des systèmes de numérisation mobiles, cette méthode atténue efficacement les incertitudes liées aux occultations et aux paramètres d‘orientation extérieure (EOP). L'étude fournit une évaluation complète de la précision spatiale en 3D, atteignant une précision moyenne de 84% pour l‘intersection sur union (IoU) pour 12 façades unilatérales différentes sur 750 images multi-vues pour 312 fenêtres et portes de tailles diverses avec des formes rectangulaires et courbes.

Introduction

In recent years, 3D building façade models with high level of detail (LoD) e.g., LoD3 according to the cityGML standard (Gröger and Plümer 2012), have become increasingly important for a range of applications including urban planning, cultural heritage, building information modeling, disaster management, and computer graphics (Pu and Vosselman 2009; Malihi et al. 2018). To meet the increasing demand for higher LoDs in building models which is shown in Figure 1, it is essential to consider not only the geometric details of buildings but also the specific features of their façades (Sun et al. 2022). Among these features, windows, and door elements need specific attention in the building façade, given their significant role in different applications such as energy loss or flood hazard modeling. With the availability of high-resolution remote sensing data and mobile scanning as well as AI algorithms, creating LoD3 building models is becoming more accessible than ever (Pu and Vosselman 2009; Malihi et al. 2018).

Figure 1. Four levels of detail (LoDs) represent building models. As the level of detail increases, the geometric detail and semantic complexity also increase.

The existing studies in the literature on these subjects can be divided into three categories: (1) 2D detection, (2) 3D detection, and (3) 2D detection and 3D projection.

In the (1) 2D detection category, the focus is on detecting the façade elements in 2D image space. The previous studies in these categories used façade parsing (Liu et al. 2020; Lu et al. 2023; S. Wang et al. 2022) and machine learning algorithms (Pu and Vosselman 2009; Teeravech et al. 2014; Wen et al. 2019). Neuhausen and König (2018) detected 2D rectangular-shaped windows in the street view façade images using the Viola and Jones cascade classifier. The area of façade parsing underwent a substantial transition with the emergence of deep learning algorithms, marking a pivotal shift in this field. Liu et al. (2020) applied Mask R-CNN and Fully Convolutional Neural (FCN) deep learning networks to locate windows and doors by their symmetry shapes using façade close-range images. In another study, Li et al. (2020) detected 2D locations of windows from façade images as four corners using a Fully Convolutional Network (FCN) and heat map analysis.

Nordmark and Ayenew (2021) proposed a method to segment windows’ 2D location from façade street view images using a Mask R-CNN deep neural network. Sun et al. (2022) segmented windows from other façade elements in façade images using the Mask R-CNN deep network. Wang et al. (2022) proposed a method for parsing façade images, which includes three parts: a CNN backbone, object detection, and segmentation parts. For the façade image, the segmentation decoding part outputs the pixel-level semantic class, while the object detection part outputs the generalized bounding boxes in the object level boxes in the object level. Liu et al. (2022) generated the 3D façade model by incorporating both segmentation and detection results using symmetry information along with shape grammar concepts.

Zhang et al. (2022) automatically detected façade elements using the pyramid scene parsing deep network by taking advantage of their symmetrical prior knowledge from façade images. Zhuo et al. (2023) segmented façade images using an attention-U-ResNet deep learning model and injected the resulting segments in the SE-ResNet deep model to extract accurate window corners which resulted in the parsed façade model. A method developed by Lu et al. (2023) for parsing building façades uses a revised SOLOv2 algorithm to accurately segment windows while ignoring occlusions. Ma et al. (2022) proposed a deep learning network to parse the façade images by segmenting visible and invisible parts of the façade occluded by obstructions. Most studies in this category assume the image plane is parallel to the object (i.e., building front) plane (Ma et al. 2022; Zhang et al. 2022; Liu, Li, and Zhu 2022), which is not the case especially in mobile mapping systems.

In the (2) 3D detection category, some studies in the literature investigate façade parsing using point cloud data. These studies either perform façade semantic segmentation (W. Wang et al. 2021; Z. Li et al. 2017) and windows or doors detection from 3D point cloud data acquired from Structure from Motion (SfM) or LiDAR (W. Wang et al. 2021; Z. Li et al. 2017; Salehi and Mohammadzadeh 2017; Chen et al. 2019). Z. Li et al. (2017) developed a hierarchical method of parsing urban building façades. As part of this process, Terrestrial Laser Scanning (TLS) point clouds were segmented to identify façade elements, and knowledge-based information was used to label the distinct elements, then boundaries between the segmented elements were extracted, and the building façade models were generated.

In Previtali, Díaz-Vilariño, and Scaioni (2018), an optimization-based approach was proposed to model indoor scenes by detecting regular patterns with Indoor Mobile Mapping Systems (IMMS) point clouds. This method employed a voxel-based visibility analysis to identify openings in walls, such as windows and doors, as well as to distinguish between occluded and empty regions. Following the detection of wall elements, the regularized indoor models of the buildings were generated.

Jung et al. (2018) developed 3D volumetric models of building interiors containing multiple rooms utilizing point clouds, aiming to enhance the level of detail in Building Information Models (BIM). In the first step of the proposed method, the ceiling and floor heights were estimated using histogram analysis of point clouds. After segmenting the point cloud into different rooms, the floor-wall boundaries were extracted from those segments. Following that, the volumetric wall components were represented with a cuboid shape, then windows and doors were detected and added to the wall components, and finally, the 3D volumetric model was generated. Zheng et al. (2018) employed indoor laser scanning point clouds to identify windows and doors and subsequently subdivide the space. The proposed method initiates by extracting local geometric regularities from the point cloud scanlines to detect windows and doors within the point cloud. When windows and doors are detected, the scanner trajectory is partitioned into distinct spaces based on these detections. Finally, the labels for windows and doors, along with the results of space subdivision, are preserved as point cloud labels.

Lotte et al. (2018) produced SfM point clouds from the façade images, and then segmented them by a CNN network into the different segments such as wall, window, roof, balcony, door, and shop. They labeled them by a ray tracing method which resulted in a 3D façade model. Malihi et al. (2018) developed a method to use perceptual organization and density-based filtering algorithms to detect 3D locations of the windows in the SfM point cloud. Chen et al. (2019) reconstructed a 3D façade model using Mobile Laser Scanning (MLS) point cloud by a histogram analysis, and a peak extraction method accordingly. Xia and Wang (2019) separated facades from point clouds derived from ground-based LiDAR for urban planning. The proposed method involved extracting edges from the point cloud. Next, a window detection method was developed to extract windows from the point cloud. Then, windows and edge information were used to separate facade blocks into individual facade instances.

Pexman, Lichti, and Dawson (2021) created 3D building models and 2D floor plans based on terrestrial laser scanner scans by delineating the story levels of historic buildings and detecting their windows and doors. The proposed method started by rotating the input point cloud around the Z axis, aligning it with the orthogonal X and Y axes. The M-estimator Sample Consensus (MSAC) was used to identify walls after separating building story levels from the rotated point cloud. By analyzing the histograms of two walls, it is possible to detect windows, doors, and points located between them. Lastly, the detected walls, doors, and windows are translated into a CAD model using their outlines. For 2D and 3D shapes, the minimum and maximum values of their X Y, or Z coordinates are stored. By doing so, simple planar features can be represented in a 2D or 3D CAD model. A method proposed by W. Wang et al. (2021) exploits façade knowledge information with semantic segmentation. It utilizes a hierarchical mesh multi-view domain for semantic segmentation of the façade SfM point cloud. Wysocki, Hoegner, and Stilla (2022) detected the 3D location of the buildings’ underpasses by comparing a co-registered Mobile Laser Scanning System (MLS) data with the existing semantic 3D building models in a Bayesian network. There are, however, still problems with deep learning algorithms in terms of localizing windows and doors using 3D point cloud data mainly due to the uncertainty of LiDAR around glass (Kamra et al. 2023).

(3) 2D detection and 3D projection: in these studies, the authors focused on 2D parsing or detecting and then they used camera location-orientation models to project the detected windows and doors to 3D object space. Salehi & Mohammadzadeh (2017) fused the geometrical and spectral features coming from airborne LiDAR point cloud and optical images to generate a 3D building roof model including dormers and chimneys at LOD3. Wen et al. (2019) developed an approach for extracting 2D lines from both image and point cloud data, subsequently establishing correspondences between these lines to generate a comprehensive 3D building model at LoD3. Pantoja-Rosero et al. (2022) proposed an automated method that can detect the 2D location of windows and doors from multi-view façade images. The resulting 2D locations are then projected into 3D object space by assisting EOP parameters. The projected locations are then integrated into a LoD2 model produced through a polyfit process, resulting in the generation of the LoD3 model. The recent work by Cheng et al. (2023) describes a method for projecting 2D opening boundaries onto 3D object space. It was accomplished by extracting these boundaries both from the projected indoor point cloud and from the RGB image. The difference in reflectivity of windows and doors was leveraged efficiently to accomplish this task.

Most of these studies did not consider the effect of occlusion, which can be a significant issue due to the existence of vegetation, cars, pedestrians, etc. in urban areas. The other effects that are not extensively addressed in the literature are the spatial uncertainty in the location of the projected façade elements in 3D due to EOP errors (Pantoja-Rosero et al. 2022; Wen et al. 2019; Cheng et al. 2023), oblique view angles, and consequently the misalignment between projected 2D and 3D features.

In this paper, we propose a new method for window and door detection (hereafter referred to as façade elements for brevity) in the 2D image space benefiting from state of art deep learning solutions and utilizing multi-view images to reduce spatial uncertainties. Using Faster-RCNN (Ren et al. 2015; Esri 2022) and Segment Anything (SAM) (Kirillov et al. 2023) deep learning models, we detect the façade elements in the image space (2D), and then project them into the 3D object space using a pinhole camera model and collinearity equations. We utilize the multi-view capabilities of the mobile scanning systems to reduce spatial uncertainties due to occlusion and EOP errors. To address the uncertainties, we propose an Inverse Distance Weighting (IDW) - inspired function to improve the 3D location of facade elements.

Data preparation

This study exploits data from two different sources. The first set of data comprises multi-view camera images obtained directly from single-sided building façades at the University of New Brunswick (UNB) campus in Fredericton, captured using a Digital single-lens reflex (DSLR) camera. The second dataset consists of street-view camera images and MLS point clouds acquired using the Leica MLS device. We applied our proposed method to analyze 750 multi-view images of 12 single-sided facades (FC 1 – FC 12), as illustrated in Figure 2. Specifically, we focused on 312 facade elements of various sizes, including rectangular and curved shapes, from buildings in urban areas of Fredericton, NB (most of the buildings are of red brick type). As shown in Figure 3, in some images, certain parts of the buildings, such as facade elements, were obstructed by trees, and other obstacles. To process the data, the multi-view DSLR camera images were utilized to create SfM point clouds using the Pix4D mapper software. Subsequently, building footprint polygons of Fredericton city from GeoNB's data catalog were employed to extract façade-specific point clouds. Both datasets encompassed information regarding EOP and internal orientation parameters (IOP) $\left(x_p,y_p,f\right)$. EOP defines the position and orientation of the camera at the time an image is captured. It consists of 6 parameters: 3 projection center coordinates $\left(X^L,Y^L,Z^L\right)$ and 3 rotations $\left(\omega ,\phi ,\kappa \right)$ around X, Y, and Z axes, respectively. Camera position is determined using a Global Positioning System (GPS), while rotations are measured using an Inertial Navigation System (INS) mounted on the measuring device. Moreover, IOP defines the camera calibration parameters, which include three parameters such as $\left(x_p,y_p\right)$ as image principal point coordinates, and $f$ as the camera focal length. Table 1 contains the accuracy information of the EOPs for the two datasets while the specifications of the DSLR camera and the MLS device are provided in Table 2.

Figure 2. The windows are obstructed by trees and billboard.

Figure 3. Dataset compilation from single-sided facades (FC 1- FC 12).

Table 1. EOPs accuracies for two datasets, coordinates in meters, and angles in degree unit.

		EOPs accuracy
		${\sigma }_{X_L}(m)$	${\sigma }_{Y_L}(m)$	${\sigma }_{Z_L}(m)$	${\sigma }_{\omega }{}^{°}$	${\sigma }_{\phi }{}^{°}$	${\sigma }_{\kappa }{}^{°}$
Dataset	SfM	0.001	0.001	0.001	0.013	0.013	0.006
	MLS	0.01	0.01	0.02	0.013	0.013	0.004

Table 2. The specifications of data collection devices.

Device	Sensor	Specifications
Canon EOS 5D Mark III DSLR camera	Frame camera	Number of cameras: 1 Sensor width: 37.10 mm Sensor height: 24.73 mm Sensor: Full frame Data: RGB image, pixels Width: 5760 pixels Height: 3840 pixels Pixel size: 6.44 µm
Leica Pegasus: Two Ultimate Laser scanning device	Frame and Pavement cameras	Number of cameras: 4 Sensor: CMOS 12 MP (megapixels) Data: RGB image, pixels Width: 3008 pixels Height: 4096 pixels Pixel size: 3.45 µm
	Scanner	Number of scanners: 1 Type: Z + F PROFILER® 9012 Mode: single Data: 3D point cloud [X, Y, Z], m Point density: ∼ 12000 point/m^2

Methodology

As illustrated in the flowchart in Figure 4, we utilize Faster-RCNN and SAM deep learning models for detecting facade elements in multi-view images. These elements are then projected into the 3D object space. Subsequently, we integrate 3D polygon candidates representing facade elements. This integration aims to reduce spatial uncertainty and improve the 3D localization accuracy of these elements. To clarify the process, the pseudo-code of the work as depicted in Figure 5, along with detailed explanations of each step, is provided as follows.

Figure 4. The overall flowchart of the proposed method.

Figure 5. The pseudo-code of the proposed method.

Faster – R-CNN

Faster R-CNN is an object detection algorithm that integrates deep convolutional neural networks with region proposal networks. This integration allows for the effective and precise detection of objects in images (Ren et al. 2015). We used the Esri Faster R-CNN pre-trained windows and doors extraction model (Esri 2022; Kuznetsova et al. 2018). To detect windows and doors, multi-view façade images are inputted into the model, generating 2D bounding boxes $\{{(x_{\text{min}},y_{\text{min}},\text{w and h})}_{i\prime }\hspace{0.17em}|\hspace{0.17em}i\prime =1,\dots ,n\prime \}$ for each facade element in every façade image in which ${\text{x}}_{\mathrm{\text{min}}},{\mathrm{y}}_{\mathrm{\text{min}}},\text{w and h}$ refer to 2D coordinates of the top-left corner, width, and height of a bounding box respectively. As we use multi-view images, i’ refers to each single image and n’ is total number of images in which a single window is detected. We detected bounding boxes (green) in two different single images as can be seen in Figure 6(a,b)

Figure 6. The overlaid bounding boxes (green) on single view façade images captured by DSLR cameras (a), and Leica MLS device (b).

Segment anything (SAM)

As shown in Figure 6(a,b), the bounding boxes generated by Faster R-CNN fail to precisely represent the true boundaries of facade elements. To improve the boundary detection step, a segmentation algorithm like SAM is needed. SAM model has garnered significant attention in computer vision research (Zhang et al. 2023). Notably, SAM possesses the capability of zero-shot image segmentation, enabling its application in specialized scenarios like facade elements segmentation. SAM can segment a façade image into different segments including all façade structures. These segments can be selected only for façade elements by using SAM’s promptability feature. We provide the detected bounding boxes as approximate locations of facade elements as input prompts as can be seen in Figure 7(b), which helps the SAM model to generate the segments corresponding to façade elements as depicted in Figure 7(c) (Kirillov et al. 2023; Y. Wang, Zhao, and Petzold 2023).

Figure 7. a) A single view façade image captured Fredericton residential area b) detected windows and doors bounding boxes by Faster-RCNN c) refined 2D SAM segments.

2D–3D correspondence

To find the 3D location of façade elements, we project the points of the façade point cloud into each individual image using a Pin-Hole camera model which can be seen in Figure 8(b). Moreover, to determine the depth of the 2D border points of facade elements’ segments, there are two possible options. The first option involves identifying matched points on the borders of facade elements across various multi-view images of the façade, followed by triangulation. However, matching linear features from one image to another can be time-consuming, and prone to errors due to matching errors (Han et al. 2014; Moghimi et al. 2022; Collins, Mesejo, and Bartoli 2014). Moreover, the varying number of edge points for the same facade elements across different multi-view images of the façade further complicates the process. Alternatively, assuming façade elements lay on a single plane, we augmented the original system of collinearity equations in Equation (1)(1) $$\{\begin{array}{c}x=x_p-f \frac{m_{11}\left(X-X^L\right)+m_{12}\left(Y-Y^L\right)+m_{13}\left(Z-Z^L\right)}{m_{31}\left(X-X^L\right)+m_{32}\left(Y-Y^L\right)+m_{33}\left(Z-Z^L\right)} \\ y=y_p-f \frac{m_{21}\left(X-X^L\right)+m_{22}\left(Y-Y^L\right)+m_{23}\left(Z-Z^L\right)}{m_{31}\left(X-X^L\right)+m_{32}\left(Y-Y^L\right)+m_{33}\left(Z-Z^L\right)} \end{array}$$(1) with an additional plane. This augmentation is crucial to establish determinacy and facilitate the solution for the depth.

Figure 8. a) A single view façade image captured Fredericton residential area b) projected point cloud overlaid on the image.

The concept of collinearity consists of a straight line connecting the exposure station, the point of interest within the frame, and its corresponding image. As a result of this relationship, the object in the real world, its representation in the image, and the observer’s perspective are aligned (Wolf, Dewitt, and Wilkinson 2014). In the collinearity equations, ($x$, $y$) are image coordinates, $\left(x_p,y_p\right)$ are image principal point coordinates, $f$ is the camera focal length, $m_{13 }{\dots m}_{33}$ are elements of rotation matrix (M) between object space and image space coordinate systems, (X, Y, Z) are ground coordinates in the object space, and $\left(X^L,Y^L,Z^L\right)$ are camera perspective center coordinates in the object space. (1) $$\{\begin{array}{c}x=x_p-f \frac{m_{11}\left(X-X^L\right)+m_{12}\left(Y-Y^L\right)+m_{13}\left(Z-Z^L\right)}{m_{31}\left(X-X^L\right)+m_{32}\left(Y-Y^L\right)+m_{33}\left(Z-Z^L\right)} \\ y=y_p-f \frac{m_{21}\left(X-X^L\right)+m_{22}\left(Y-Y^L\right)+m_{23}\left(Z-Z^L\right)}{m_{31}\left(X-X^L\right)+m_{32}\left(Y-Y^L\right)+m_{33}\left(Z-Z^L\right)} \end{array}$$(1)

The known parameters involved in this process include the elements of the rotation matrix ($m_{13 }{\dots m}_{33}$), camera perspective center coordinates $\left(X^L,Y^L,Z^L\right)$, image coordinates (x, y), image principal center coordinates ($x_p,y_p$), and camera focal length (f), all obtained from the camera pose file.

To determine the plane parameters for each facade element, we extract their corresponding 3D points from the point cloud as shown in Figure 9(d) by overlaying the points within their segment borders onto the projected point cloud as shown in Figure 9(b). Moreover, a predefined buffer (100 pixels), determined by considering the approximate façade elements sizes and Ground Sampling Distance (GSD) of images is applied to the 2D borders’ points. This buffer is designed to encompass a broader range of 3D points from the surrounding façade walls. This step is crucial because the areas of facade elements often contain noisy points due to mirrors and curtains behind the window glass. Subsequently, the 3D points associated with facade elements are removed as seen in Figure 9(e) from the buffered 3D points. The remaining 3D points are then processed through the MSAC algorithm to identify the inlier points, which allows the extraction of the plane parameters (a, b, c, and d) (Torr and Zisserman 2000).

Figure 9. 2D-3D correspondence and buffering process. (a) projected point cloud on a single view façade image b) overlaid 2D border segment (red color) c) SfM point cloud d) buffered corresponding 3D point cloud, e) removed noisy window points.

After extracting the plane parameters and formulating the plane equation for each façade element, we then augment the respective collinearity equations as can be seen in Equation (2)(2) $$\{\begin{array}{c}x+f \frac{m_{11}\left(X-X^L\right)+m_{12}\left(Y-Y^L\right)+m_{13}\left(Z-Z^L\right)}{m_{31}\left(X-X^L\right)+m_{32}\left(Y-Y^L\right)+m_{33}\left(Z-Z^L\right)}-x_p=0\\ y+f \frac{m_{21}\left(X-X^L\right)+m_{22}\left(Y-Y^L\right)+m_{23}\left(Z-Z^L\right)}{m_{31}\left(X-X^L\right)+m_{32}\left(Y-Y^L\right)+m_{33}\left(Z-Z^L\right)}-y_p=0\\ \mathit{\text{aX}}+\mathit{\text{bY}}+\mathit{\text{cZ}}+d=0\end{array}$$(2) , resulting in 3D location (X, Y, Z) depicted in Figure 10(d) for every 2D border’ points (x, y) shown in Figure 10(b). This procedure is repeated for every border of the facade element in all multi-view images. When this process is completed, we have the 3D location estimates for façade elements. (2) $$\{\begin{array}{c}x+f \frac{m_{11}\left(X-X^L\right)+m_{12}\left(Y-Y^L\right)+m_{13}\left(Z-Z^L\right)}{m_{31}\left(X-X^L\right)+m_{32}\left(Y-Y^L\right)+m_{33}\left(Z-Z^L\right)}-x_p=0\\ y+f \frac{m_{21}\left(X-X^L\right)+m_{22}\left(Y-Y^L\right)+m_{23}\left(Z-Z^L\right)}{m_{31}\left(X-X^L\right)+m_{32}\left(Y-Y^L\right)+m_{33}\left(Z-Z^L\right)}-y_p=0\\ \mathit{\text{aX}}+\mathit{\text{bY}}+\mathit{\text{cZ}}+d=0\end{array}$$(2)

Figure 10. The process to generate facade elements 3D locations. a) A single view façade image captured UNB campus b) overlaid a 2D polygon from SAM (red color) c) the projected 3D polygon overlaid on SfM point cloud d) zoomed in the projected 3D polygon.

Spatial combination of candidates

Following the generation of 3D polygons for each façade element in each image, overlapping polygons across the entire set of multi-view images need to be identified. In this study, we employed the IoU metric (Tripodi et al. 2020) iteratively to detect instances of overlapping polygons as can be seen in Figure 11(a).

Figure 11. Three samples of overlapping 3D polygons (window border candidates) across three image views. a) Overlapping 3D polygons for a single window b) 2D polygons in different view angles.

Figure 12. Overlaid 3D polygons on the single-sided façade point clouds (FC 1-12).

As shown in Figure 11(a), façade elements 3D candidates, do not entirely overlap due to their location uncertainties coming from their EOP or IOP errors, random errors, and unmodeled biases. Therefore, we aim to develop a method to combine all candidates, effectively reducing the total spatial uncertainty for each facade element. In this work, we developed a spatial weighting function, which was derived from the IDW concept (Shepard 1968). In practice, the image that “sees” a single façade element under a vertical angle (where the image plane is parallel to the object plane) can better represent the window compared to oblique images. However, since due to obstacles, all façade elements might not be well-represented in vertical angle images, it is beneficial to utilize multi-view angle images to increase the chance of obstacle avoidance. As a result, multiple 3D borders can be generated which should be “averaged” to improve precision. However, images with oblique angles should not get equal weights as vertical images. The view angle (α) is found by the vector from the camera perspective center $\left(X^L,Y^L,Z^L\right)$ to the center of 3D window candidate polygon $\left\{V_1=\mathit{\text{mean}}\left(X_i,Y_i,Z_i\right)-\left(X^L,Y^L,Z^L\right)\right|i=1,\dots ,n\}$ ($i=1,\dots ,n$ is number of points from the façade element candidate polygon) and the normal vector of the image plane $\left(V_2=\left(a,b\right)\right)$. Equations (3) and (4) are used to calculate the view angle and uncertainty-reducing function. $$\mathit{\text{cosine}}\left(\alpha \right)=\frac{V_1.V_2}{\left|V_1{V}_2\right|},0\le \alpha \le {90}^o\hspace{0.17em}$$ $$W_i=\frac{\text{cosine}\left(\alpha \right)}{{\mathit{\text{Distance}}}^2}\hspace{0.17em}$$

On the other hand, to further improve precision, the 3D borders that are closer to each other should get higher weights. Inspired by the IDW function, Equation (5) is used to find the enhanced borders of the façade elements. $$\left\{\begin{array}{c}\hat{X}=\frac{\sum X_i\text{*}{W}_i}{\sum W_i}\\ \hat{Y}=\frac{\sum Y_i\text{*}{W}_i}{\sum W_i}\\ \hat{Z}=\frac{\sum Z_i\text{*}{W}_i}{\sum W_i}\end{array}\right.\hspace{0.17em}$$

Where $X_i,Y_i,Z_i$: are candidate coordinates, and $W_i$ are the weights computed in Equation (4), and $(\hat{X},\hat{Y},\hat{Z}$) are the combined 3D polygon coordinates. After computing the combined 3D polygon coordinates, they are injected into the Ramer-Douglas-Peucker algorithm, as described in Salehi and Mohammadzadeh (2017), to simplify the overall shape of 3D polygons.

In practice, out of all the overlapping candidates for every façade element, we begin by selecting the one with the maximum area. Following this, we use the KNN search algorithm to determine which candidates’ points in each polygon are closest to the maximum area candidate points, and we designate these as KNN points. Then, we calculate their distance to the maximum area of candidate points.

Results and evaluation

To evaluate the accuracy of the final 3D facade elements’ polygons, we used the IoU metric as shown in Equation (6)(6) $$\mathit{\text{IoU}}=\frac{A^O\mathrm{ }}{A^U}$$(6) between the computed 3D polygons and ground truth 3D polygons drawn manually over the 3D point cloud (Tripodi et al. 2020). (6) $$\mathit{\text{IoU}}=\frac{A^O\mathrm{ }}{A^U}$$(6) where $A^O$ is the area of overlap between 3D computed polygon and ground truth, and $A^U$ is the area of union between them. In this research, we processed 750 multi-view images for 12 single-sided façades to compute the 3D locations of 312 façade elements. The results for all single-sided building facades (FC 1 – FC 12) tested with our proposed method are presented in Table 3, and corresponding figures can be found in Figure 12. Furthermore, we computed the mean ($\overline{X})$ and standard deviation (σ) of IoU values using Equation (7)(7) $$\sigma =\sqrt{\frac{1\mathrm{ }}{N-1}\sum _{i=1}^N{\left|{\mathit{\text{IoU}}}_i-\overline{X}\right|}^2}$$(7) for each single-side façade, considering all the facade elements within that particular façade. (7) $$\sigma =\sqrt{\frac{1\mathrm{ }}{N-1}\sum _{i=1}^N{\left|{\mathit{\text{IoU}}}_i-\overline{X}\right|}^2}$$(7)

Figure 13. Detected incorrect objects (yellow dash line circles) in the two different image views.

Table 3. Mean and standard deviation of IoU values for the final 3D polygons of facade elements, and the number of façade elements and images used in the implementation.

Façade #	Number of images	Number of windows	Number of doors	IoU [0 - 1] (unitless)
				$\overline{\mathit{X}}$ $\pm$σ
FC 1	137	42	0	0.82 $\pm$ 0.08
FC 2	15	7	2	0.90 $\pm$ 0.02
FC 3	15	18	1	0.81 $\pm$ 0.14
FC 4	69	40	2	0.81 $\pm$ 0.16
FC 5	47	23	2	0.84 $\pm$ 0.17
FC 6	55	21	1	0.76 $\pm$ 0.21
FC 7	170	43	1	0.89 $\pm$ 0.04
FC 8	77	28	1	0.89 $\pm$ 0.15
FC 9	40	11	1	0.85 $\pm$ 0.20
FC 10	40	16	0	0.89 $\pm$ 0.05
FC 11	28	11	1	0.80 $\pm$ 0.33
FC 12	57	40	0	0.81 $\pm$ 0.14
Total	750	300	12	-
Average	-	-	-	0.84$\pm$0.16

The values of column 5 are dimensionless decimal numbers, indicating the ratio of overlap.

Where N is the number of façade elements and IoU values range from 0 to 1, with 1 indicating a perfect overlap between the detected and ground truth façade elements.

Discussion

We conducted tests on our proposed method using 12 single-sided facades, encompassing a total of 312 rectangular and curved-shaped facade elements with varying shapes and sizes. It is important to note that this experiment specifically targeted red brick buildings and focused on a single side, which may not fully represent all cases. However, the adaptability of the method to more diverse building scenarios, including other sides or the entire structure, with different shapes and architectures, is evident. Despite the noise present in the facade point clouds and the obscuration of facade elements due to various occlusions, our proposed method demonstrates effective detection. Additionally, the method showcases the capability to model facade elements with diverse shapes and architectures, irrespective of their size or visibility amid occlusions. Furthermore, the efficiency of the proposed method is underscored by its successful implementation on a substantial number of multi-view images (750 images), suggesting its adaptability for large-scale implementation.

Across all facades (FC 1 to 12), our proposed method achieved an overall IoU score of 0.84 as shown in Table 3. Furthermore, a standard deviation of 0.16 suggests moderate variability in the accuracy of detections across different images and facades. The accuracy of detection varies depending on the type of facade. For example, FC 2 obtained a very high IoU score of 0.90, which indicates very accurate facade element detection, whereas FC 6 obtained a relatively lower IoU score of 0.76, which indicates less accurate detection in this facade. In contrast, FC 6 attained a relatively lower IoU score of 0.76, suggesting less accurate detection in this particular facade. These discrepancies likely stem from challenges such as variations in image quality, lighting conditions, complete obstruction of windows by trees in specific views, or architectural diversity among different facades. While the proposed method has demonstrated effectiveness in predicting facade elements in 2D through the trained Faster R-CNN model, it is essential to acknowledge certain limitations. Specifically, predicting very small facade elements can pose challenges, and there may be difficulty in distinguishing between billboards or other rectangular objects and actual facade elements, as illustrated in Figure 13.

The proposed method demonstrates strong performance when applied to segmenting an entire image using SAM. However, challenges persist, notably in terms of over-segmentation and difficulty in distinguishing between different parts of facade elements. Additionally, a noticeable misalignment occurs between the registered projected point cloud and the 2D image, particularly along the borders of facade elements, a discrepancy more pronounced when dealing with the MLS dataset. This misalignment may arise from uncertainties in in-field data registration and inaccuracies in RGB point cloud coloring algorithms. Furthermore, inaccuracies resulting from imprecise EOP or IOP during data collection could contribute to this misalignment. Despite these uncertainties, the proposed method effectively leverages multiple 3D candidates from different view angles for each facade element, thereby mitigating uncertainties and enhancing the final accuracy of facade element localization in 3D.

Furthermore, in the 2D to 3D correspondence, we extract localized point clouds corresponding to each facade element. This approach proves advantageous for fitting more accurate planes for individual facade elements, as opposed to attempting to fit a single plane for all facade elements simultaneously. However, challenges arise when fitting local planes for facade elements located near the intersection of two cardinal walls or when one facade plane deviates from the others. This challenge becomes more pronounced when the image view angle is narrow, leading to the identification of incorrect plane points.

The proposed method introduces an alternative approach to augment the original system of collinearity equations by incorporating an additional plane. This augmentation facilitates the computation of the depth of the 2D borders’ points of facade elements’ segments, skipping the need for matching linear features from one image to another and mitigating potential matching errors. However, the accuracy of this process hinges on precise plane parameters and EOP derived from the camera’s pose, as well as the alignment between the projected point cloud and the image. Any inaccuracies in these parameters can lead to imprecise 3D polygons. To address this, the proposed method leverages a modified spatial IDW-inspired function, considering both the view angle and the distance of each 3D point along the border of 3D facade element candidates.

Conclusion and future directions

Our study proposes a novel method for detecting facade elements in image space (2D) by combining Faster R-CNN and SAM, followed by a projection into 3D object space. Our findings demonstrate the effectiveness of our approach with an % 84 IoU score for the accurate detection of facade elements in 3D, across the dataset of 12 different single-sided façades over 750 multi-view images for 312 facade elements of various sizes with rectangular and curved shapes. This high performance is attained by addressing key limitations present in current techniques. These limitations encompass occlusion effects and spatial uncertainties arising from errors in EOPs, oblique view angles, and misalignment between the projected point cloud and image (2D) which are essential for generating a LOD3 façade model. To overcome these challenges, we employed multi-view capabilities to reduce uncertainties arising from occlusion and EOP errors. To address these uncertainties, we introduced an IDW-inspired spatial weighting function that significantly enhanced the accuracy of 3D localization for facade elements.

Furthermore, there are two potential avenues for improving the proposed method. Firstly, in the registration process between point clouds and image data, enhancement can be achieved by adopting more automated and robust registration methods. This refinement would contribute to a more seamless alignment between the two data sources. Secondly, augmenting the original collinearity equations by introducing additional constraints, such as incorporating the linearity of other objects, has the potential to further enhance the registration process and contribute to the overall refinement of the system. Looking ahead, our future efforts will concentrate on two key areas. The first area of focus will be the integration of indoor elements of buildings into our 3D model. This advancement will enable a more comprehensive 3D representation of the entire building structure, encompassing both exterior and interior environments. The second area of emphasis will be on occluded elements, which require in-depth research and investigation. Additionally, various architectural and functional façade elements will be incorporated to create a more intricate and precise representation of the structures.

Acknowledgments

We extend our gratitude to the NBIF-POSS and NSERC-Discovery funding agencies for their invaluable support throughout this project. Special thanks are also due to Esri and Meta companies for providing access to the pre-trained deep learning models essential to our research. Our sincere appreciation goes to Leica Geosystems and Eagle Engineering and Consulting companies for generously supplying the required dataset, which was vital to our work. We are grateful to GeoNB, a government subdivision, for granting us access to the building footprint dataset, a crucial component of our research. Additionally, the utilization of Generative AI tools significantly enhanced the quality of our writing. We would also like to acknowledge the support received from Pix4D and Trimble Inc. companies, without which this research would not have been possible.

Disclosure statement

No potential conflict of interest was reported by the author(s).