UAV image matching from handcrafted to deep local features

ABSTRACT

Local feature matching between images is a challenging task, particularly when there are significant appearance variations, such as extreme viewpoint changes. In this work, we present LoFTRS, a deep learning-based image matching framework that integrates semantic constraints into the matching process. Our key insight is that a local feature matcher with deep layers can capture more human-intuitive and simpler-to-match features. In addition to image segmentation module, we also propose a detector-free Transformer module. It uses vector-based attention to model relevance among all features and achieves efficient and effective long-range context aggregation. Transformer module applies a relative position encoding to explicitly disclose relative distance information, further improving the representation of features. We evaluate the performance of LoFTRS comparing to various popular handcrafted and deep learning-based methods. We investigate the relationship between matching quality and the performance of subsequent processing steps, such as the accuracy and completeness of the model generated by SfM. The experimental results show that the proposed LoFTRS achieves equal or better image matching performance in terms of matching score, average track length, RMSE, and the number of 3D points.

KEYWORDS:

• Image processing feature matching
• handcrafted methods
• deep learning
• UAV
• SfM
• Transformer

Introduction

Unmanned Aerial Vehicle (UAV) have been used on a large scale as new remote sensing platforms for oblique photogrammetry and computer vision (Pajares, 2015). With the advantages of flexible data acquisition and low economic cost, high resolution and multi-view images can be provided by carrying various non-surveying cameras for different application areas (Daakir et al., 2017), including object recognition (Chiabrando et al., 2018), route inspection (Jiang et al., 2017), and agricultural planning (Habib et al., 2016). After data acquisition is complete and before application, image local feature matching is a critical step in many computer vision fields (Jiang & Jiang, 2017). For example, in Structure from Motion (SfM) and Multi-View Stereo (MVS) (Radenovic et al., 2016; Schonberger & Frahm, 2016; Schonberger et al., 2015; Schönberger et al., 2016), image retrieval (Philbin et al., 2007; Sattler et al., 2016; Tolias et al., 2016; Torii et al., 2015), and image-based localization (Sattler et al., 2011, 2016; Zeisl et al., 2015).

When performing 3D reconstruction of real scene datasets in the absence of camera poses and known scene structures, a variety of problems may be encountered that can affect the quality and accuracy of the reconstruction. Real-world data usually contains noise, illumination variations, and distortions, which can lead to inaccuracies in the generated 3D models; occlusion and masking between objects can lead to missing certain regions, affecting the completeness of the reconstruction; illumination variations in different times or conditions can lead to color and luminance differences between the images, which in turn affects the consistency of the reconstruction (Vijayanarasimhan et al., 2017); some objects may lack sufficient texture information, resulting in a lack of detail in the reconstructed model; camera or object motion during shooting may lead to image blurring, which may affect depth estimation and matching (Huang et al., 2018); inaccurate camera parameter estimation may lead to geometric transformation errors, which affects the accuracy of the reconstruction; processing of large-scale real-scene datasets requires high computational resources, and data management and processing may become complex.

In the last decades, Scale-invariant feature transform (SIFT) (Lowe, 2004) and its variants, such as RootSIFT (Arandjelović & Zisserman, 2012) and DSP-SIFT (Dong & Soatto, 2015), have been the most used local features in image matching due to their invariance to changes in scale, rotation, and illumination, as well as their high robustness to changes in viewpoints. To handle different data and improve the performance of local features, there are some methods for different applications in photogrammetry and remote sensing, such as L2-SIFT (Sun et al., 2016) and AB-SIFT (Sedaghat & Ebadi, 2015).

Recently, the rise of neural networks has been widely applied in the field of computer vision, e.g. in target detection and recognition (Krizhevsky et al., 2017; Sharif Razavian et al., 2014). As a result, neural networks have also been applied to the problem of feature descriptor learning (Simo-Serra et al., 2015) to filter more discriminative representations for local features, e.g. SuperPoint (Hu & Mordohai, 2012) and D2Net (Jegou et al., 2008).

The results show a significant improvement over traditional handcrafted methods (SIFT, Speeded Up Robust Features (SURF) (Bay et al., 2006) or DAISY (Tola et al., 2009)). However, these methods are usually trained and evaluated on Brown benchmarks (Schonberger & Frahm, 2016) for patch validation and classification, as described in (Schonberger et al., 2017) and (Jin et al., 2021). The intrinsic purpose of these methods is not the same as that of image matching. Meanwhile, there are many modular steps included in SfM-based workflows, such as feature extraction, descriptor matching, outlier elimination, and Bundle Adjustment (BA) (Harris & Stephens, 1988). Local feature matching is just a precursory step in the direction of subsequent image processing. Good results in this step do not guarantee a more substantial improvement in the final reconstructed model. Therefore, evaluating its performance in the full SfM pipeline is reasonable and necessary (Jin et al., 2021).

However, to the best of our knowledge, no such evaluation has been performed for feature extraction and matching in SfM-based orientation and MVS-based reconstruction for processing UAV images. Although comprehensive evaluations have been performed in (Jin et al., 2021) and (Schonberger et al., 2017), the data is from community-based imagery with poor spatial resolution of the images, and at the same time, some of the new learning-based features (Schonberger et al., 2017) were not evaluated, and the collective amount of data for SfM-based orientation in (Jin et al., 2021) is small, which is not adapted to the practical needs of UAV imagery.

In this paper, we conduct a comprehensive experimental evaluation of learned and handcrafted feature descriptors to better understand their performance. Specifically, the paper makes the following contributions:

We perform a comprehensive evaluation of classical handcrafted and deep learning-based methods for feature matching and point cloud reconstruction.

Compared to previous evaluations, we conduct a more detailed study of the matching performance of different descriptors using a wider range of evaluation criteria and scenarios. As illustrated in Figure 2, we propose a deep learning-based image matching framework called LoFTRS, which incorporates semantic constraints into deep learning-based matching process. We analyze the performance of the matching process and compared the impact of Semantic Deep Matcher on image-based reconstruction to various popular handcrafted and deep learning-based methods. Particularly, we also explore the connection between the matching quality and the performance of the subsequent processing steps, e.g. the accuracy and completeness of the model generated by SfM.

Related work

This study deals with the performance evaluation of hand-crafted features and deep learning-based features in the context of SfM-based image reconstruction. Figure 1 illustrates the overall flow of the reconstruction. Therefore, in this section, we will review some of the work related to local feature matching, including hand-crafted features and deep learning-based features.

Figure 1. Reconstruction process.

Hand-crafted features

In the absence of deep learning-based features, traditional local features, also known as hand-crafted features, play an important part in the field of image matching. Even nowadays, in some specific environments, deep learning methods are not as effective as hand-crafted features. The advantages and disadvantages of Handcrafted (Tareen & Saleem, 2018) used in this article are summarized in the following table.

A good local feature detector should have the two features of detecting distinguishable features and covariance constraints, i.e. repeatedly detecting consistent features under different transformations. Typical representative algorithms include.

Harris operator (Harris) (Harris & Stephens, 1988): proposed a classical method for accurate detection of corner features by analyzing the local second-order matrix of an image and constructing a window translation information quantity change function. The Harris operator has rotation and contrast invariance but lacks scale invariance and is limited by the selection of the threshold value.

Scale-invariant feature transform (SIFT) (Lowe, 2004): used Difference of Gaussians (DoG) as a detector to select keypoints by scale space pyramid. SIFT features are scale invariant and they can be generated for even small objects. However, SIFT requires a significant amount of memory, especially when working with high-resolution images or large databases. This memory intensity can be a challenge in resource-constrained environments.

Speeded Up Robust Features (SURF): Bay’s proposed SURF (Bay et al., 2006) improves the extraction and description of features in a more efficient way. SURF uses a Hessian matrix determinant approximation of the image. When the Hessian matrix obtains the local maxima, it determines that the current point is a brighter or darker point than other points in the surrounding neighborhood and thus locates the position of the key point. However, SURF is not robust enough to adapt to changes in scale and rotation.

Oriented FAST and Rotated BRIEF (ORB): ORB (Rublee et al., 2011) feature description algorithms have much better runtime than SIFT and SURF and can be used for real-time feature detection. ORB features are based on the FAST corner-point feature point detection and description technique and are invariant to noise and perspective affine. The disadvantage is that the ORB scale transformation has a relatively low coping ability.

AKAZE: Pablo (Alcantarilla & Solutions, 2011) uses Fast Explicit Diffusion (FED) to construct the scale space faster than any other nonlinear model at the time, while being more accurate than AOS. An efficient Modified Local Difference Binary Descriptor (M-LDB) is introduced, which adds uniqueness to the gradient information of the scale space constructed by combining FED. The AKAZE algorithm is faster than the SIFT and SURF algorithms, while the repeatability and robustness are greatly improved over the ORB algorithm.

Based on the comparison of SIFT, SURF, ORB, and AKAZE (Tareen & Saleem, 2018), the following conclusions can be drawn:

The quantitative comparison shows that the order of feature-detector-descriptors for detecting a high quantity of features is:

$\mathrm{ORB}>\mathrm{SURF}>\mathrm{SIFT}>\mathrm{AKAZE}>\mathrm{KAZE}.$

The feature detection description for each feature-point is:

$\mathrm{ORB}>\mathrm{ORB}\left(1000\right)>\mathrm{SURF}\left(64\mathrm{D}\right)>\mathrm{SURF}\left(128\mathrm{D}\right)\text{break}>\mathrm{AKAZE}>\mathrm{SIFT}>\mathrm{KAZE}.$

The order of efficient feature-matching per feature-point is:

$\mathrm{ORB}\left(1000\right)>\mathrm{AKAZE}>\mathrm{KAZE}>\mathrm{SURF}\left(64\mathrm{D}\right)\text{break}>\mathrm{ORB}>\mathrm{SIFT}>\mathrm{SURF}\left(128\mathrm{D}\right).$

The feature detector descriptors can be ranked according to their speed of total image matching as follows:

$\mathrm{ORB}\left(1000\right)>\mathrm{AKAZE}>\mathrm{KAZE}>\mathrm{SURF}\left(64\mathrm{D}\right)\text{break}>\mathrm{SIFT}>\mathrm{ORB}>\mathrm{SURF}\left(128\mathrm{D}\right).$

Deep learning-based local features

Deep learning descriptors are often considered supervised learning problems. The goal is to develop a representation that can combine two matching features as closely as possible while keeping mismatched features separate from each other in the measurement space.

Descriptor learning is also known as patch matching because it often uses local patches that have been cropped and centered on keypoints. In general, existing methods consist of two forms, namely, metric learning (Wang et al., 2017) and descriptor learning (Luo et al., 2019). Usually, these two forms are practiced together. Using original patches or created descriptors as input, metric learning algorithms often train a discriminative metric for similarity evaluation. In contrast, descriptor learning typically creates descriptor representations from unprocessed images or patches.

Deep learning-based descriptors can be seen as an extension of those based on classical learning (Schonberger et al., 2017). For example, recent deep methods have adopted and modified the Siamese structure (Chopra et al., 2005) and often use loss functions such as hinge, siamese, triple, ranking, and contrast losses. More precisely (Zagoruyko & Komodakis, 2015), introduced their depth comparison and showed how to learn directly from the original image pixels using a generalized patch similarity function. In this case, the similarity function is encoded using CNN models (Krizhevsky et al., 2017) of different siamese types. These models are then taught to distinguish between positive and negative image patches. Siamese with shared or unshared weights and a central surround form are two of several network architectures that have been tried. To learn both descriptors and metrics, MatchNet (Han et al., 2015) was proposed. A Siamese-like description network and a fully convolutional decision network were used to implement the method. As a result, the performance of the description is greatly improved (Jiang et al., 2014). proposed a unique deep ranking method to discover fine-grained image similarities. The model uses a triplet hinge loss and ranking function to define fine-grained image similarity associations. Global visual features and image semantics are captured using a multi-scale neural network design. Siamese and triple networks were used for implementation, combining triple combination and global training loss. TFeat (Balntaset al., 2016) suggested the use of three sets of training samples for CNN-based patch description and matching.

The method was implemented using a fast, hard negative mining method and shallow convolutional networks (Tian et al., 2017). used a progressive sampling technique in L2Net to improve the relative distance-based loss function in Euclidean space. To improve efficiency, the authors of this study considered the compactness of descriptors and intermediate feature mappings. By combining direct hinge-triplet loss with “hardest-in-batch” mining, HardNet (Mishchuk et al., 2017) improves performance over L2Net, and PN-Net (Balntaset al., 2016) trains both positive and negative constraints, utilizing the concepts of online augmentation and distance metric learning. Compared to hinge loss or SoftMax ratio, the proposed SoftPN loss function has a faster conversion speed. Generalized rank formula learning is introduced through descriptor learning based on average precision attention. In this technique, true matches should be prioritized over all false path matches, which are specified as a constraint and optimized using binary and real-valued local feature descriptors. For generative adversarial networks (Mirza & Osindero, 2014) to train discriminative but compressed binary representations of image patches, in contrast, by exploiting a pool of kernelized subspaces, Wei et al. (2018) learned a discriminative depth descriptor that excludes loss functions, network structure, regularization, and hard negative mining. SOSNet (Yi et al., 2016) trained their network by combining local patch similarity constraints with spatial geometric constraints on the points of interest, a more modern technique that significantly enhances matching.

As mentioned in the CNN-based detectors, an increasing number of end-to-end learning methods integrate the feature description together with the detectors into the complete matching pipeline. These methods are like those that have been specifically designed for the description reviewed above. The main difference may lie in the method of training and the design of the entire network structure. The core challenge is to make the whole process differentiable and trainable. For example, LIFT uses an end-to-end CNN network to perform keypoint detection, direction estimation, and characterization simultaneously. To train interest point detectors and descriptors for multi-view geometric problems, SuperPoint (DeTone et al., 2018) proposes a self-supervised framework for training interest point detectors and descriptors for multiple-view geometrical problems. The fully convolutional model can operate on full-size images and simultaneously compute the positions of pixel-level interest points and their associated descriptors.

LF-Net (Ono et al., 2018) confines the end-to-end pipeline to a single branch to optimize the entire process in a differentiable manner. It also employs a fully convolutional network that operates on full-sized images to produce a feature-rich score map. This map can then be used to extract keypoint locations as well as their feature attributes, such as scale and orientation. Additionally, it performs a differentiable form of Non-Maximum Suppression (NMS), namely, softargmax, to improve subpixel location accuracy and enhance keypoint saliency. Similar to LF-Net, RF-Net (Shen et al., 2019) selects high-response pixels as keypoints on multiple scales, but the response maps are constructed by receptive feature maps.

However (Bhowmik et al., 2020), suggest that improving low-level matching scores does not necessarily lead to better performance in high-level vision tasks. Therefore, they embedded the feature detector in a complete vision pipeline, where the learnable parameters are trained in an end-to-end manner. The authors address the discrete nature of keypoint selection and descriptor matching by applying principles from reinforcement learning. In 2020 (Luo et al., 2020), presented ASFeat, which employs joint learning of local feature detectors and descriptors to explore local shape information of feature points and improve the accuracy of point detection. Another learning-based method for detection involves estimating orientation (Yi et al., 2016), while the spatial transformation network (STN) (Jaderberg et al., 2015) can also serve as a valuable reference for deep learning-based detectors with rotation invariance (Ono et al., 2018; Yi et al., 2016).

Materials and methods

As shown in Figure 2, we present a detector-free local feature matching framework with transformers and semantics, called LoFTRS, which incorporates semantic constraints into the matching process. LoFTRS consists of two main branches: image segmentation module and detector-free transformer matching module.

Figure 2. Incorporating semantic constraints to Transformer-based local feature matching.

Image segmentation module

In the image segmentation module, we followed (Li et al., n.d.) to uses the same detection architecture design as DINO with only minimal modifications. In the transitive decoder, Mask DINO adds a mask branch for segmentation and extends several key components used in DINO for the segmentation task. Mask DINO predicts boxes and masks with two parallel heads in a loosely coupled manner, like some traditional modulators. However, this approach can result in inconsistent predictions. To address this issue, we added a mask prediction loss to the original box and classification loss in two-part matching. This encourages more accurate and consistent matching results for a query.

Detector-free transformer module

In this work, we present PE-LoftrMatcher, a Transformer-based deep learning module that builds upon our investigation of local feature matching in detector-free methods. As shown in Figure 3, PE-LoftrMatcher has designed a deep-narrow Transformer layer to capture more human-intuitive and simpler-to-match features. Additionally, position encoding (PE) has been integrated into each Transformer layer to convey position information in deep layers. Moreover, a network-based refinement block is proposed to extract more precise matches. Our key insight is that local feature matching with deep layers can capture more human-intuitive and simpler-to-match features. The detailed network architecture of PE-LoftrMatcher is shown in Figure 4, PE-LoftrMatcher interleaves Slimming Transformer by L times to perform long-range context aggregation. Slimming Transformer leverages vector-based attention to model relevance among all keypoints and achieves long-range context aggregation efficiently and effectively.

Figure 3. PE-LoftrMatcher framework flowchart.

Figure 4. Slimming transformer framework diagram.

We flatten the updated enhanced features $F_A^{ftm}$， $F_B^{ftm}$ be the input sequence for deep feature aggregation, obtaining $F_A^{seq},F_B^{seq}\in R^{\mathit{N}\times \mathit{C}}$. Following (Sarlin et al., n.d.), we view keypoints with features $F_A^{seq}$，$F_B^{seq}$ in image pairs as nodes to construct GNN, in which the global context aggregation intra-/inter-image is performed.

Vector-based attention (VAtt) layer

Instead of approximating self-attention in a context-independent way, we transform the query vector into a global query context and use element products to model the correlation between all key points. Technically, during each functional enhancement, we utilize self- or cross-attention to aggregate long-term contextual information. For self-attention, the input has intra- and inter-image features. U and R are identical (either $\left({\mathrm{F}}_{\mathrm{A}}^{\mathrm{s}\mathrm{e}\mathrm{q}},{\mathrm{F}}_{\mathrm{A}}^{\mathrm{s}\mathrm{e}\mathrm{q}}\right)$/or $\left({\mathrm{F}}_{\mathrm{B}}^{\mathrm{s}\mathrm{e}\mathrm{q}},{\mathrm{F}}_{\mathrm{B}}^{\mathrm{s}\mathrm{e}\mathrm{q}}\right)$). For cross-attention, the input features U and R are different (either $\left({\mathrm{F}}_{\mathrm{B}}^{\mathrm{s}\mathrm{e}\mathrm{q}},{\mathrm{F}}_{\mathrm{B}}^{\mathrm{s}\mathrm{e}\mathrm{q}}\right)$ or $\left({\mathrm{F}}_{\mathrm{A}}^{\mathrm{s}\mathrm{e}\mathrm{q}},{\mathrm{F}}_{\mathrm{A}}^{\mathrm{s}\mathrm{e}\mathrm{q}}\right)$). First, SlimFormer converts the input features U and R into queries, keys, and values $\text{Q},\text{K},\text{V}\in {\text{ R}}^{\text{N}\times \widehat{\text{C}}}$.

(1) $\mathrm{Q}=\mathrm{U}{\mathrm{W}}_{\mathrm{Q}},\mathrm{K}=\mathrm{R}{\mathrm{W}}_{\mathrm{K}},\mathrm{V}=\mathrm{R}{\mathrm{W}}_{\mathrm{V}}$(1)

where ${\text{W}}_{\text{Q}},{\text{W}}_{\text{K}},{\text{W}}_{\text{V}}\in {\text{ R}}^{\widehat{\text{C}}\times \widehat{\text{C}}}$ denotes the learnable weight of the feature transformation. Then, we encode the relative position of the query vector Q and the key K.

(2) $\tilde{\mathrm{Q}}=\mathrm{DPE}\left(\mathrm{Q}\right),\tilde{\mathrm{K}}=\mathrm{DPE}\left(\mathrm{K}\right)$(2)

where $\mathrm{DPE}\left(\cdot \right)$ denotes the relative position encoding operation as follows: Next, modeling the contextual information of input features based on $\tilde{\mathrm{Q}},\tilde{\mathrm{K}}$ interactions, V, is a key issue in the transformer-like architecture. In the original Transformer, the dot-product attention mechanism leads to quadratic complexity, making it impractical to build a deer Transformer layer. One potential way to reduce computational complexity is to summarize the attention matrix before modelling its interactions.

We introduce vector-based attention that effectively models remote interactions between pixel tokens to alleviate this bottleneck. Instead of computing a quadratic attention graph that encodes all possible interactions between candidate matches, we form a compact representation of the query key interactions by computing the correlation between the global query vector and each key vector based on the attention of the vector $\mathrm{Q}{\mathrm{K}}^{\mathrm{T}}$. Specifically. We first use MLP to compute the weight ${\tilde{\mathrm{Q}}}_{\mathrm{imp}}\in {\mathrm{R}}^{1\times \mathrm{N}}$ for each query vector:

(3) ${\tilde{\mathrm{Q}}}_{\mathrm{imp}}=\mathrm{Soft}max\left(\mathrm{MLP}\left(\tilde{\mathrm{Q}}\right)\right)$(3)

where $\mathrm{Soft}max\left(\cdot \right)$ denotes the softmax operation

The global query vector $\mathrm{Q}\in {\mathrm{R}}^{1\times \tilde{\mathrm{C}}}$ is set to a linear combination of $\tilde{\mathrm{Q}}$:

(4) $\text{Q}={\tilde{\text{Q}}}_{\text{imp}}\otimes \tilde{\text{Q}}$(4)

where $\otimes$ denotes matrix multiplication

We then use elementwise multiplication between the global query vector Q and each of the key vectors to model their interaction and obtain the context-aware key vector ${\tilde{\mathrm{K}}}_{\mathrm{Q}}\in {\mathrm{R}}^{\mathrm{N}\times \hat{\mathrm{C}}}$.

${\tilde{\mathrm{K}}}_{\mathrm{Q}}=\mathrm{Q}\odot \tilde{\mathrm{K}}$

$K_Q={\tilde{K}}_{\mathit{Qimp}}\otimes {\tilde{K}}_{\mathit{Q}}$

(5) $\mathrm{\Lambda }={\mathrm{K}}_{\mathrm{Q}}\odot \mathrm{V}$(5)

Subsequently, we use the MLP and shortcut structures to derive global messages $\mathrm{M}\in {\mathrm{R}}^{\mathrm{N}\times \tilde{\mathrm{C}}}$.

(6) $\mathrm{M}=\mathrm{MLP}\left(\mathrm{\Lambda }\right)+\tilde{\mathrm{Q}}$(6)

For convenience, we define the process of the vector-based attention layer as follows:

(7) $\mathrm{M}=\mathrm{VAtt}\left(\mathrm{U},\mathrm{R}\right)$(7)

Feed-forward network (FFN)

Inspired by conventional Transformers, we employ a feed-forward network applied to M to extract discriminative features for effectively deep features aggregation. The feed-forward network consists of two fully connected layers and a GELU activation function. The hidden dimension between the two fully connected layers is extended by a scale rate $\mathit{\gamma }$ to learn abundant feature representation. This process can be formulated as:

(8) $\mathit{FFN}\left(\mathit{U},\mathit{M}\right)=\mathit{ML}{P}_{1/\gamma }\left(\mathit{GELU}\left(\mathit{ML}{P}_{\mathit{\gamma }/3}\left(\left[\mathit{U}||\mathit{M}\right]\right)\right)\right)$(8)

Layer scale strategy

Intuitively, people obtain different message after observing images each time, which inspires us to propose a layer-scale strategy. Specifically, in accordance with ResNet (He et al., n.d.), we utilize a shortcut structure to realize. Then, we design a learnable scaling factor $\mathit{\xi }$ to adaptively balance original features U and enhanced message $\tilde{M}$, which is formulated as.

(9) $\tilde{U}=\mathit{U}+\mathit{\xi }\tilde{M}$(9)

By incorporating $\mathit{\xi }$ into SlimFormer, SlimFormer can easily simulate the human behavior that humans acquire different matching cues each time they scan an image pair.

Self-/cross-SlimFormer

In summary, the SlimFormer is formatted as:

(10) $\mathit{S}lim\left(\mathit{U},\mathit{R}\right)=\mathit{U}+\mathit{\xi FFN}\left(\mathit{U},\mathit{VAtt}\left(\mathit{U},\mathit{R}\right)\right)$(10)

We perform L times of SlimFormer for feature enhancement. During the l-th feature enhancement, we use self-/cross attention mechanism to integrate intra-/inter-image information.

Evaluation pipeline

This study follows the basic workflow of SfM-based image reconstruction, which is divided into two parts. In the first part, feature extraction is performed on each UAV image, initial matching is obtained from each pair of images, and then RANSAC is used to remove outliers. According to the workflow of feature matching, the feature evaluation is divided into two steps, which are feature extraction and feature matching. In the second part, the verified matches are fed into the classical SfM pipeline for sparse reconstruction to obtain accurate camera poses and scene structures. Subsequently, the results from sparse reconstruction are used to perform dense reconstruction.

Table 1 presents a summary of the advantages and disadvantages of the classical manual methods used in this paper.

Table 1. Comparison of commonly used handcrafted methods: SIFT, SURF, ORB, and AKAZE.

Method	Advantage	Disadvantage
SIFT	Scale invariant Rotation invariant Invariant to illumination changes	Computationally expensive Not suitable for real-time applications
SURF	Faster and more efficient than SIFT Scale invariant Rotation invariant Invariant to illumination changes	Loss accurate than SIFT Not suitable for Low-contrast images
ORB	High quantity of feature & matches Computationally efficient Invariant to scale and rotation	Less accurate than SIFT & SURF Not suitable for large-scale image matching
AKAZE	Computationally efficient Invariant to scale, rotation & illumination charges	Less accurate SIFT & SURF Not suitable for large-scale image matching Less robust and less stable

According to the pipeline, certain key metrics are chosen and utilized to evaluate the performance of detectors and descriptors, as outlined in Table 2. These metrics are categorized into two groups, where matching represents the feature extraction and matching procedure, and reconstruction represents the SfM-based sparse reconstruction and the MVS-based dense reconstruction.

Table 2. Description of evaluation matches.

Category	Name	Metric	Description
Matching	Matching score	M_s	The ratio between the numbers of verified inliers and initial features extracted from images (larger value indicates better results)
	Match pairs of distribution intervals	Mp_s	The number of correspondences/matching keypoints at different intervals (larger value indicates better results)
	Matching time	M_t	Time costs for feature matching (smaller value indicates better results)
Reconstruction	Average track length	Atr	The number of average image views that features are successfully matched and tracked (larger value indicates better results)
	Number of connected images	Img	The number of registered images after sparse reconstruction (larger value indicates better results)
	Number of sparse points	S_pts	The number of 3D points after sparse reconstruction (larger value indicates better results)
	RMSE of re-projection error	Error	The root mean squared error of reprojection error, which is a distance metric in pixels corresponding to the image distance between projected points and measured points. (Smaller value indicates better results)
	Number of dense point clouds	D_pts	The number of 3D points after dense reconstruction (larger value indicates better results)
	Reconstruction time	Time	Time costs for reconstruction (smaller value indicates better results)

Feature extraction and matching

Not only did we want to use a comprehensive set of metrics that would allow us to embed the analysis into existing work, but we also wanted to evaluate the parameters associated with feature-dependent algorithms. Therefore, we propose 3 different metrics: matching score, matching time and interval distribution of the number of matching pairs. This is shown in Figure 5.

Figure 5. Main methods and evaluation metrics of feature extraction and matching.

Matching score

The matching score indicates the ratio between the numbers of verified inliers and initial features extracted from images. It describes the number of initial features that will lead to a correct match; the matching score may be affected by fuzzy descriptors and matching criteria:

(11) $\mathrm{Matching}\mathrm{score}=\frac{\mathrm{Correct}\mathrm{Matches}}{\mathrm{Features}}$(11)

Overall, the matching score describes the performance of the descriptor and is influenced by the robustness of the descriptor to transformations present in the data.

Interval distribution of the number of matching pairs: Statistics on distribution intervals of the number of matched pairs of different methods on the same data can reflect the feature extraction and matching performance of different methods more intuitively.

Matching time

The time spent by an image in the matching process is equivalent to the sum of the time costs spent in feature extraction and feature matching. The total time spent on all matching pairs is recorded here.

Sparse and dense reconstruction

After image matching, the processes of sparse and dense reconstruction have been evaluated as well, respectively. As shown in Figure 6, six separate metrics are adopted to evaluate the accuracy, and completeness of reconstruction results using different image matching methods.

Figure 6. Sparse reconstruction process and evaluation indicators.

Re-projection error. For sparse point cloud reconstruction, we can judge the accuracy of relative positioning by calculating the reprojection error. The smaller the reprojection error, the more accurate the influence matching algorithm is and the more accurate the relative positioning is, which allows us to reconstruct a model with higher accuracy. There is always a distance between the image position computed from the camera’s projection matrix and the actual image position for each 3D point after calibration. We sum up the errors to construct a least squares problem and then find a good camera position to minimize it:

(12) $\underset{B_j,A_i}{\mathit{min}}{\sum }_{i=1}^n{\sum }_{j=1}^m{p}_{\mathit{ij}}\parallel \mathit{P}\left(B_j,A_i\right)-a_{\mathit{ij}}{\parallel }^2$(12)

Where $A_i$ and $B_j$ denote a 3D point and a camera, respectively; $\mathit{P}\left(B_j,A_i\right)$ is the predicted projection of point $A_i$ on camera $B_j$; $a_{\mathit{ij}}$ is the observed image point; $||\bullet ||$ denotes the L2-norm; $p_{\mathit{ij}}$ is an indicator function with $p_{\mathit{ij}}$ = 1 if point $A_i$ is visible in camera $B_j$; otherwise, $p_{\mathit{ij}}$ = 0.

The reprojection error considers not only the computational error between the single response matrices but also the measurement error of the image, making it suitable for evaluating sparse reconstruction results.

Average track length

Evaluating the average number of images for a 3D point is often a way of assessing the quality and reliability of a 3D reconstruction; the more times a 3D point is observed in multiple images, the higher its average number of images will be. This indicates that the point has multiple observations in different images, which can increase the accuracy and stability of the 3D point. The average image count can also reflect the sparsity of the 3D point cloud. If a point is observed in only a few images, it may mean that there is a lack of sufficient viewpoints in certain areas.

Number of registered images

This number can affect the quality of the results of the task, the efficiency of the computation, and the applicability of the algorithm. When using multi-view images for 3D reconstruction, the number of registered images determines how many viewpoints from which a particular 3D point can be observed. More registered images mean more information for reconstruction, which usually leads to a more accurate and complete 3D model.

Number of sparse point clouds

For sparse reconstruction, the number of sparse points will be directly affected by the accuracy and completeness of feature matching, which can intuitively respond to the feature detector and descriptor performance. Therefore, the completeness of the image is evaluated by the number of sparse points in this assessment.

Number of dense point clouds

For dense reconstruction, the number of dense points will be affected by the accuracy and completeness of the sparse reconstruction, which is further affected by the performance of the feature detectors and descriptors. Therefore, the number of dense points is used in this evaluation.

Reconstruction time

The time spent by the image in the reconstruction process, including the sparse reconstruction time and the dense reconstruction time.

By using the above metrics, the efficiency of feature learning, accuracy, and completeness of 3D reconstruction can be comprehensively assessed.

Results

Datasets

All four datasets were collected with a UAV platform, as described in Figure 7.

Figure 7. Images of four UAV scene (a) scene 1 (b) scene 2 (c) scene 3 (d) scene 4.

Six configurations were constructed for the four scenes. Data 1 and Data 4, consisting mainly of buildings, are repetitive textured data generated from Scene 1. The former is down-sampled data, while the latter is image patch data. Data 2, an ISPRS public dataset (Rottensteiner, n.d.), was generated from Scene 2 and contains a large number of plants and several buildings. Data 3 is generated from Scene 3. It contains a library building with relatively consistent lighting conditions and small rotation angles between images. Data 5 and Data 6 are generated from Scene 4, the public SWJTU_BLD (Zhu et al., n.d.) dataset. Data 5 is Ground-UAV data, whereas Data 6 is mixed Ground-UAV and UAV-UAV data. The specific data are described in Table 3.

Table 3. Description of the UAV test data.

Number of scenes	Data	Description for data	Left image (Pixels)	Right image (Pixels)
1	1	Down-sampled data for Scene 1; repetitive textured buildings	1090 × 725	1090 × 725
	4	Image patch data for Scene 1; repetitively textured buildings	1300 × 980	1300 × 980
2	2	Data for Scene 2; large clusters of plants and small clusters of buildings	1260 × 1260	1260 × 1260
3	3	Data for Scene 3; relatively consistent lighting conditions with small rotation angles between images	1300 × 1300	1300 × 1300
4	5	Ground-UAV data for Scene 4; open source SWJTU_BLD data	1260 × 1260	1260 × 1260
	6	Scene 4 Ground-UAV and UAV-UAV datasets; open source SWJTU_BLD data	1260 × 1260	1260 × 1260

For performance evaluation, all reconstruction experiments were performed on a Windows PC with a 3.6 GHz Intel Core i9-9900K CPU and a 4GB NVIDIA GeForce RTX 2080 Ti graphics card.

The deep learning matching method was instead performed on an Ubuntu PC with a 2.2 GHz Intel Xeon E5–2630 CPU and a 12GB NVIDIA GTX 1080 Ti graphics card.

In general, learned descriptors can be further enhanced by training them with the UAV data. In this evaluation, we did not do this for two main reasons. First, existing benchmark datasets, such as Brown, feature wide-baseline images, including viewpoint and illumination variations. They also represent common features of UAV images. Second, some CNN models rely on synthetic datasets for pre-training intermediate sub-models. In addition, different learning models require different data formats, e.g. patch-based, and image-based datasets. For descriptor comparability, pre-trained models and their source code releases were used without retraining. This can be used to verify their adaptation to UAV images. It is worth noting that the default parameters of all the chosen methods were used to make the evaluation as fair as possible. However, there are many reasons for the failure of the reconstruction, such as non-optimized parameter configurations and unsuitable datasets. In other words, we only consider their performance on UAV images without further parameter tuning.

Also, to ensure data consistency, we chose to use the inputs of all methods in the form of images instead of the traditional faceted inputs, and we resized all datasets for all methods to work properly. For the method we used, it is shown in Table 4.

Table 4. Detailed information on the assessment methodology.

Method	Descriptor length	Platform	URL
SIFT	128	CPU	https://github.com/openMVG/openMVG/tree/develop/src/openMVG/features
AKAZE	128	CPU	https://github.com/openMVG/openMVG/tree/develop/src/openMVG/features
ORB	256	CPU	https://docs.opencv.org/3.4/d1/d89/tutorial_py_orb.html
SURF	64	CPU	https://docs.opencv.org/3.4/df/dd2/tutorial_py_surf_intro.html
CapsSIFT	128	GPU	https://github.com/qianqianwang68/caps/tree/master/CAPS
CapsSuperPoint	256	GPU	https://github.com/qianqianwang68/caps/tree/master/CAPS
DogAffnet-Hard	128	CPU	https://github.com/ducha-aiki/affnet
R2D2	128	GPU	https://github.com/m-hildebrandt/R2D2
D2Net	512	GPU	https://github.com/mihaidusmanu/d2-net
Patch2Pix	256	GPU	https://github.com/GrumpyZhou/patch2pix
LoFTR	128	GPU	https://github.com/zju3dv/LoFTR

Comparison and evaluation of matching results

To verify the robustness and accuracy of the proposed matching methods, we use the six datasets derived from the above four UAV datasets for point matching using the 12 methods listed in Table 5, at the same time, it is visualized by histograms, as shown in Figure 8:

Figure 8. Comparison of matching scores datasets.

Table 5. Matching score.

Method	Matching score
	Data 1	Data 2	Data 3	Data 4	Data 5	Data 6
SIFT	0.0717	0.0803	0.0651	0.1254	0.0125	0.0444
AKAZE	0.0515	0.0488	0.039	0.1	0.005	0.0357
ORB	0.0177	0.0172	0.0113	0.0792	0.01	0.009
SURF	0.0892	0.0543	0.0587	0.2596	0.1337	0.0577
CapsSIFT	0.2819	0.2188	0.2477	0.2901	0.1219	0.1924
CapsSuperPoint	0.29	0.2402	0.2451	0.3365	0.1293	0.2152
DogAffnet-Hard	0.4146	0.1521	0.1361	0.2049	0.0501	0.1147
R2D2	0.0691	0.078	0.0658	0.1309	0.0244	0.0781
D2Net	0.2341	0.2255	0.2084	0.2691	0.1497	0.2040
Patch2Pix	0.0395	0.0247	0.0228	0.1738	0.0389	0.0132
LoFTR	0.0516	0.0378	0.0318	0.189	0.0382	0.0229
Ours	0.2318	0.1228	0.1377	0.295	0.0933	0.1445

Matching score

The performance of detectors and descriptors is first evaluated by feature matching. Matching scores are used as a metric for comparison. The results are displayed in Table 5, where the first, second, and third-ranked values are shown in red, blue, and purple, respectively. The table shows that different methods have unique approaches to cope with complex and realistic scenarios under different data conditions, resulting in superior matching scores. This illustrates the importance of selecting the appropriate method for the specific data being analysed. In data 1, DogAffnet-Hard, CapsSuperPoint, and CapsSIFT achieved better matching scores of 0.4146, 0.2, and 0.2819, respectively. This suggests that the deep learning approach is better at feature detection for simple single building than the classical approach, resulting in more match pairs. However, further investigation is needed to determine if this translates to better reconstruction results. For data 2 with a high amount of vegetation coverage, CapsSuperPoint, D2Net, and CapsSIFT achieved the highest scores with 0.2402, 0.2255, and 0.2188, respectively. It is evident that for textures with high repetition, all methods exhibit varying degrees of score decline. This highlights the limitations of existing feature detection methods on repetitive textures. However, when compared to classical methods, existing feature detection methods are less effective in detecting features on repetitive textures.

The limitations of current feature detection methods on repetitive textures are highlighted, with deep learning methods outperforming classical methods in terms of detection results and feature pairs. In the case of Data 3, which contains numerous buildings and road clusters, all detection methods struggle with the same issue of repetitive texture detection. This suggests that existing feature detection methods lack the necessary discrimination for large building and road networks. However, CapsSIFT, CapsSuperPoint, and D2Net achieved higher scores on data 3 with 0.2477, 0.2451, and 0.2084, respectively. In data 5 and 6 of the ground-space association, the effectiveness of all detection methods decreased, although the deep learning methods performed better. By comparing the matching results of these datasets, it is evident that the deep learning approach outperforms the classical approach in scenes of higher complexity and is better suited to realistic scenes.

Average track length

The statistical results of average track length are listed in Table 6. A visualized histogram is shown in Figure 9. We can conclude that when considering the metrics Tie_pt, SIFT and its two variants, RootSIFT and DspSIFT are the best descriptors among all evaluated methods. This means that these three methods can provide the largest and strongest matches for the subsequent SfM reconstruction. This result, combined with the sparse point cloud 3D points (number of sparse points), shows that SIFT does not reconstruct more sparse 3D points, although it can provide the largest and strongest matches for the subsequent SfM reconstruction. However, AKAZE provides basically complete sparse reconstruction with more points than SIFT as well as large and strong matches, since AKAZE can produce more sparse 3D points and a long average track length in data 1, data 2, data 3, and data 6. When considering interval distribution of the number of match pairs, SIFT has the largest number of corresponding feature points in the 0–100 interval, followed by the 100–500 interval; correspondingly, AKAZE has the largest number of corresponding feature points in the 100–500 interval, followed by the 0–100 interval; and for the images in the UAV dataset used in this paper, the matching result with the number of corresponding feature points of 100–500 has the greatest impact on the reconstruction, and the 100–500 corresponding feature points between the images can both ensure that there are enough 2D correspondences, and also build a more complete point cloud model without consuming a lot of computational resources.

Figure 9. Average track length.

Table 6. Average track length.

Method	Mean track length
	Data 1	Data 2	Data 3	Data 4	Data 5	Data 6
SIFT	5	5	3	4	3	4
AKAZE	4	4	4	3	3	3
ORB	2	2	2	2	-	2
SURF	2	2	2	2	-	2
CapsSIFT	0	-	-	2	-	-
CapsSuperPoint	2	-	-	2	2	-
DogAffnet-Hard	2	-	-	3	-	-
R2D2	2	2	2	2	-	-
D2Net	0	2	2	2	2	-
Patch2Pix	2	2	2	2	2	-
LoFTR	2	4	2	2	-	-
Ours	3	3	4	3	2	-

Qualitative image-matching results

The above shows the qualitative image matching results of Data 1 and Data 6, which can visualize the matching performance of different methods. The first column and the second column in Figures 10 and 11 represent different image pairs, showing the same and different sides of the building, respectively. The qualitative analysis of the matching results shows that the classical method has fewer matching lines and is more focused on the main building, resulting in fewer outliers. However, due to the low quantity of feature correspondences, it is difficult to obtain more sparse 3D points. On the other hand, LoFTRS produces a large number of feature correspondences with high accuracy. When compared to CapsSuperPoint, D2Net, and LoFTR, LoFTRS produces fewer but more accurate correspondences. In comparison to SIFT, SURF, ORB, AKAZE, and R2D2, LoFTRS generates a significantly greater number of correspondences. This enables the acquisition of more sparse 3D points and leads to a more comprehensive 3D model during the reconstruction process. Taking both accuracy and robustness into consideration, LoFTRS achieves the best performance among both handcrafted and deep learning-based approaches.

Figure 10. Data 1 qualitative image matching results.

Figure 11. Data 6 qualitative image matching results.

Distribution of match pairs

The distribution of matching pairs of different methods can be by counting the number of matching pairs of each method. Most of the classical methods are concentrated in less than 500 pairs, and most of the learning methods are concentrated in more than 500 pairs. Deep learning methods can obtain deeper features of the image through convolution, thus obtaining more matching features than classical methods. Depending on the number of corresponding feature points finally matched, we categorize the results with different numbers of corresponding feature points into four intervals: 0 ~ 100, 100 ~ 500, 500 ~ 1000, and above 1000.

Data 1 and Data 4 are characterised by consistent lighting conditions, large rotation angles between images, less than 50% overlap, high noise, and extreme similarity between different sides of the building. This can result in more outliers during feature matching. Based on experimental results in Table 7 and Figure 12, AKAZE, LoFTRS, SURF, SIFT, Patch2Pix, and LoFTR are advantageous methods for UAV data with low overlap. In addition, LoFTRS are evenly distributed in every interval, showing that LoFTRS is robust for UAV building data with consistent lighting condition, large rotation angles and extreme similarity between different sides of the building.

Table 7. Interval distribution of the number of match pairs.

Method	Data 1				Data 2				Data 3
	0 ~ 100	100 ~ 500	500 ~ 1000	>1000	0 ~ 100	100 ~ 500	500 ~ 1000	>1000	0 ~ 100	100 ~ 500	500 ~ 1000	>1000
AKAZE	278	109	29	35	824	196	111	91	696	227	71	148
CapsSIFT	0	412	7	0	4	1036	2	0	9	1112	21	0
CapsSuperPoint	0	0	233	186	0	257	684	181	0	11	615	516
D2Net	0	0	0	419	0	0	0	1207	0	0	4	1138
R2D2	0	298	55	66	0	830	78	299	0	887	88	167
ORB	396	18	5	0	959	83	0	0	1093	38	9	2
SURF	183	185	26	25	803	142	48	49	94	888	67	101
SIFT	285	81	27	16	519	186	103	48	493	187	76	76
Patch2Pix	15	243	65	96	60	692	77	213	66	783	79	214
LoFTR	171	91	48	109	733	21	38	257	538	315	26	265
DogAffnet-Hard	312	107	0	0	663	374	5	0	877	243	22	0
Ours	93	121	54	120	200	161	169	187	458	89	74	321
	Data 4				Data 5				Data 6
Method	0 ~ 100	100 ~ 500	500 ~ 1000	>1000	0 ~ 100	100 ~ 500	500 ~ 1000	>1000	0 ~ 100	100 ~ 500	500 ~ 1000	>1000
AKAZE	369	510	221	93	229	0	0	0	2122	1184	183	79
CapsSIFT	37	1136	20	0	314	418	0	0	680	3805	9	0
CapsSuperPoint	0	444	486	63	0	377	355	0	0	865	2675	954
D2Net	0	0	12	1181	0	1	20	0	0	0	47	4447
R2D2	0	887	8	167	11	721	0	0	0	2324	716	1433
ORB	1001	172	20	0	732	0	0	0	1093	174	1	0
SURF	139	794	200	60	138	594	0	0	94	717	90	48
SIFT	306	600	108	19	632	0	0	0	493	1202	173	51
Patch2Pix	148	241	176	631	668	64	0	0	66	1281	66	1043
LoFTR	238	150	100	705	468	264	0	0	538	652	322	1814
DogAffnet-Hard	460	718	15	0	732	0	0	0	877	1240	16	0
Ours	166	334	384	256	62	270	316	711	458	1171	1226	772

Figure 12. Distribution of intervals for each data.

Six configurations were constructed for the four scenes. Data 1 and Data 4, consisting mainly of buildings, are repetitive textured data generated from Scene 1. The former is down-sampled data, while the latter is image patch data. Data 2, an ISPRS public dataset (Rottensteiner, n.d.), was generated from Scene 2 and contains a large number of plants and several buildings. Data 3 is generated from Scene 3. It contains a library building with relatively consistent lighting conditions and small rotation angles between images. Data 5 and Data 6 are generated from Scene 4, the public SWJTU_BLD (Zhu et al., n.d.) dataset. Data 5 is Ground-UAV data, whereas Data 6 is mixed Ground-UAV and UAV-UAV data.

Data 2 comprises a significant number of plants and multiple buildings. Matching features for image datasets that contain green plants has proven to be a challenging task. AKAZE, LoFTRS, SURF, SIFT, Patch2pix, and LoFTR are methods that can obtain corresponding feature points at different intervals in the matching result. Based on the experimental results, LoFTRS is evenly distributed in every interval, indicating its robustness for UAV building data with many plants and several buildings. Additionally, LoFTRS produces the highest number of correspondences in intervals 100–500 and 500–1000 simultaneously, which is advantageous for 3D reconstruction.

Data 3 contains a library building with relatively consistent lighting conditions and small rotation angles between images. The dataset exhibits consistent lighting conditions, small rotation angles between images, and with over 80% overlap. Based on the experimental results, LoFTRS is evenly distributed in every interval, indicating its robustness for UAV building data with consistent lighting conditions, small rotation angles between images.

Data 5 is Ground-UAV data, whereas Data 6 is mixed Ground-UAV and UAV-UAV data. It is important to note that the SWJT ground-view images were not inherently acquired on the ground; they were also captured by a low-cost drone flying in vertical lift. However, because they have similar characteristics to other ground-view images, they are also considered ground-view images. Based on the experimental results, only LoFTRS is evenly distributed in every interval, indicating its robustness for Ground-UAV and UAV-UAV data. Additionally, LoFTRS produces the highest number of correspondences in intervals 100–500 and 500–1000 simultaneously, which is advantageous for 3D reconstruction.

Efficiency

When analysing matching results, it is essential to consider the time spent. Table 8 displays the matching time statistics for the six datasets. Figure 13 illustrates the results of the visualization. Generally, as the data volume increases, the time spent by each method also increases. However, it is important to note that the time spent is affected by factors other than the amount of data, such as image size, parameter settings, and operating platforms. Compared to traditional classical methods, deep learning methods generally require more time. This is likely due to the time-consuming process of acquiring feature points and using pixels for convolution to obtain features. For small volumes of single buildings, AKAZE and ORB achieved the best time results using the traditional method, possibly due to the acceleration algorithm employed by these two methods. However, in the joint ground-air scenario, ORB and SURF demonstrate better time efficiency. Among the deep learning methods, LoFTRS requires the least amount of time and is more efficient than SIFT for Data 1.

Figure 13. Comparison of matching times datasets.

Table 8. Matching time.

Method	Matching time
	Data 1	Data 2	Data 3	Data 4	Data 5	Data 6
SIFT	259	179	250	496	71	191
AKAZE	43	147	231	774	69	293
ORB	50	189	204	286	-	149
SURF	131	346	925	482	-	229
CapsSIFT	1106	1106	1496	2732	1444	8615
CapsSuperPoint	188	461	542	836	461	2623
DogAffnet-Hard	8545	3341	3704	5547	2946	16454
R2D2	637	2661	3216	6482	2854	13605
D2Net	577	2674	2966	5561	1892	11954
Patch2Pix	628	4101	4756	5870	3554	19009
LoFTR	233	6136	1382	1411	580	3607
Ours	156	564	1321	3961	451	2515

Comparison of sparse reconstruction

Number of Sparse Points: The results are presented in Table 9, the visualization is shown in Figure 14 with the top three values highlighted in red, blue, and purple. For the simple monolithic building data 1, R2D2, AKAZE, and SURF obtained the largest number of sparse points, with 25,577, 12788, and 10,188, respectively. In general, handcrafted methods are better suited for obtaining sparse points on this type of data and linking pairs of points based on obtaining a sufficiently large number of sparse points. On the other hand, the matched pairs acquired by deep learning lack sufficient relevance, resulting in fewer sparse points. For Data 2 with high vegetation coverage and repetitive textures, SIFT, AKAZE, and LoFTRS obtain a better number of sparse points than the other methods, which are 28,759, 21949, and 18,843, respectively. For Data 4, LoFTRS, LoFTR and SIFT obtain the largest number of sparse points than the other methods, which are 76,516, 63184, and 31,602, respectively.

Table 9. Number of sparse points and reprojection error.

Method	Sparse points						Reprojection Error
	Data 1	Data 2	Data 3	Data 4	Data 5	Data 6	Data 1	Data 2	Data 3	Data 4	Data 5	Data 6
SIFT	4748	28759	15230	31602	450	24045	0.8259	0.7051	1.075	0.9277	1.0736	0.9640
AKAZE	12788	21949	32708	27476	465	13772	0.6119	0.6499	0.6379	0.7827	1.0539	0.8179
ORB	5522	5300	1185	5619	-	4770	0.6119	0.5732	0.4495	0.4384	-	0.8621
SURF	10188	1236	7672	4157	-	41939	0.3991	0.669	0.967	0.5666	-	0.7465
CapsSIFT	-	-	-	24	-	-	-	-	-	0.7232	-	-
CapsSuperPoint	35	-	-	1036	12	-	0.3199	-	-	0.6864	-	-
DogAffnet-Hard	70	-	-	268	-	-	0.6008	-	-	0.6161	-	-
R2D2	25577	22	29918	10395		-	0.53	0.8036	0.5415	0.5794	-	-
D2Net	24	118	18	1338	6	-	-	0.8773	0.5045	0.851	0.9240	-
Patch2Pix	2035	1572	107	73	57	-	0.4369	0.4028	1.5851	1.5119	-	-
LoFTR	9331	14	6670	63184	-	-	0.2626	0.4339	1.2558	0.4286	-	-
Ours	6402	18843	15174	76516	13	-	0.6551	0.6614	0.7566	0.9884	0.4970	-

Figure 14. Comparison of the number of sparse points in each data.

Reprojection error

Table 9 displays the reprojection error results for each method, with smaller values indicating greater accuracy and Figure 15 illustrates the results of the visualization. For Data 1, LoFTR, CapsSuperPoint and SURF achieve more accurate results, while for Data 3, ORB, D2Net and R2D2 achieve more accurate results. Finally, for Data 4, LoFTR, ORB and SURF achieve more accurate results. As the complexity of the data scenario increases, the deep learning-based approach can obtain feature points more accurately, resulting in more matches to some extent. In general, LoFTRS achieves more accurate sparse reconstruction than SIFT.

Figure 15. Comparison of reprojection errors for each data.

Number of registered images

The extent of image data included in this sparse point cloud is determined by the quantity of registered images. A more comprehensive reconstruction of the sparse point cloud is achieved with a greater number of registered images for the same dataset. The visualization is shown in Figure 16. Table 10 demonstrates that handcrafted methods perform well for Data 1, while only LoFTRS and R2D2 among the deep learning-based methods can reconstruct a complete point cloud. The results indicate that deep learning methods can generate more matching pairs for texture-similar data, but most of them are incorrect, resulting in poor performance in terms of the completeness of sparse reconstruction. Additionally, Data 4 can register more images than Data 1 for the same scene. By comparing the number of sparse points, it is evident that image patch data retains more image features than down-sampling data, particularly for deep learning-based LoFTRS. LoFTRS can generally register the same or a larger number of images compared to SIFT, as demonstrated in Data 1, Data 2, Data 3, and Data 4.

Figure 16. Number of registered images.

Table 10. Number of registered images.

Method	Register images
	Data 1	Data 2	Data 3	Data 4	Data 5	Data 6
SIFT	34	62	52	133	88	194
AKAZE	34	62	52	169	89	123
ORB	31	40	13	37	0	98
SURF	34	12	34	33	0	174
CapsSIFT	0	0	0	3	0	0
CapsSuperPoint	2	0	0	13	2	0
DogAffnet-Hard	4	0	0	14	0	0
R2D2	34	50	52	32	0	0
D2Net	2	2	2	10	2	0
Patch2Pix	2	2	2	2	2	0
LoFTR	4	34	16	9	0	0
Ours	34	62	54	320	2	0

Efficiency

Figure 17 illustrates the time spent on sparse reconstruction. Table 11 shows the time spent on sparse reconstruction. The reconstruction of the monolithic building data takes significantly longer than the matching process for the handcrafted method. In comparison to the handcrafted method, LoFTRS requires less time for single-building reconstruction. In general, LoFTRS achieves the best efficiency among the deep learning-based methods and it’s more efficient than AKAZE.

Figure 17. Comparison of sparse reconstruction time cost datasets.

Table 11. Sparse reconstruction time.

Method	Sparse time
	Data 1	Data 2	Data 3	Data 4	Data 5	Data 6
SIFT	220	66	179	165	110	719
AKAZE	19	35	69	513	216	325
ORB	8	416	12	87	-	151
SURF	6	96	166	75	-	235
CapsSIFT	-	-	-	7	-	-
CapsSuperPoint	12	-	-	34	-	-
DogAffnet-Hard	5	-	-	9	-	-
R2D2	114	190	114	569	-	-
D2Net	-	8	13	69	-	-
Patch2Pix	6	6	6	30	-	-
LoFTR	47	538	567	577	-	-
Ours	10	38	40	306	-	-

Comparison of dense reconstruction

Number of Dense Points: The number of dense points is significantly affected by the number of sparse points, as well as other factors such as the number of isolated points. It is important to note that some learned methods, such as Patch2Pix and LoFTR, do not perform well on sparse points, resulting in no output for dense reconstruction. Table 12 presents the results of dense points for all methods, with the top three ranked values highlighted in red, blue, and purple. Figure 18 illustrates one of the results of the visualization. When considering Data 1, Data 2, Data 3 and Data 4, SIFT and AKAZE achieve a greater number of dense points, while only LoFTRS achieves a large number of dense points among deep learning-based methods.

Figure 18. Comparison of the number of dense points for each data.

Table 12. Number of dense points.

Method	Dense points
	Data 1	Data 2	Data 3	Data 4	Data 5	Data 6
SIFT	3403030	2711649	2707365	3873160	114727	-
AKAZE	3228598	2822392	8748309	5348417	213406	-
ORB	1740738	66285	480717	699013	-	-
SURF	2508052	-	-	391870	-	14703455
CapsSIFT	-	-	-	-	-	-
CapsSuperPoint	-	-	-	-	-	-
DogAffnet-Hard	-	-	-	4524	-	-
R2D2	19960	-	40419	391208	-	-
D2Net	-	-	-	-	-	-
Patch2Pix	-	-	-	-	-	-
LoFTR	168	-	-	-	-	-
Ours	3328999	2824367	16945	1726468	-	-

Efficiency

The time of dense reconstruction is a big expense for each method. The results of the dense reconstruction for all methods are put in Table 13, the visualization is shown in Figure 19, while the methods with no dense results are not counted in time. As can be seen in the table, the traditional methods spend more time in this step, which may be since they acquire more isolated points and feature points, which takes more time in dense reconstruction. The time spent by LoFTRS is lower than that of the classical methods, excluding the learning methods that cannot obtain dense results.

Figure 19. Comparison of time cost of dense reconstruction for each data.

Table 13. Dense reconstruction time.

Method	Dense reconstruction time
	Data 1	Data 2	Data 3	Data 4	Data 5	Data 6
SIFT	1538	649	1826	559	156	-
AKAZE	599	1114	1111	1111	64	-
ORB	262	7	189	87	-	-
SURF	183	-	-	85	-	9102
CapsSIFT	-	-	-	-	-	-
CapsSuperPoint	-	-	-	-	-	-
DogAffnet-Hard	-	-	-	54	-	-
R2D2	1006	-	1006	63	-	-
D2Net	-	-	-	-	-	-
Patch2Pix	-	-	-	-	-	-
LoFTR	5	-	-	-	-	-
Ours	300	1048	617	1191	-	-

Conclusions

In this work, we present LoFTRS, a deep learning-based image matching framework that integrates semantic constraints into the matching process. We evaluate the performance of LoFTRS comparing to various popular handcrafted and deep learning-based methods. We investigate the relationship between matching quality and the performance of subsequent processing steps, such as the accuracy and completeness of the model generated by SfM. The experimental results show that the proposed LoFTRS achieves equal or better image matching performance in terms of matching score, average track length, RMSE, and the number of 3D points. In the future, we aim to investigate whether other deep learning-based methods can achieve comparable or superior performance to handcrafted methods after retraining.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

Due to the nature of the research, due to commercial supporting data is not available.