A cross-view intelligent person search method based on multi-feature constraints

ABSTRACT

Person searches aim to simultaneously locate and identify persons to be queried from different scenes, which is crucial for disaster emergency management and public safety. However, the high variability of environmental features in different video scenes, along with the susceptibility of people searching for occlusions or dense populations, results in existing methods suffering from inefficiency and poor accuracy in searching for cross-view persons. Therefore, we propose a cross-view intelligent person search method based on multifeature constraints. First, we establish the global-local context-aware (GLCA) module, which fully extracts the differential personnel features. Second, we construct the semantic complementarity and feature aggregation (SCFA) module for personnel-scale feature constraints in different contexts. Third, we constrain the method in terms of person spatial, person identity, and detection confidence features to improve person search accuracy. Finally, we construct a case experiment dataset, select two public benchmark datasets, and conduct a detailed experimental analysis based on them. The results show that our method can be applied to personnel search tasks in complex scenarios well, and the search results outperform those of 25 other state-of-the-art algorithms, with mAPs improved by 0.41%−19.71%. This approach effectively enhances the informatization level of disaster emergency rescue and public safety management.

KEYWORDS:

• Cross-view image
• intelligent person search
• multifeature constraints
• disaster management
• global-local context-aware

1. Introduction

Human activity information is crucial for building digital earth spaces and supporting sustainable development in the future (Che et al. 2023; Hu et al. 2021). In particular, with the rapid growth of virtual reality (VR), augmented reality (AR), and mixed reality (MR) technologies in recent years, computer-simulated human activity data have been used in various virtual simulations (e.g. disaster simulation, urban simulation, and construction simulation) to help us better understand and plan human activities (Li et al. 2024; Wu et al. 2023; Zhang et al. 2023). However, there is always an unbridgeable gap between reality and VR. Virtual human activity data may lead to inaccurate or unrealistic simulation results (Ray and Claramunt 2023; Zhu et al. 2024a), hindering the future development of urban and disaster digital twins (Therias and Rafiee 2023; Zhu et al. 2024b).

Person search is a technique for accurately obtaining actual human activity data. It consists of two phases, pedestrian detection and person reidentification, and aims to locate and identify individuals of interest within diverse scenes simultaneously. It can be used as an effective digital search tool for the timely collection of personal information from multiple surveillance videos and can assist in tasks such as social security maintenance, disaster emergency rescue, and urban digital twins (Qian et al. 2022; Singh, Khare, and Jethva 2022; Wang et al. 2023). Different from the traditional person reidentification method (Figure 1), person search encompasses two subtasks: pedestrian detection and person reidentification (Figure 2) (Fiaz et al. 2022). Paralleling these subtasks, person search grapples with the twofold challenges of pedestrian detection and person reidentification. Despite the considerable scholarly attention devoted to this area of research, person search remains a formidable task due to variations in context, scale, and other factors across different scenarios.

Figure 1. Person reidentification: matching with manually cropped pedestrians. The green box represents the correct recognition result, and the red box represents the incorrect recognition result.

Figure 2. Person search: finding from whole-scene images. The green box represents the correct search result.

Benefiting from the ability of deep learning to automatically and rapidly extract features (Chen et al. 2023; Feng et al. 2023; Xie et al. 2020; Xie et al. 2022a; Xie et al. 2022b), researchers have developed many deep learning-based person search methods. The existing methods for person search can be divided into two broad categories: two-step methods and end-to-end methods. The two-step method (Dong et al. 2020; Kalayeh et al. 2018; Lan, Zhu, and Gong 2018; Si et al. 2018; Zheng, Zheng, and Yang 2018) divides the person search task into two subtasks: person detection (Li et al. 2020; Xie et al. 2022) and person reidentification (Kim et al. 2023; Zhang and Wang 2023). Each subtask is separately optimized to enhance adaptivity. However, the independent processing of these subtasks makes the methodology time-consuming and labor-intensive. The end-to-end method integrates detection and reidentification into the same network framework and optimizes the two subtasks simultaneously (Chen et al. 2020; Fiaz et al. 2022; Zhao et al. 2022). This approach makes the whole process more efficient and convenient. Although the end-to-end method has achieved some results, there are still several shortcomings.

1. In various scenarios, the semantic features of the target individual, the surrounding environment, and nearby individuals tend to shift. Nevertheless, the majority of existing methods primarily focus on information in the vicinity of the target person's center (Yan et al. 2021; Yu et al. 2022), neglecting to fully explore and utilize a broader scope of relevant scene information and background person data.

2. Different from ordinary pedestrian detection, person search encompasses the task of matching the same individual across multiple cameras. First, contrary to continuous tracking within the same viewpoint, images from different surveillance scenes may result in occlusion of the target individual and variances in scale. Second, in densely populated settings, similar individuals can interfere with the target person search, thereby exacerbating the person search complexity, as illustrated in Figure 3.

3. Most existing methods strive to enhance the accuracy of person reidentification by refining physical models or algorithms. However, the simple distance function employed in person reidentification networks fails to gauge the similarity of pedestrian feature vectors comprehensively and precisely. This leads to a diminished accuracy of person reidentification algorithms (An, Wang, and Liu 2023; Wang and Guo 2022; Yun et al. 2021; Zheng et al. 2017).

Figure 3. Examples of person search interference in different scenarios.

Given the above shortcomings, this paper proposes a multifeatured-constrained cross-view person intelligent search method to adapt the pedestrian search task in complex environments, starting from the performance characteristics of people in the scene. First, we built a global-local context-aware (GLCA) module to fully capture the differential features of personnel contexts. Second, we constructed a semantic complementarity and feature aggregation (SCFA) module to enhance the performance of personnel features at different scales. Finally, based on the previous features, we combined the spatial features, personal identity features, and detection confidence features to constrain the pedestrian search and realize an accurate person search for cross-view images.

The remainder of this paper is organized as follows. Section 2 introduces related work. Section 3 presents a detailed description of the proposed method and the loss functions. Section 4 introduces the dataset descriptions and hyperparameter configuration. Section 5 presents a comprehensive analysis of the experiments. Sections 6 and 7 provide the discussion and conclusion, respectively.

2. Related work

2.1. Person reidentification

The aim of person reidentification is to match specific pedestrians in a large number of pedestrian images. Early studies mainly used handcrafted features for reidentification tasks (Lan, Zhu, and Gong 2018). As convolutional neural networks (CNNs) continue to evolve, deep learning-based methodologies have established strong footholds in the domain of person reidentification (Huertas-Tato et al. 2022; Wu et al. 2022). Sun et al. (2018) proposed refined part pooling to reassign outliers in part models. Wang et al. (2018) split pedestrian images into patches to learn features from multiple granularities. Zhang et al. (2020) proposed an effective relationship-aware global attention module that captures global structural information for better attentional learning. Wang et al. (2021) introduced a new pyramid spatiotemporal aggregation framework that gradually aggregates frame-level features and fuzes hierarchical spatiotemporal features into a final video-level representation. Qu, Zhang, and Zhang (2023) proposed a parallel mixed-attention network to complement spatial and channel information in Re-ID. Zhang et al. (2023) proposed a complementary network for person reidentification, which aims to extract more discriminative and robust local representations through complementary cues.

Although the person reidentification task can effectively identify a specific person from a large number of images, most existing person reidentification algorithms focus on matching cropped pedestrian images between queries and candidates. However, in real-world scenarios, pedestrian bounding box annotations are often unavailable. Therefore, person reidentification is not suitable for identifying target pedestrians in practical complex scenarios.

2.2. Person search

Person search usually includes two stages, pedestrian detection and person reidentification, which aim to locate and identify specific pedestrians from given images or videos. Compared to the singular task of person reidentification, the person search task is better suited for real-world applications because the image library comprises natural images that encompass both individuals and their surrounding environments. The existing person search methods can be divided into two-step methods and end-to-end methods. The two-step method requires independent optimization of two subtask frameworks. Chen et al. proposed a mask-guided person search method that models the foreground person and the original image block separately, obtaining rich representations from two separate CNN streams (Chen, Huang, and Tao 2018). Wang et al. (2020) proposed a task-consistent two-stage person search framework that included an identity-guided detector module to generate a query-like bounding box for the reidentification stage. Han et al. (2019) proposed a refinement framework for person localization based on reidentification, which can obtain more reliable bounding boxes and provide more discriminative feature embeddings for reidentification tasks.

The end-to-end method of person search combines the tasks of pedestrian detection and reidentification into a unified model, aiming to construct an efficient network that provides search results without the need for additional steps. The first end-to-end person search framework was proposed by Xiao et al. (2017). Since then, numerous successful works have emerged as subsequent researchers have delved into this field (Fiaz et al. 2022; Li and Miao 2021; Yan et al. 2021; Yu et al. 2022; Zhao et al. 2022). For example, Yan et al. (2021) introduced the first anchorless person search framework and designed a feature-aligned network to solve the mismatch problem. Considering the connection between detection and reidentification, Li and Miao (2021) designed a sequential network for person search and proposed the contextual-based global matching (CBGM) algorithm to extract context information from different people. Zhao et al. (2022) focused on the noise in person searches and proposed a context-contrasted loss to enhance the identification features among pedestrians while maintaining feature consistency for the same identity. Yu et al. (2022) proposed a cascade occluded attention transformer (COAT) for end-to-end person searches, which solves the issues of scale, viewpoint, and occlusion.

In general, these existing methods use the information from the detected bounding box to extract reidentification features in the reidentification stage. However, when considering an entire scene image or video, information from not only the human-centered adjacent area but also the background information of the scene and other people present can be obtained. Additionally, the scale of different scenes varies significantly, leading to occlusion or situations with a high density of individuals. Therefore, leveraging this abundant information to supplement or enhance the reidentification features can improve the search performance.

2.3. Context information for person search

The context information has been proven effective in the tasks of various computer vision tasks (Xie et al. 2022). In the detection task, the target object is not only related to itself or adjacent information but also strongly connected to the global scene information. Therefore, leveraging global context information is often employed to enhance target features and achieve more accurate localization (Chen, Huang, and Tao 2018; Liu et al. 2018). In the reidentification task, researchers often explore additional information between the target person and adjacent individuals to establish connections between their features and improve reidentification performance (Yan et al. 2020; Zhu et al. 2020). In the task of person searching, researchers usually examine the local context of individuals through image matching (Li and Miao 2021). Additionally, Yan et al. (2021) studied context cues at three levels – detection, memory, and scene – in unconstrained natural images using weak supervision. However, these methods often utilize either global scene information or local adjacent person information without fully exploiting both types of information to enhance the final person reidentification representation.

3. Method

3.1. Research framework overview

The proposed method is shown in Figure 4. The method takes the image as input data and outputs the results of a person search, forming an end-to-end mapping that includes two subtasks: pedestrian detection and person reidentification. First, we designed the GLCA module to learn the global context information and local person context. The model established the relationship between the target person and the scene and adjacent persons. Second, we constructed the SCFA module to build the connection between global and local context information and aggregate multiple feature information. Finally, we incorporated a precise detection frame coordinate constraint (Constraint 1), a person identity feature constraint (Constraint 2), and a detection frame confidence constraint (Constraint 3) to improve the comprehensive measurement capability of our framework for pedestrian features.

Figure 4. Person intelligent search framework.

3.2. Global-local context-aware module

Contextual information has been proven to play an important role in person search-related tasks (Yan et al. 2019; Yan et al. 2021). However, in previous works, most researchers used GNNs to learn local contextual information, which significantly increased the complexity of the method. Therefore, to address this issue and consider the diverse characteristics of individuals in various scenarios, a simple and effective spatial-channel correlation attention (SCCA) submodule is developed to efficiently collect and embed the image's contextual information, as depicted in Figure 5. This submodule is deployed in both the pedestrian detection and person reidentification stages, thereby effectively addressing the problem of insufficient feature extraction caused by occlusions and person-intensive situations. It is noteworthy that from Figure 5, the GLCA module may appear somewhat similar to the CBAM module (Woo et al. 2018) previously proposed by researchers, as both utilize channel and spatial attention mechanisms. However, in the module we propose, the methods of connecting channel attention and spatial attention are different and more complex. Moreover, the designed module makes better use of original features, allowing the original features from each stage to be better preserved in subsequent processes. This module is capable of more effectively exploring contextual semantics in the task of person search.

Figure 5. The global-local context-aware module framework.

The output of the backbone in the detection stage is further processed because it contains rich global response information, enhancing the context features among objects at the global level. In the reidentification stage, the features corresponding to the person area are first extracted. Then, the information shared between adjacent local areas is learned to reduce the differences and achieve more accurate person reidentification. It is important to note that when the target person is occluded in the image or video, the feature expression of the target person tends to weaken. Additionally, in scenes with a large number of people, similar individuals can interfere with the characteristics of the target person, leading to insufficient feature extraction. However, the GLCA module leverages both the global scene context information and the local person context information to enhance the reidentification features and improve target person identification, thereby increasing the distinctiveness between the target person and the background individuals.

Specifically, the same method is used to extract the global and local contexts, but the inputs of the modules are different. For the global context, the RoI align pooled features are sent to the submodule, and for the local context, the filtered person features are sent to the submodule. First, the module input (represented by F) is sent to adaptive maximum pooling in parallel. Second, two 1 × 1 convolutions follow, after which the two convolutions are added, and the channel attention map is obtained by using the sigmoid function. Then, the original input is multiplied by the channel attention and added by the jump connection to obtain F_CA. Then, F_CA is pooled by the same maximum pooling and average pooling and fuzed by a concatenation operation. Finally, the spatial attention map is obtained by the sigmoid function and multiplied by F_CA to obtain F_(CA-SA). Mathematically, this process can be described simply as shown in Equations (1(1) ${\mathcal{F}}_{\mathrm{CA}}={\mathcal{M}}_C\otimes \mathcal{F}\oplus \mathcal{F}$(1) ) and (2): (1) ${\mathcal{F}}_{\mathrm{CA}}={\mathcal{M}}_C\otimes \mathcal{F}\oplus \mathcal{F}$(1) (2) ${\mathcal{F}}_{\mathrm{CA}-\mathrm{SA}}={\mathcal{M}}_S\otimes {\mathcal{F}}_{\mathrm{CA}}\oplus {\mathcal{F}}_{\mathrm{CA}}$(2) where $\mathcal{F}$ represents the original input. ${\mathcal{M}}_C$ and ${\mathcal{M}}_S$ represent the channel attention map and spatial attention map, respectively. ${\mathcal{F}}_{\mathrm{CA}}$ represents the feature maps generated by multiplying and adding. ${\mathcal{F}}_{\mathrm{CA}-\mathrm{SA}}$ is the final output of the submodule.

3.3. Semantic complementarity and feature aggregation module

The results of feature extraction reveal that the global context encompasses abstract semantic information. The local context comprises more intricate and detailed features. However, the existing methods directly aggregate the two methods (Zheng et al. 2021). Although some results have been achieved, this method further increases the difference between the characteristics of people, resulting in rough or even incorrect discriminant features. Therefore, considering the obvious scale differences among people in the same or different scenes, this paper proposes a semantic complementarity module, which develops a cross-fusion strategy to establish a collaborative mechanism between global and local context information, as shown in Figure 6.

1. The obtained global and local context information is subjected to the sigmoid function to obtain the attention map to obtain the weight information in different global and local feature maps and assist in focusing on key areas, which can be described as Eqs. (3(3) ${\mathcal{A}}_1=\sigma (\mathrm{Con}{v}_{3\times 3}(\mathrm{Con}{v}_{3\times 3}({\mathcal{F}}_{\mathrm{Global}})))$(3) ) and (4(4) ${\mathcal{A}}_2=\sigma (\mathrm{Con}{v}_{3\times 3}(\mathrm{Con}{v}_{3\times 3}({\mathcal{F}}_{\mathrm{Local}})))$(4) ). (3) ${\mathcal{A}}_1=\sigma (\mathrm{Con}{v}_{3\times 3}(\mathrm{Con}{v}_{3\times 3}({\mathcal{F}}_{\mathrm{Global}})))$(3) (4) ${\mathcal{A}}_2=\sigma (\mathrm{Con}{v}_{3\times 3}(\mathrm{Con}{v}_{3\times 3}({\mathcal{F}}_{\mathrm{Local}})))$(4) where $\sigma (\cdot )$ represents the sigmoid function. $\mathrm{Con}{v}_{3\times 3}(\cdot )$ represents a 3 × 3 convolution. ${\mathcal{F}}_{\mathrm{Global}}$ represents the global context of the detection stage. ${\mathcal{F}}_{\mathrm{Local}}$ represents the local context of the reidentification stage.

2. By performing cross-multiplication and addition, effective feature complementation is carried out, resulting in feature maps that contain more valuable information. This process aims to minimize feature differences prior to aggregation. It can be mathematically expressed through Eqs. (5(5) ${\mathcal{C}}_1=\mathrm{AMP}(({\mathcal{A}}_1\oplus 1)\otimes \mathrm{Con}{v}_{3\times 3}({\mathcal{F}}_{\mathrm{Global}}))$(5) ) and (6(6) ${\mathcal{C}}_2=\mathrm{AMP}(({\mathcal{A}}_2\oplus 1)\otimes \mathrm{Con}{v}_{3\times 3}({\mathcal{F}}_{\mathrm{Local}}))$(6) ). (5) ${\mathcal{C}}_1=\mathrm{AMP}(({\mathcal{A}}_1\oplus 1)\otimes \mathrm{Con}{v}_{3\times 3}({\mathcal{F}}_{\mathrm{Global}}))$(5) (6) ${\mathcal{C}}_2=\mathrm{AMP}(({\mathcal{A}}_2\oplus 1)\otimes \mathrm{Con}{v}_{3\times 3}({\mathcal{F}}_{\mathrm{Local}}))$(6) where ${\mathcal{C}}_i\{i=1,2\}$ represents the cross-refined features. $\mathrm{Con}{v}_{3\times 3}(\cdot )$ and $\mathrm{Con}{v}_{1\times 1}(\cdot )$ represent 3 × 3 convolution and 1 × 1 convolution, respectively. ${\mathcal{A}}_i\{i=1,2\}$ represents the attention maps. ${\mathcal{F}}_{\mathrm{Global}}$ represents the global context in the detection stage. ${\mathcal{F}}_{\mathrm{Local}}$ represents the local context in the reidentification stage. $\mathrm{AMP}(\cdot )$ represents AdaptiveMaxPool.$\oplus$ represents elementwise addition.$\otimes$ represents elementwise multiplication.

3. As Equation (8(8) $f=\sigma (\mathrm{Con}{v}_{1\times 1}(\mathrm{Fuse}))\otimes \mathrm{Fuse}$(8) ) shows, the reidentification features are connected with two complementary contextual features, and the features are unified through an attention mechanism to ensure that the network focuses on the most important information. Specifically, the aggregated features are fed into the channel attention mechanism to weight each position and channel independently. After that, norm-aware embedding (NAE) is used to update the shared feature vectors of detection and reidentification. Finally, OIM loss is used to learn and optimize the final representation of the target person. (7) $\mathrm{Fuse}=\mathrm{Cat}({\mathcal{C}}_1,{\mathcal{C}}_2,{\mathcal{F}}_{\mathrm{Select}})$(7) (8) $f=\sigma (\mathrm{Con}{v}_{1\times 1}(\mathrm{Fuse}))\otimes \mathrm{Fuse}$(8) where$\mathrm{Cat}(\cdot )$ represents the concatenate operation. ${\mathcal{C}}_i\{i=1,2\}$ represents the cross-refined features. ${\mathcal{F}}_{\mathrm{Select}}$ represents the selected person features in the reidentification stage. $\mathrm{Con}{v}_{1\times 1}(\cdot )$ represents the 1 × 1 convolution.$\otimes$ represents elementwise multiplication.

3.4. Multifeature constraints for person reidentification

In Sections 3.2 and 3.3, person-related features are thoroughly extracted to constrain the network's focus on the target individuals. However, existing person reidentification methods commonly rely on distance functions as the standard; consequently, these methods may not comprehensively and accurately measure the similarity of pedestrian feature vectors, leading to low recognition accuracy. A more comprehensive feature-based constraint is needed to further guide the network to address this issue. Therefore, this paper introduces a triple feature constraint in the pedestrian reidentification stage, which includes a precise detection frame coordinate constraint, a person identity feature constraint, and a pedestrian detection frame confidence constraint.

1. Precise detection frame coordinate constraint

Figure 6. The semantic complementarity and feature aggregation module framework.

The precision detection frame coordinate is a spatial feature that refers to the location information of the bounding box for pedestrian detection, including the upper left corner coordinate, width, and height. The pedestrian detection subnetwork obtains several candidate regions, which may contain pedestrians, through the anchor box mechanism in the region proposal network. After reusing the CNN, the network generates a feature map through the candidate region pooling layer, as depicted in Figure 7(a). However, due to two floating-point rounding operations, a significant pixel error can occur. This paper proposes the use of bilinear interpolation to mitigate this error. First, the original network's floating-point rounding operation is removed, and the floating-point numbers are retained during the mapping process. Second, sampling points are set for each region, and these points are estimated using bilinear interpolation. This process avoids coordinate rounding, resulting in better preservation of the spatial position information of the candidate regions. Consequently, more accurate person detection box coordinates can be obtained, as shown in Figure 7(b).

(2)	Person identity feature constraint

Figure 7. Precise detection frame coordinate constraints.

Accurate identification of pedestrian identities requires the extraction of unique characteristics that can effectively express their individuality. These specific identity features play a crucial role in measuring the differences between pedestrians. This paper incorporates personality characteristic constraints that describe pedestrian identities and utilizes the online instance matching loss function (Xiao et al. 2017) to update network parameters. The approach involves constructing a lookup table and a circular queue for pedestrians with labeled and unlabeled identities, respectively, within the dataset for optimization purposes, as illustrated in Figure 8.

Figure 8. Person identity feature constraints.

Assuming that the number of labeled pedestrians in the dataset is $L$ and the feature dimension is $d$, a $d\times L$ lookup table $V\in R^{d\times L}$ is designed to store the labeled pedestrians in the dataset. In forward propagation, let the sample pedestrian feature $x\in R^d$ be labeled $t$; then, the cosine distance $V^{T}x$ between $x$ and the pedestrian feature in the lookup table is calculated. In the backpropagation, if the model matches the identity information of sample $x$, its features are updated to the lookup table. The calculation method is shown in Equation (9(9) $v_t\leftarrow \gamma v_t+(1-\gamma )x$(9) ). (9) $v_t\leftarrow \gamma v_t+(1-\gamma )x$(9) where $v_t$ is the detection box feature after L2 regularization, $\gamma$∈[0,1].

In addition, considering the large number of unmarked pedestrians in person reidentification, if feature information can be effectively utilized, this approach can also promote the correct identification of marked pedestrians. In this paper, a Q-cycle queue $U\in R^{d\times Q}$ is designed to store unmarked pedestrians in the dataset. In forward propagation, the Euclidean distance $U^{T}x$ between pedestrian features in $x$ and cyclic queues is calculated. Then, the two sets of distance vectors obtained by the lookup table and the loop queue are combined, and the probability distribution of the sample is calculated according to Eqs. (10(10) $p_i={\displaystyle \frac{\exp \left({\displaystyle \frac{v_i^{T}x}{T}}\right)}{{\sum }_{j=1}^L\exp \left({\displaystyle \frac{v_i^{T}x}{T}}\right)+{\sum }_{k=1}^Q\exp \left({\displaystyle \frac{u_k^{T}x}{T}}\right)}}$(10) ) and (11(11) $q_i={\displaystyle \frac{\exp \left({\displaystyle \frac{u_i^{T}x}{T}}\right)}{{\sum }_{j=1}^L\exp \left({\displaystyle \frac{v_j^{T}x}{T}}\right)+{\sum }_{k=1}^Q\exp \left({\displaystyle \frac{u_k^{T}x}{T}}\right)}}$(11) ). (10) $p_i={\displaystyle \frac{\exp \left({\displaystyle \frac{v_i^{T}x}{T}}\right)}{{\sum }_{j=1}^L\exp \left({\displaystyle \frac{v_i^{T}x}{T}}\right)+{\sum }_{k=1}^Q\exp \left({\displaystyle \frac{u_k^{T}x}{T}}\right)}}$(10) (11) $q_i={\displaystyle \frac{\exp \left({\displaystyle \frac{u_i^{T}x}{T}}\right)}{{\sum }_{j=1}^L\exp \left({\displaystyle \frac{v_j^{T}x}{T}}\right)+{\sum }_{k=1}^Q\exp \left({\displaystyle \frac{u_k^{T}x}{T}}\right)}}$(11) where $p_i$ denotes the probability that sample $x$ is predicted to be $i$, $q_i$ denotes the probability that sample $x$ is predicted to be the i-th unlabeled pedestrian feature in the circular queue, and T is the control parameter of the probability distribution. The ultimate goal of the online instance matching loss function is to maximize the log-likelihood of sample x, as shown in Equation (12(12) $L_{\mathrm{OIM}}=E_x(\log p_t)$(12) ). (12) $L_{\mathrm{OIM}}=E_x(\log p_t)$(12)

This condition calculates the gradient descent of sample x through the online instance matching loss function and constantly updates the parameter values in the network to achieve the purpose of effectively training the neural network.

(3)	Pedestrian detection frame confidence constraint

The detection frame confidence is a feature used to measure whether a bounding box contains a real pedestrian target. The primary objective of person reidentification is to identify and match the most similar image to the given pedestrian within the candidate pedestrian database constructed from multiple cameras based on the target pedestrian image. Therefore, it is crucial to calculate the feature similarity between the target and the candidate pedestrians. Person reidentification determines the similarity between pedestrian images by evaluating the similarity between their features. Pedestrian images with the highest feature similarity are considered to have the same identity. Thus, designing an effective method for measuring feature similarity is of utmost importance in person reidentification. In this paper, the feature similarity measurement algorithm is further constrained by incorporating the confidence scores of all the candidate pedestrian detection boxes derived from the output results of the pedestrian detection subnetwork, as illustrated in Figure 9.

Figure 9. Pedestrian detection frame confidence constraints.

First, the confidence in all the candidate pedestrian detection boxes is linearly normalized to [0,1]. Second, the cosine distance between the candidate pedestrian detection frame and the pedestrian to be identified is calculated via Equation (13(13) $d_{\cos }(X_1,X_2)=(X_1\ast X_2{)}^T\cdot U$(13) ). Finally, using Equation (14(14) $d_{\mathrm{weight}-\cos }(X_1,X_2)=d_{\cos }(X_1,X_2)\cdot W$(14) ), the normalized confidence of the candidate pedestrian detection frame is used as the weight constraint and multiplied by the cosine distance to measure the feature similarity between the candidate pedestrian and the pedestrian to be identified. (13) $d_{\cos }(X_1,X_2)=(X_1\ast X_2{)}^T\cdot U$(13) (14) $d_{\mathrm{weight}-\cos }(X_1,X_2)=d_{\cos }(X_1,X_2)\cdot W$(14) where $X_1$ and $X_2$ represent the feature column vectors of the candidate pedestrians and the pedestrian to be identified, respectively. $(\cdot {)}^T$ represents the vector transpose. $X_1\ast X_2$ represents the multiplication of the corresponding elements in the feature vectors $X_1$ and $X_2$. $U$ denotes the constant vector $[1,1,\dots ,1{]}^T$. $W$ is the normalized confidence interval.

3.5. Hybrid loss functions

The loss function is used for supervision in both the detection and reidentification stages. The regression loss uses the smooth-L1-loss function (Equation (15(15) ${\ell }_{\mathrm{reg}}={\displaystyle \frac{1}{N}\sum _{i=1}^N{\mathcal{L}}_{\mathrm{smoot}{h}_{L_1}}(t_i,y_i)}$(15) )), the classification loss uses the cross-entropy loss function (Equations (16(16) ${\ell }_{\mathrm{cls}1}={\displaystyle \frac{1}{N}\sum _{i=1}^N{y}_i{\mathcal{L}}_{\mathrm{CE}}(p_i,y_i)}$(16) ) and (17(17) ${\ell }_{\mathrm{cls}2}={\displaystyle \frac{1}{N}\sum _{i=1}^N{y}_i{\mathcal{L}}_{\mathrm{CE}}(f_{\mathrm{SCFA}},y_i)}$(17) )), and the reidentification loss uses the OIM loss (Equation (18(18) ${\ell }_{\mathrm{re}-\mathrm{ID}}^{\mathrm{SCFA}}=\mathrm{OIM}(f_{\mathrm{SCFA}})$(18) )). The overall loss function is shown in Equation (19(19) ${\ell }_{\mathrm{joint}}={\lambda }_1{\ell }_{\mathrm{reg}1}+{\lambda }_2{\ell }_{\mathrm{cls}1}+{\lambda }_3{\ell }_{\mathrm{reg}2}+{\lambda }_4{\ell }_{\mathrm{cls}2}+{\lambda }_5{\ell }_{\mathrm{re}-\mathrm{ID}}^{\mathrm{SCFA}}$(19) ). (15) ${\ell }_{\mathrm{reg}}={\displaystyle \frac{1}{N}\sum _{i=1}^N{\mathcal{L}}_{\mathrm{smoot}{h}_{L_1}}(t_i,y_i)}$(15) where $N$ denotes the number of positive samples, $t_i$ denotes the coordinates of the i-th positive sample, and $y_i$ denotes the coordinates of the corresponding ground-truth box. (16) ${\ell }_{\mathrm{cls}1}={\displaystyle \frac{1}{N}\sum _{i=1}^N{y}_i{\mathcal{L}}_{\mathrm{CE}}(p_i,y_i)}$(16) (17) ${\ell }_{\mathrm{cls}2}={\displaystyle \frac{1}{N}\sum _{i=1}^N{y}_i{\mathcal{L}}_{\mathrm{CE}}(f_{\mathrm{SCFA}},y_i)}$(17) where $N$ denotes the number of samples, $p_i$is the predicted classification probability of the i-th sample, and $y_i$denotes the corresponding ground-truth label. ${\mathcal{L}}_{\mathrm{CE}}(\cdot )$ is the cross-entropy loss function. The setting of ${\lambda }_i(i=1,2,3,4,5)$ follows that of Li and Miao (2021). (18) ${\ell }_{\mathrm{re}-\mathrm{ID}}^{\mathrm{SCFA}}=\mathrm{OIM}(f_{\mathrm{SCFA}})$(18) (19) ${\ell }_{\mathrm{joint}}={\lambda }_1{\ell }_{\mathrm{reg}1}+{\lambda }_2{\ell }_{\mathrm{cls}1}+{\lambda }_3{\ell }_{\mathrm{reg}2}+{\lambda }_4{\ell }_{\mathrm{cls}2}+{\lambda }_5{\ell }_{\mathrm{re}-\mathrm{ID}}^{\mathrm{SCFA}}$(19)

4. Dataset descriptions and hyperparameter configuration

4.1. Dataset descriptions

1. Benchmark datasets

The experimental datasets used in this paper mainly include the CUHK-SYSU dataset (Xiao et al. 2017) and the PRW dataset (Zheng et al. 2017). CUHK-SYSU is one of the large-scale datasets commonly used for person re-identification research, which mainly consists of images extracted from road surveillance videos and movie scenes. The average resolution of the image is 800 × 600 pixels. We follow Xiao et al. to adopt the standard training and test split, with a training set of 5,532 identities and 11,206 images, and a test set of 2,900 query persons and 6,978 images. PRW dataset is derived from surveillance videos on the campus of Tsinghua University, captured by six cross-regional cameras. The scene images in the dataset have an average resolution of 1920 × 1080 pixels. Following the standard data split, the training set contains 5,704 images with 482 different identities, while the test set includes 2,057 query persons and 6,112 images.

(2)	Self-constructed dataset

To evaluate the generalization of the proposed method, a small dataset sourced from Southwest Jiaotong University is constructed. This dataset consists of four different scenes captured by two cross-regional cameras. The images in the dataset have a resolution of 1280 × 720 pixels, and the total time of the dataset is 305 s, with a total number of 15,450 frames. To further verify the generalization of the method, the entire dataset is used as the testing set without any training process.

4.2. Hyperparameter configuration

1. Implementation details

All experiments are performed on a server equipped with NVIDIA GeForce RTX 3090 24G and employ the ResNet50 as the backbone. We use the classification model pre-trained on the ImageNet to initialize our model. During the training process, the CUHK-SYSU dataset went through 16 epochs, while the PRW dataset underwent 13 epochs. The batch size was set to 4, and each image was resized to 900 × 1500 pixels for both the CUHK-SYSU and PRW datasets. Stochastic Gradient Descent (SGD) was used as an optimizer to optimize the model training process, with a learning rate of 0.003, momentum of 0.9, and weight decay of 0.0005. During the testing process, the threshold of non maximum suppression (NMS) was set to the same value as Zheng et al. (2021) for consistency. Additionally, the parameters used for CBGM (contextual-based global matching) were consistent with those used by Li and Miao (2021).

(2)	Evaluation metrics

We used evaluation metrics in the person search task, namely mean average precision (mAP) and top-k accuracy.

The mAP measures the average performance of an algorithm on the entire test set, which is calculated by computing the Average Precision (AP). AP refers to the average precision of the t-th image to be identified, as shown in Equation (20(20) $\mathrm{AP}={\displaystyle \frac{1}{p}\left({\displaystyle \frac{m_1}{\mathrm{lo}{c}_1}+{\displaystyle \frac{m_2}{\mathrm{lo}{c}_2}+\cdots +{\displaystyle \frac{m_p}{\mathrm{lo}{c}_p}}}}\right)}$(20) ). (20) $\mathrm{AP}={\displaystyle \frac{1}{p}\left({\displaystyle \frac{m_1}{\mathrm{lo}{c}_1}+{\displaystyle \frac{m_2}{\mathrm{lo}{c}_2}+\cdots +{\displaystyle \frac{m_p}{\mathrm{lo}{c}_p}}}}\right)}$(20) where $p$ represents the number of pedestrian images in the candidate pedestrian library that match the image to be identified, $m_i$ represents the i-th image that matches the image to be identified after sorting the candidate pedestrian library, and $\mathrm{lo}{c}_i$ denotes the position of the i-th matching result in the sorting list. The mAP is the average of AP for all the images to be identified, as shown in Equation (21(21) $\mathrm{mAP}={\displaystyle \frac{1}{c}\sum _{i=1}^{c}A{P}_t}$(21) ). (21) $\mathrm{mAP}={\displaystyle \frac{1}{c}\sum _{i=1}^{c}A{P}_t}$(21) Where $c$ represents the number of images to be identified, and $A{P}_k$ denotes the AP value for the t-th image to be identified.

Top-k accuracy represents the probability of correctly identifying the target image within the top-k images when sorting the image similarities in descending order, which can be calculated using Equation (22(22) $\mathrm{Top}-k={\displaystyle \frac{{\sum }_{i\in 1,2,\dots ,N}{S}_i}{N}}$(22) ). (22) $\mathrm{Top}-k={\displaystyle \frac{{\sum }_{i\in 1,2,\dots ,N}{S}_i}{N}}$(22) where N represents the number of images in the candidate pedestrian library. When the i-th image in the candidate pedestrian library matches the image to be identified, $S_i$ is set to 1. otherwise, $S_i$ is set to 0. Top-1 means that the first image matched each time is exactly the image to be identified, which is the most fundamental goal of the person search task.

5. Experimental evaluation

5.1. Comparison with state-of-the-art methods

In order to validate the effectiveness of the proposed method, 25 advanced methods are selected to compare the performance with the proposed method on CUHK-SYSU and PRW data. These methods can be divided into two categories, namely two-step and end-to-end. The two-step methods include MGTS (Chen et al. 2018), CLSA (Lan, Zhu, and Gong 2018), RDLR (Han et al. 2019), and Faster R-CNN + PCB + OR (Yao and Xu 2021). The End-to-End methods include OIM (Xiao et al. 2017), IAN (Xiao et al. 2019), QEEPS (Munjal et al. 2019), HOIM (Chen et al. 2020), APNet (Zhong et al.2020), BINet (Dong et al. 2020), NAE (Chen et al. 2020), NAE + (Chen et al. 2020), DIOIM (Dai et al. 2020), BUFF (Yang et al. 2020), IIDFC (Hou et al. 2021), OR (Yao and Xu 2021), PGSFL (Kim et al. 2021), AlignPS (Yan et al. 2021), AlignPS + (Yan et al. 2021), DMRNet (Han et al. 2021), SeqNet (Li and Miao 2021), SeqNet + NAE (Li and Miao 2021), CANR (Zhao et al. 2022), CANR + (Zhao et al. 2022), and QGN (Munjal et al. 2023). The detailed results comparing all the methods are recorded in Table 1.

Table 1. The results of proposed method are compared with the existing excellent neural network.

Methods		Publication	CUHK-SYSU		PRW
			mAP	Top-1	mAP	Top-1
Two-Step	CNN + MGTS	ECCV 2018	83.00	83.70	32.60	72.10
	CNN + CLSA	ECCV 2018	87.20	88.50	38.70	65.00
	FPN + RDLR	ICCV 2019	93.00	94.20	42.90	70.20
	Faster R-CNN + PCB + OR	TIP 2021	92.93	93.69	43.01	65.87
End-to-End	OIM	CVPR 2017	75.50	78.70	21.50	66.60
	IAN	PR 2019	76.30	80.10	23.00	61.90
	QEEPS	CVPR 2019	88.90	89.10	37.10	76.70
	HOIM	AAAI 2020	89.74	90.83	39.77	80.36
	APNet	CVPR 2020	88.90	89.30	41.90	81.40
	BINet	CVPR 2020	90.00	90.70	45.30	81.70
	NAE	CVPR 2020	91.50	92.40	43.30	80.90
	NAE+	CVPR 2020	92.10	92.90	44.00	81.10
	DIOIM	PR 2020	88.70	89.60	36.00	76.10
	BUFF	MM 2020	90.70	91.60	42.20	81.00
	IIDFC	TCSVT 2021	91.96	93.34	43.43	83.37
	PGSFL	CVPR 2021	92.30	94.70	44.20	85.20
	AlignPS	CVPR 2021	93.10	93.40	45.90	81.90
	AlignPS+	CVPR 2021	94.00	94.50	46.10	82.10
	DMRNet	AAAI 2021	93.20	94.20	46.90	83.30
	CANR	TCSVT 2022	92.39	93.21	43.41	83.81
	CANR+	TCSVT 2022	93.86	94.52	44.78	83.86
	SeqNet	AAAI 2022	93.80	94.60	46.70	83.40
	SeqNet + CBGM	AAAI 2022	94.80	95.70	47.60	87.60
	SeqNet + NAE	AAAI 2022	93.80	94.60	46.70	83.40
	QGN	PR 2023	91.50	92.10	42.90	81.90
	Ours	\	95.21	95.75	46.78	84.69
	Ours + CBGM	\	95.45	95.65	47.63	87.41

It can be observed in Table 1 that for the CUHK-SYSU dataset, the proposed method is as high as 95.21% on mAP and 95.75% on Top-1. For the PRW dataset, the proposed method is up to 46.78% on mAP and 84.69% on Top-1. In addition, in order to verify whether the CBGM algorithm can be complementary to the proposed method, this paper deploys it in the test phase. On the CUHK-SYSU dataset, the combined method further increases the mAP by 0.25%, but Top-1 is reduced by 0.1%. Compared with other methods, the combined method shows the most advanced performance. On the PRW dataset, both metrics show improvement, with an increase of 0.85% in mAP and 2.72% in Top-1 accuracy. Compared with other methods, the fusion method still achieves the best performance. In summary, our method achieves state-of-the-art comparative performance on the CUHK-SYSU dataset without the help of additional algorithms and additional auxiliary information. On the PRW dataset, although slightly less than SeqNet + CBGM, the proposed method still achieves competitive results. Sufficient quantitative analysis strongly proves the effectiveness of the proposed method in the person search task.

It is worth noting that, as the size of the image library continues to increase, the performance of person search will also be affected to varying degrees. To validate this effect, a set of experiments on the CUHK-SYSU dataset with different gallery sizes are conducted. The library size includes 50, 100, 500, 1000, 2000, and 4000. Figure 10 summarizes the performance comparison of proposed method with other state-of-the-art methods. First, as the number of libraries increases, the performance of all methods tends to decrease. However, the proposed method decreases more slowly than most methods. Second, the proposed method consistently maintains the best performance under different number of libraries. At the same time, with the increasing number of libraries, the advantages of the proposed method are more significant. In general, the experimental results prove the robustness of the proposed method.

Figure 10. Performance comparison in terms of different gallery sizes on CUHK-SYSU.

5.2. Different scenario analysis

In order to further validate the effectiveness of the proposed method in person search tasks, qualitative analysis is conducted (Section 5.2 and 5.3) to examine the coping ability of the proposed method in different scenarios. It is worth noting that the dataset includes both real-world scenes and movie scenes, so this paper analyze them separately. Figure 11 shows the search results in different scenarios.

Figure 11. Comparison of visual results of person search in real scenes.

In real-world scenes, this paper analyzes three challenging situations: occlusion, long-distance small target, and person-intensive scenarios. For the case where the person is occluded, Figure 11 shows the search results for two scenarios: the first group of images displays the search results when the target person in the gallery are occluded, while the last group of images shows the search results when the people in the query images are occluded. From the first group of images, what can be observed is that the target person in the gallery are heavily occluded by umbrellas, even the target person is completely occluded, which is extremely challenging. However, the proposed method can accurately search for and identify the target person. In the last group of images, the target person in the query is occluded by adjacent people, which may reduce the features of the target person learned by the method, resulting in uncertain or incorrect results. However, the proposed method can still search out the target person in such situations. From the second group of images, it can be seen that the target person is gradually moving away from the camera's field of view and there is a partial occlusion, but the proposed method can cope with this situation effectively. From the third group of images, we can see the results of the method in this paper when dealing with person-intensive scenarios. In the figure, the density of person not only increases, but also the posture of the target person and the position of the backpack also change. This change may potentially cause confusion between the target person characteristics and other person characteristics, increasing the uncertainty of the final result. However, the proposed method is still able to handle such scenarios well.

In the movie scenes, this paper analyzes from three aspects: illumination change, long-distance small target, and occlusion. First, from the first and second groups of images in Figure 12. It can be observed that the light received by the target person has changed significantly. In particular, in the second group of images, the light becomes very dim and the dress color and posture of the adjacent person are very similar to the target person, which will undoubtedly bring difficulties to the identification. Although SeqNet is also able to search and locate the target person, the proposed method has higher similarity and more accurate detection box. Second, in the third group of images, the position distance of the target person in the gallery changes and is occluded by the adjacent person, which makes the similar features around the target person blurred. Similarly, in the fourth group of images, the target person in the gallery is also partially occluded, which causes other methods to misretrieve the target person, and the proposed method has better results in dealing with these situations.

Figure 12. Comparison of visual results of person search in movie scenes.

In general, the method of this paper has better search performance than other methods in dealing with occlusion, long-distance small target, person-intensive scenarios, and illumination change. This is attributed to the fact that the proposed method not only focuses on global or local context information, but also makes the network pay attention to both information and interact effectively. By strengthening the feature representation of target person, the search performance of the target person is increased.

5.3. Generalization analysis

In order to verify the generalization and migration of the method proposed in this paper, robustness test experiments are carried out on self-built data sets. Specifically, we do not train from scratch on this data set, but directly apply the model trained on the SUHK-SYSU data set to the new data set to verify the generalization of the model, as shown in Figure 13.

Figure 13. Method generalization analysis in different scenarios.

From the first set of search results, it can be seen that in different video clips, the posture state of the person has changed greatly compared to the query, and some clips have received the occlusion of the riding tool. In the last clip, the body area of the target person is almost beyond the camera field of view, which will cause some interference to the person search. However, the method of this paper can accurately search for the target person in the case of these disturbances. From the second set of search results, it can be seen that for men wearing black clothes, it can be found that the two pedestrians with their peers are similar to the target. In addition, there are different degrees of occlusion of the target, which causes some interference to the identification of the target. In camera 4, the brightness of the lens has also changed, and the light has become darker, which makes the person characteristics of several black-clothed males side by side more similar. However, the method proposed in this paper can eliminate these interference factors, which not only realizes the identification of small-scale target pedestrians in two videos, but also achieves accurate identification of partially occluded target pedestrians.

5.4. Effects of person detection

Better detection results can usually bring some improvement to the performance of person search. Therefore, this paper selected some representative methods to establish a comparative experiment on the final search performance when using the real box and the detected bounding box. From Table 2, it can be seen that when using ground truth boxes, all methods show improved performance, which can prove that the detection results have a substantial impact on the final search performance. Additionally, it is worth noting that when using ground truth boxes, the proposed approach is improved by 0.47% on the CUHK-SYSU dataset, while OIM, NAE, and DIOIM are improved by 2.4%, 2.6%, and 2.6% respectively. The ground truth box brings less improvement to the proposed method than other methods, as well as on the PRW dataset, which also shows that the proposed method benefits more from the enhanced person re-identification features.

Table 2. Comparative results using the detected(det.) and ground truth(gt.) bounding boxes.

Methods	Boxes	CUHK-SYSU		PRW
		mAP	Top-1	mAP	Top-1
OIM	det.	75.50	78.70	21.50	66.60
OIM	gt.	77.90	80.50	25.20	71.10
NAE	det.	91.50	92.40	43.30	80.90
NAE	gt.	94.10	94.60	–	–
DIOIM	det.	83.80	84.60	30.40	71.50
DIOIM	gt.	86.40	86.90	33.70	73.40
SeqNet	det.	93.80	94.50	–	–
SeqNet	gt.	94.60	95.30	–	–
Ours	det.	95.20	95.75	46.78	84.69
Ours	gt.	95.67	95.82	48.02	85.80

5.5. Ablation study

In order to validate the effectiveness of the proposed modules, five groups of ablation experiments are established. Table 3 records the performance of different components. It can be seen that when the proposed module is added, it will bring a certain improvement to the final performance. Compared with baseline, when GCA (Global Context Attention) is added, the mAP and Top-1 accuracy are improved by 1.71% and 1.24% respectively. When LCA (Local Context Attention) is added, the mAP and Top-1 accuracy are improved by 0.33% and 0.31% respectively. It can be observed that when the module is added alone, the performance of the method is not much improved. This is because LCA aims to capture the characteristics of local adjacent people. When encountering dense people or similar characteristics between adjacent people and target people, the characteristics will be blurred, and the final image retrieval performance will be obtained. Similarly, when SCFA (Semantic Context Fusion Attention) is added, the improvement in mAP is not significant due to the blind integration and interaction learning of features from two stages, resulting in misplacement of semantic features. However, when all the modules are integrated together, the performance of the entire model has been significantly improved.

Table 3. Different module ablation experiment of the proposed method.

No.	Baseline	GCA	LCA	SCFA	CUHK-SYSU
					mAP	Top-1
1	✓				92.89	93.83
2	✓	✓			94.60	95.07
3	✓		✓		93.22	93.52
4	✓			✓	93.08	93.48
5	✓	✓	✓	✓	95.21	95.75

Additionally, we calculate the FLOPs and the number of parameters for the method proposed in this paper and for the highly competitive SeqNet. To ensure a fair comparison, the input images for both models are adjusted to 900×1500 pixels, with a consistent batch size as well. The method presented in this paper has a parameter count of 33.04M and FLOPs of 386.79G, while SeqNet's parameter count and FLOPs are 79.51M and 521.02G, respectively. Combining sections 5.1, 5.2, and 5.4, it is evident that the method introduced in this paper achieves higher performance with lower complexity, further demonstrating its superiority in the task of person search.

6. Discussion

6.1. Applicability and implications

With the development of deep neural networks and the demand for intelligent video surveillance, the attention has increased significantly on people's search techniques in computer vision (Fiaz et al. 2022; Qian et al. 2022). In this paper, we innovatively utilized the global scene context and local people context to complement personal features. Furthermore, we combined multiple feature constraints to form an intelligent person search method for cross-view images. Extensive experiments and analyses were conducted using two public datasets, and the results show that our method outperforms 25 other state-of-the-art methods and can be better applied to the person search task in complex scenes. In addition, we built a new person search dataset to further demonstrate the generalizability of the proposed method.

Our intelligent search methodology accurately provides digital earth data with essential data on human activities and movements, real data that help us better understand the planet's phenomena, resources, and social issues (Li et al. 2021; Zhang et al. 2023). Our approach can help experts in GIS understand population distribution, human flow dynamics, and urban activity. This information can be used for personnel virtual simulation, and creating urban or disaster digital twins is essential (Therias and Rafiee 2023). Our method can accurately provide important information to safety and disaster emergency managers about the location of people, which can help build digital disaster emergency responses and maintain public safety (Singh, Khare, and Jethva 2022; Zhu et al. 2024b, 2024c).

6.2. Limitations

Figure 14 shows the shortcomings of the proposed method in different scenarios. In the two sets of display scenes, the target person in the query has a lower resolution and is far from the camera. The leftmost image shows the target person in the query, and the right image shows the wrong person (black box selection). This is because the proposed method does not perform too much processing during the feature extraction stage, which may lead to insufficient extraction of deeper features from the target person. As a result, there is a deviation in the search results of our method in low-resolution images, and it is impossible to accurately locate low-resolution target personnel.

Figure 14. Methods for insufficient case visualization analysis.

7. Conclusion and future work

A person search can provide human activity information for the construction of digital earth, thus promoting the development of informatization in disaster emergency response and public safety. In this paper, we propose a cross-view person intelligent search method that innovatively exploits global-local context information and introduces multiple feature constraints, realizing accurate cross-view person searches in complex environments. The main contributions of this paper are as follows.

First, the GLCA module is proposed and deployed in the pedestrian detection and person rerecognition stages. This module can utilize the global scene context and local person context well to supplement person rerecognition details, which solves the problem of insufficient extraction of person feature information in cases of occlusion and person-intensive situations.

Second, the SCFA module is designed to establish a collaborative mechanism between global and local context information. This module enhances the aggregation ability of the search method for personnel features of different scales. This approach solves the problem of scale differences between people in the same or different scenes.

Third, the person spatial feature constraint, person identity feature constraint, and detection confidence feature constraint are included to comprehensively and accurately measure pedestrian feature vector similarity, which improves the discriminative ability of ambiguous person features in the process of pedestrian reidentification.

Finally, an end-to-end framework for intelligent people search is formed. The experimental results show that our method outperforms 25 other deep learning methods for person searches and can better deal with complex situations such as occlusions, scale changes, and high person density, as well as having higher generalizability and migration. In addition, the proposed method can compensate for the lack of real human activity information in existing disaster twin scenes. It can provide accurate human activity data for disaster emergency rescue and public safety management.

In future research, pedestrian sex, walking posture, clothing color, backpack and other attribute features will be considered to describe pedestrians more comprehensively, as this approach can improve the ability of target people to search for objects with low resolution in complex scenes. Moreover, we will incorporate the people search method into the disaster twin scene to construct a digital twin where the natural and social environments are intertwined.

Author contributions

Conceptualization, Jun Zhu and Yakun Xie; Methodology, Jinbin Zhang; Resources, Hongyu Chen, Tianyi Zhang, Hengchao Gu and Huijie Lian; Software, Jinbin Zhang and Tianyi Zhang; Validation, Jun Zhu and Yakun Xie; Writing – original draft, Jinbin Zhang and Hongyu Chen; Writing – review & editing, Jun Zhu and Yakun Xie. All authors have read and agreed to the published version of the manuscript.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The data that support the findings of this study are available from the corresponding author, upon reasonable request.

Additional information

Funding

This paper was supported by National Key Research and Development Program of China (Grant No. 2022YFC3005703), National Natural Science Foundation of China (Grant Nos. 42301473, 42271424, and 42171397), Postdoctoral Innovation Talents Support Program (BX20230299), China Postdoctoral Science Foundation (2023M742884).