Spatio-temporal data fusion techniques for modeling digital twin City

ABSTRACT

The digital twin city technique maps the massive city environmental and social data on a three-dimensional virtual model. It presents the operational status of physical world and supports intelligent city governance. However, the inefficient utilization of distributed data resources, and the lack of sharing and collaboration among multiple departments restrict the data formulation of digital twin city construction. This research proposes a new cross-domain spatio-temporal data fusion framework for supporting complex urban governance. It integrates the heterogeneous urban information generated and stored by different government departments using multiple-information techniques. A specified geographic base reflecting the real city status is established, using geographical entities with unified address as identifiers to encapsulate the urban elements information. We introduce a comprehensive urban spatio-temporal data center construction process, which has already supported multiple urban governance projects. The two distinct advantages in using this data fusion system are: 1) The proposed Bert+PtrNet+ESIM-based address mapping method associates the urban elements information to their corresponding geographic entities with 99.3% F1-Score on real-world dataset. 2) The Wuhan spatio-temporal data center operation illustrates the capability of our framework for complex urban governance, which significantly improves the efficiency of urban management and services. This integrated system engineering provides reference and inspiration for further spatio-temporal data management, which contributes to the future social governance in digital twin city platform.

KEYWORDS:

• Digital twin city
• smart city
• system integration
• artificial intelligence
• social governance
• integrated spatio-temporal data center

1. Introduction

Urbanization process creates employment and income for residents, which promotes the national economic development. But it also leads to the problems of medical pressure, population explosion, traffic jam and environmental pollution. In recent years, city managers tried to use emerging information technologies, such as artificial intelligence, big data, cloud computing and Internet of Things, to efficiently handle various demands of urban development. Cities are equipped with all kinds of sensors and cameras. With the help of mobile communication technologies, including 5G/6G and NB-IoT, data collected by these terminals are quickly transmitted, integrated, and analyzed in the government cloud platform. It enables the comprehensive sensing, ubiquitous interconnection and pervasive computing of urban information, which provides supports for urban planning, construction and management. This is the so-called smart city (Camero and Alba 2019; Cocchia 2014; Nam and Pardo 2011; Su, Li, and Fu 2011).

Digital twin is a technology to construct digital objects that exactly correspond the properties of physical entities (Jones et al. 2020; Tao et al. 2018, 2019). This technology has been gradually introduced into smart city management in recent years (Deng, Zhang, and Shen 2021; Farsi et al. 2019; D. Li, Yu, and Shao 2021; Rathore et al. 2017; Shahat, Hyun, and Chunho 2021). Spatio-temporal information data is the foundation and support of digital twin city. Geographical entities often have various spatio-temporal attributes such as sound, light, heat, electricity, mechanics, chemistry, biology and location, which reflect the basic information of the corresponding position. This information can be obtained through information sensing devices such as radio frequency identification, infrared sensors, laser scanners, and global navigation systems. Using 3D reconstruction and City Information Modeling (CIM) to map the real city the spatio-temporal information data on a three-dimensional digital model, a virtual twin city reflecting the physical world information can be built in the cloud.

City spatio-temporal information data are collected separately by multiple departments. These data generated by different businesses are scattered and stored in different spatial database systems, which are heterogeneous in terms of storage structure, index, semantic expression, format, life cycle, applicable application scenarios, etc. As a result, these multi-source heterogeneous data have low degree of standardization, which are difficult for digital twin city construction.

Therefore, how to efficiently make use of massive urban spatio-temporal data to guide urban management and service is one of the most important topics in digital twin city. It is of great significance to improve the utilization efficiency of decentralized spatio-temporal information data resources, to promote the information sharing and collaboration among multiple fields. In this research, we design a new geographic base framework for heterogeneous spatio-temporal data fusion to support urban governance in digital twin city construction. This framework designs a geographical entity with unified address for encapsulating urban information. Our contributions are highlighted below in three technical aspects:

1. We choose geographical address information as a new entry point for cross-domain data fusion. Based on the unique identification characteristics of address, we effectively integrate and coordinate urban governance elements data from different departments to maximize the application scope.

2. We match the urban elements data such as people, enterprises and things across business fields to the household level. It enables the analysis of government structured information achieves higher spatial and temporal resolution in the digital twin city model. A hierarchical twin model expressed by multi-level and multi-scale data is established.

3. We generate knowledge map model based on spatial semantics to dig and express the complex relationships between entities in a deeper level. It provides intelligent reasoning and automated decision support for spatial information analysis, which assists urban governance and decision-making for complex city management services.

Through the real construction experience of Wuhan spatio-temporal data center, we verify the effectiveness and practicability of database federation, geo-semantic association model, data aggregation and knowledge map for urban governance. Our research not only provides a new data support method for complex urban governance, but also expands the application scenario of digital twin city technology.

2. Literature review on spatio-temporal data fusion

Spatio-temporal data fusion refers to integrating data in time and space dimension to realize urban information enhancement (Canalle, Salgado, and Lóscio 2021; Hakimi et al. 2023; Lau et al. 2019). We select the academic papers database including open journals, conference papers and dissertations, and use keywords such as spatio-temporal database, spatio-temporal data fusion, urban governance, social governance, digital twin cities, smart cities and data integration to extract research papers with more than 50 citations and in the last six years, to screen out representative and influential research. From these research papers, three main technical routes for spatio-temporal data fusion are presented: surface monitoring, urban planning and social governance.

1. Surface monitoring: Fuse the remote sensing images of different satellites to generate image sequences with diverse spatial scales and temporal dynamics, to track the evolution of topography, landforms, climate patterns and ecology (Zhang et al. 2015). Scholars fused the data of multi-modal optical sensors to improve the temporal and spatial resolution of the results (X. Li et al. 2020; Liang et al. 2018; Q. Wang et al. 2020; Weng, Fu, and Feng 2014). It mainly includes the spatial and temporal adaptive reflectance fusion model based on spatial weight (STARFM) (Gao et al. 2006; Schmidt et al. 2015), the method based on spectral unmixing (Ping, Meng, and Su 2018), and the method under Bayesian framework (Xue, Leung, and Fung 2017, 2019).

Compared with the optical methods, in recent years, some scholars have explored new methods to integrate multi-source heterogeneous data, such as optical, radar and ground data, to discover complementary spatio-temporal information. For example, the fusion of InSAR and GNSS data (Liu et al. 2019), the fusion of Landsat-8 and MODIS data with fine- and coarse-resolutions (Liao et al. 2017), and the fusion of multi-source social media data (De et al. 2021). These research provide new ideas for wide-area environmental monitoring. As an effective tool, deep learning method is also introduced into spatio-temporal data fusion to realize end-to-end feature extraction and data fusion, to deal with the nonlinear relationship in multi-source data modeling (T. Wang et al. 2019; Yang et al. 2022; Yu et al. 2021).

2. Urban planning: Fuse the data from traffic monitoring system, satellite remote sensing, Geographic Information System (GIS) and other sensors to support urban management and analysis. Different data analysis methods are used to mine potential spatio-temporal patterns. Traffic forecasting is one of the important topics in urban planning. The traffic data of different time periods are fused with the GIS spatio-temporal data to identify traffic hot spots and predict congestion points, to support urban layout planning and reduce the degree of urban traffic congestion.

For example, k-means clustering algorithm is used to classify the temporal and spatial distribution of congested roads (Dao, Nguyen, and Zettsu 2019), and convolutional neural network is used to predict traffic congestion with 2D/3D raster image (Song et al. 2019). Urban GIS data is also integrated with BIM to visualize the city in multi-dimensional and multi-spatial scales, and perform spatial analysis (Dou et al. 2020).

3. Social governance: Fuse the data from multiple fields, times, places and modes to support urban population management and urban event decisions. To meet the growing demand for highly complex social governance applications, it is necessary to integrate structured, semi-structured and unstructured data to establish a knowledge base and analysis model of social governance elements at the city level, and evaluate their performance from multiple aspects.

For example, a dynamic modeling method with CNN and LSTM (Long Short-Term Memory), using the trajectory data of human activities with temporal and spatial characteristics, predicts the changes of population density and geographical features in urban areas (Jiang et al. 2021). The season autoregressive integrated moving average model based on SARIMA and spatio-temporal clustering can be used to predict urban management events, assisting community resource allocation and urban governance planning (Yue, Liu, and Song 2018).

Although the above research has made remarkable progress, there still exists obvious research gaps when it is applied to the meso and micro levels of urban governance. At present, the research tends to focus on vertical application, and it is difficult to directly migrate to more complex urban governance practices with high requirements for multi-data fusion and dynamic analysis. Especially at the micro level, how to apply spatio-temporal data fusion in the rapidly changing environment of cities, and how to effectively integrate scattered data sources with different time and space scales from different domains to support rapid and accurate government affairs decision-making is an urgent challenge.

To sum up, the existing methods are difficult to meet the complex social governance needs of cross-domain integration. It is necessary to establish a new conceptual framework to support the fusion of heterogeneous social governance elements data for complex urban governance scenarios, including developing a flexible data fusion architecture, exploring real-time dynamic data analysis methods, and enhancing the application practice in the field of urban governance. Through aligning enough urban information with internal knowledge relationship to different unified entity object, the complex relationships between urban entities can be expressed, then the digital infrastructure system can be established to carry out more refined urban analysis and visualization support at the city level.

3. Construction of urban geographic base with geographic entity

We integrate multiple-information techniques to establish a spatio-temporal data fusion platform for digital city management. As shown in Figure 1, briefly speaking, the following steps are required to execute:

Figure 1. Digital twin city framework based on the spatio-temporal fusion of social elements data. A new geographic base framework adopts geographical entities with unified address to encapsulate urban information. This spatio-temporal data fusion framework supports multiple type of urban governance services.

• Remote sensing, global positioning, 5G/6G IoT and other technologies are used to collect spatio-temporal data. These data are transferred to the spatio-temporal big data platform on distributed cloud.

• The unified address of geographic entities is used as an identifier to encapsulate social information. Data of different urban elements, such as people, buildings, organizations, objects and events, can be associated to their corresponding geographic entities.

• Data fusion computing, knowledge construction and reasoning are executed to generate the data blocks of multiple geographic entities. It reflects the spatio-temporal properties of the corresponding geographic elements, to support smart city applications.

Digital twin city contains complete urban geographic information, including not only spatial location, spatial scope, but also semantic-based digital representation of different urban elements. We define the geographical entity of digital twin city as a natural or artificial object exists in the real world with geographical location, attributes and relationships, including buildings, roads, rivers and so on. Each geographic entity has its unique identity, name, address, and can be expressed with their corresponding spatial and temporal information to realize spatial analysis, decision-making, and on-demand service.

We define the geographic base of urban information as a service platform for managing city spatio-temporal dataset represented by geographic entities, which reflects the real city geographical status and characteristics. Traditional geographic information system treats urban elements such as people, buildings, organizations, etc., as independent layers and execute separate management. On the other hand, this new-defined geographic base adopts the geographical entity as a core factor to manage their corresponding urban elements data. Geographic base integrates multi-source heterogeneous urban spatio-temporal data, such as remote sensing images, three-dimensional mapping, Internet of Things, etc., and forms a complete urban dataset with the characteristics of uniqueness, stability and extensive correlation ability.

In the operation of smart cities, spatio-temporal data are collected from different sources and are stored in distributed departments. These data involve multidimensional information of urban elements such as people, objects and organizations. This kind of multi-source heterogeneous fragmented data is difficult to sort out and has low processing efficiency in cross-domain integration applications. Geographical address, as an accurate description of each entity in the city governance, indicates the unique position of different entity at different scales. Therefore, the ubiquitous effect of address makes it an important identifier for aligning urban governance elements from the spatial dimension.

As shown in Figure 2, we take geographic address as an identifier to encapsulate the urban elements data of people, buildings, organizations, events and objects scattered in various departments, which is named as “block data” structure. Block data is the collection of all kinds of data related to geographical entities formed in a physical region or political region. It realizes the aggregation of urban elements heterogeneous data to produce “blocks” for each geographic entity.

Figure 2. Taking the unified address of geographical entity as an identifier, the block data structure is constructed to encapsulate the urban elements data of people, buildings, enterprises, events and objects.

We execute geo-semantic address mapping to identify the unique address for different spatio-temporal data, then use data aggregation and knowledge reasoning to aggregate the core elements of urban governance into blocks. The association of natural resources data, basic geographic information data and urban operation data provides comprehensive support for land spatial planning. The association of urban infrastructure data, public service data and socio-economic data provides accurate and convenient information support for urban services.

Spatio-temporal data fusion platform with block data architecture serves multiple types of intelligent applications in the daily management of urban operation and maintenance. The construction of this address-based platform requires the following steps:

1. Database federation: Urban governance data from various departments, such as civil affairs, public security, and urban management, is integrated in the form of database federation to realize the centralized management and analysis. It is the key to deal with governance data with high security level and frequent updates.

2. Unique ID index based on spatial semantics: We generate a globally unique unified address code to associate the information of people, buildings, things, events and organization entities. The unified address library of block data realizes online address matching for different address standards and specifications in multiple business fields. It improves the efficiency of geographic information identification and management to ensure the accurate spatial positioning in social governance.

3. Heterogeneous data aggregation: We initialize and update data in batch and stream mode through various built-in and user-defined data aggregation operators to form the knowledge map model. This model base also supports dynamic operation and online compilation.

4. Knowledge reasoning service: We apply the knowledge map model base to train the AI model and publish business-driven AI knowledge reasoning service. According to the urban governance needs, we customize the combination and arrangement of API interfaces to solve specific business problems.

The rest of this work is organized as follows. Section 4 introduces how to establish the system infrastructure on multiple distributed clouds with database federation technique. Section 5 proposes a new Bert (Bidirectional Encoder Representations from Transformers)+PtrNet (Pointer Network)+ESIM (Enhanced Sequential Inference Model) geo-semantic mapping method for Chinese address matching. Section 6 illustrates the spatio-temporal data fusion process based on data aggregation operators, knowledge construction and reasoning. Section 7 shows the experiment results of our proposed Bert+PtrNet+ESIM geo-semantic mapping method, and uses our Wuhan City Hotline construction project as an example to show the effectiveness of the cross-domain spatio-temporal data integration scheme. Finally, we conclude with a summary of our major contributions.

4. Spatio-temporal database system architecture on distributed cloud

Digital twin city management puts forward high requirements for real-time interactive massive data analysis. It requires the use of cloud computing as a platform for spatio-temporal big data mining and processing (Hao et al. 2022; Hwang 2017; Y. Li et al. 2023; Y. Li et al. 2022; Marinescu 2018). Through virtualization technology, cloud computing abstracts the underlying physical resources into elastic virtual machines or containers (Bernstein 2014; Pahl et al. 2017; Rosenblum and Garfinkel 2005) and then allocates them to users to provide storage and computing services. Since the spatio-temporal data generated by different urban services are collected and stored in distributed cloud systems of different departments, we need to establish a heterogeneous spatial database integration system to promote information sharing and collaboration among multiple departments.

Based on the design experience of traditional geographic information cloud platform, we realize the deep integration of heterogeneous spatio-temporal data and build a knowledge-based urban information service platform. The following key points are considered in the design process:

• Openness of data access. Interface mapping technology is used to realize the open access of data between systems, which is the basis of interoperability and integration.

• Depth of data integration. It is necessary to realize the integration of both structured data and data at the semantic level to achieve deep integration.

• The relevance of knowledge management. Knowledge atlas should be constructed based on spatial ontology to realize the expression, organization, correlation and reasoning of spatial knowledge.

• The universality of data services. It is necessary to realize the whole process from data integration to knowledge management to service output, and publish data and knowledge services in a common form.

• System applicability. It is necessary to consider the requirements of different application scenarios for data, services and interfaces, etc., and build a more general and extensible architecture.

• Technology integration. It is necessary to maximize the advantages of each technology, to achieve the organic integration of technologies rather than simple stacking up, to obtain strong integration efficiency.

The main functions of the six modules shown in Figure 3 cover the whole process from data access to knowledge management for data service management. The combination of these functions realizes the deep integration and advanced application of spatial data for spatial information infrastructure construction.

Figure 3. Logical architecture of heterogeneous spatial database integration.

4.1. Database federation model design

We adopt Federated Database Management System (FDBMS) technology to support query and transaction processing across multiple independent database systems (Bent et al. 2008; Butenuth et al. 2007; Sheth and Larson 1990). Database federation is an integrated strategy for cross-database access to ensure data consistency and integrity. Through database federation computing, the processing logic of data from edge database can be simplified in the data computing platform. The computation cache can be reduced, some processes requiring cross-database computation can be pushed down to edge database. It fully utilizes the computing power of edge database, and improves the efficiency of data processing and analysis.

Different databases have their own Database Query Language (DQL) (Imielinski and Virmani 1999; Reisner 1981). The key point of building database federation is to establish mapping relationship between different DQL and form an Agent between different DQLs. It is necessary to complete cross-database access using Agent as a medium.

The steps for building a database federation with an extension module built into the local database are shown in Figure 4:

Figure 4. Heterogeneous database federation interaction of local server and remote server.

1. Parsing and analyzing the DQL statements of the local database to generate an AST query Tree (Abstract Syntax Tree) and form a more low-level standardized structure.

2. Connect the remote database server to determine the type of remote database and the corresponding execution logic processing branch.

3. Overwrite the AST tree as a query plan for the remote database.

4. Send the remote execution plan to the remote database and receive the return result.

The internal execution process of the extension program is shown in Figure 5.

Figure 5. Internal structure and processing logic of extension modules deployed in local database to generate database federation.

1. Analyzer creates a query plan tree for the input SQL. This query plan tree is used to represent the structure and execution order of SQL statements.

2. Executor uses the underlying API of the corresponding remote heterogeneous database to control database connection, execute commands, retrieve results, and handle notifications and events to connect to the remote server.

3. Planner creates plain text SQL statements according to the query plan tree. This process is internally called de-parsing.

4. Executor sends the plain text SQL to the remote server, and then the executor processes the received data when necessary.

We build federation among different databases based on the above extension program realized in local database, which is convenient for decoupling and database type expansion. The federation cluster of spatial database can not only make use of the advantages of different databases in non-spatial computing, but also solve the problem that some databases do not support spatial data types.

4.2. Database gateway and interface orchestration model design

The goal of heterogeneous spatial database integration system is to provide users with a unified spatial data service access interface, to realize the intelligent management and application of massive heterogeneous spatial data. Database gateway and interface orchestration is the key technology to achieve the above goals.

Database gateway is a technical method to realize the integration of heterogeneous database. It is located between client applications and multiple heterogeneous databases, and plays the role of interface transformation and request routing. The database gateway shields the technical details of heterogeneous databases through interface transformation and request routing, allowing clients to access multiple heterogeneous data sources in a consistent manner. It is one of the key technologies to realize the integration of heterogeneous database, which effectively improves the flexibility, expansibility and reusability of the system.

When back-end development is carried out, the problems of data storage and data management are often encountered. Whether using MySQL, Oracle, Cassandra, PostgreSQL, Mongo, etc., it is required to develop or integrate middleware for each data source to query these data storage medium. As the number of projects increases or the storage medium changes, the code refactoring or even redoing bring a lot of time and labor costs.

In this case, GraphQL is proposed as a set of schema query syntax that visually displays data hierarchies, with strong typed checksums and weak typed extensions (Brito, Mombach, and Valente 2019; Hartig and Pérez 2018). It only defines the protocol between the client and the data source to access the data, and access the data storage medium through the protocol transformation. The appearance of GraphQL provides a very practical solution for efficiently querying large-scale data.

The design of database gateway based on GraphQL is shown in Figure 6.

Figure 6. Database gateway based on GraphQL to realize heterogeneous database integration.

• AssembleSchema module, which is responsible for loading database protocol and schema, and assembling GraphQLSchema.

• RegisterCallback module, which is responsible for registering the callback function to access the database interface.

• ParserSchema module, which is responsible for parsing the client query JSON string, and converting to GraphQL execution pair.

• AssembleResult module, which is responsible for returning the query data structure.

As shown in Figure 7, through the database gateway, cross-domain database resources and third-party service interface resources can be integrated into a whole. By registering the schema of the third-party service and database gateway B in database gateway A. The client only needs to send a request to the database gateway A to indirectly invoke the data resources behind the third-party service and database gateway B, which simplifies the client access logic, and allows the dynamic adjustment of the back-end data resources.

Figure 7. Database gateway schema stitch to integrate cross-domain database resources and third-party service interface resources.

The main steps of GraphQL schema integration are:

1. Discover and obtain remote GraphQL service patterns.

2. Parsing these remote patterns.

3. Necessary renaming and conflict resolution.

4. Determine the type mapping across services.

5. Design a uniform GraphQL schema.

6. Parse client queries and generate sub-queries.

7. Execute sub-queries and get results.

8. Aggregate query results.

9. Return the result to the client.

10. And manage access metadata for remote services.

The above process realizes the integration and interface unification of multiple GraphQL services.

5. Deep learning based geo-semantic address mapping model

Smart city applications require the use of heterogeneous urban elements data from multiple sources. However, not all types of data have standardized formats for geographic semantics, meaning that they do not use WKT (Well-Known Text) or WKB (Well-Known Binary) (Pebesma 2018), the two formats defined by OGC for describing geometric objects. To extract the geosemantics from the raw data and normalize the urban elements information implicitly represented in different data sources, geosemantic mapping needs to be completed to facilitate subsequent spatial computing.

Address information is a significant part for smart city application, which helps city government better manage various urban resources. The ubiquitous carrying capacity of address establishes an association relationship between geographical locations and business entities represented by multi-source heterogeneous data. This address aligning operation enhances the cross-business information association and build the spatio-temporal knowledge base of urban governance.

We propose a new method to use segmental inference model for Chinese address matching as shown in Figure 8. Specifically, the Bert model is used to convert text addresses into address vectors (Devlin et al. 2018; M. Li et al. 2023). The pointer network is used to extract three address elements from the address vectors, namely region, building and road number (Vinyals, Fortunato, and Jaitly 2015). The region elements are mainly text descriptions, while building elements and road number elements are mainly digital descriptions. The key matching feature is whether the text and digital descriptions are consistent. Finally, the ESIM (Q. Chen et al. 2016) is used to match the sample address data with a standard address in terms of three elements. The matching results of the three address elements are integrated to determine whether the address pair matches.

Figure 8. Deep learning based geo-semantic inference model based on address encoding, address element extraction and element matching for Chinese address matching.

In the three sub-address elements of region, building and road number, the region element includes the information of province, city, district, street and community. Building elements include building, unit, household and other elements information. Road number elements mainly contain the road, house number and other element information. For example, when entering “4th Floor, Wudajiao Building, No. 9, Wudayuan 1st Road, East Lake High-tech Zone, Wuhan City”, the result of region network pointer extraction is “Wuhan East Lake High-tech Zone”. The result of building network pointer extraction is “Wudajiao Building 4th floor”. And the result of road number pointer network extraction is “Wudayuan 1st Road, No. 9”.

As shown in Figure 9, the Chinese address mapping model consists of the following three layers:

Figure 9. Neural network structure of Bert, PtrNet and ESIM for Chinese address matching.

(1) Address encoding layer: it is composed of Bert pre-trained model to convert the input text address into a text vector. As shown in Figure 9(a), Bert model is a dynamic word vector model. The text address is first converted into tokens. Each token is a word, and the tokens are converted into numeric codes to generate the text encoding, location encoding and segment encoding of the address. Then the text encoding, location encoding, and segment encoding are input into the Bert model to generate the corresponding address vector.

For example, if the text address “4th Floor, Wudajiao, Wudayuan Road” is input, it is divided into a plurality of tokens. The ith token is converted into the corresponding digital code E_i, that is, [E₁,E₂,E₃,…,E₁₀]. The special character [CLS] is added at the initial position of the text coding, and the special character [SEP] is added at the end. Since different addresses have different lengths, we fill different address codes with code [0] to the same length N, that is, [CLS, E₁, E₂, …, E₁₀, SEP, 0, 0, …, 0] with the length N. Then, the corresponding position codes [0,1,2,3,…,n] and sementic codes [1,1,1, …,1,0,0, …,0] are generated, in which the position codes of effective tokens are 1 and the extended invalid tokens are 0. Finally, text code, position code and segment code are input into Bert model, and the vector H of the text is obtained. Here, each token i is encoded into an M-dimensional vector H_i, and [H₀, H₁, H₂, …, H_n] constitute a vector of text addresses.

(2) Element extraction layer: it is composed of three pointer networks (PtrNet-d, PtrNet-b, and PtrNet-c) to extract the region elements, building elements and road number elements in the address respectively. A pointer network is an end-to-end generative network, which combines a seq2seq generative network and an attention mechanism. A pointer network can decode an intermediate vector generated by the encoding layer into a pointer. It points to a token in the original sequence as a generative token. Compared with the traditional seq2seq generative network, the pointer network generates tokens that are not in the text dictionary. The coding layer of a pointer network consists of a Bi-directional Long Short-Term Memory (Bi-LSTM) neural network. The decoding layer consists of aLSTM neural network.

As shown in Figure 9(b), the encoding layer of a pointer network consists of a Bi-LSTM neural network. The decoding layer consists of aBI-LSTM neural network. The computing Equation of the pointer network is:

(1) $a_j^i=u^{\mathrm{T}}\tanh \left(W_1{\mathit{e}}_{\mathit{j}}+W_2{d}_i\right),\mathrm{\forall }\mathit{j}\in \left(1,\dots ,\mathit{n}\right)$(1)

(2) $\mathit{p}\left(C_i|C_1,\dots ,C_{\mathit{i}-1},\mathit{P}\right)=\mathrm{softmax}\left(a_j^i\right)$(2)

Here ${\mathit{e}}_{\mathit{j}}$ refers to the encoding vector of the jth word in the text. $d_i$ is the hidden state of the ith word in the decoding layer. $\mathit{u}$,$W_1$,$W_2$ are the learning parameters in the pointer network. $a_j^i$ is the attention value of the jth token in the encoding layer to the ith token in the decoding layer. Then the attention value $a_j^i$ is input into the Softmax layer to obtain the probability value of each token in the encoding layer. The index of the token with the largest probability value is selected, and the corresponding token coding vector is taken as the hidden state of the next predicted word until the output stops as a special symbol “< END >”.

(3) Matching layer: it is composed of three ESIM (ESIM-D, ESIM-B and ESIM-C), to deduce whether the region factor pair, building factor pair and road number factor pair match respectively. Finally, the matching results of the three elements are integrated to judge whether the matching is possible. The ESIM model is an interactive natural language inference model, which mainly includes four layers: input coding layer, local inference layer, inference combination layer and prediction layer.

As shown in Figure 9(c), the input coding layer consists of a Bi-LSTM network, The address element vector H to be matched is encoded by Bi-LSTM to obtain the vector $\bar{\mathit{H}}$. The standard address element vector $\bar{\mathit{P}}$ is obtained in the same way. The local inference layer is a similar weight matrix E. $\bar{\mathit{H}}$ and $\bar{\mathit{P}}$ are cross-multiplied with the similar weight E, and the weighted similar vectors ${\tilde{\mathit{H}}}_{\mathit{i}}$ and ${\tilde{\mathit{P}}}_{\mathit{j}}$ are obtained. The computing Equations are shown in Equations (3)(3) $E_{\mathit{ij}}={\overline{{\mathit{H}}_i}}^{\mathit{T}}\overline{{\mathit{P}}_j}$(3) –(5(5) $\widetilde{{\mathit{P}}_j}={\sum }_i^{l_H}\frac{\exp \left({\mathit{e}}_{\mathit{ij}}\right)}{{\sum }_m^{l_H}\exp \left(e_{\mathit{mj}}\right)}\overline{{\mathit{H}}_i},\mathrm{\forall j}\in \left[1,\dots ,l_P\right]$(5) ).

(3) $E_{\mathit{ij}}={\overline{{\mathit{H}}_i}}^{\mathit{T}}\overline{{\mathit{P}}_j}$(3)

(4) $\widetilde{{\mathit{H}}_i}={\sum }_j^{l_p}\frac{\exp \left({\mathit{e}}_{\mathit{ij}}\right)}{{\sum }_m^{l_p}\exp \left(e_{\mathit{im}}\right)}\overline{{\mathit{P}}_j},\mathrm{\forall i}\in \left[1,\dots ,l_H\right]$(4)

(5) $\widetilde{{\mathit{P}}_j}={\sum }_i^{l_H}\frac{\exp \left({\mathit{e}}_{\mathit{ij}}\right)}{{\sum }_m^{l_H}\exp \left(e_{\mathit{mj}}\right)}\overline{{\mathit{H}}_i},\mathrm{\forall j}\in \left[1,\dots ,l_P\right]$(5)

Here $l_P$ stands for the token number of standard address elements. $l_H$ stands for the token number of sample address elements. $e_{\mathit{ij}}$ stands for the value of the ith row and the jth column in the similar weight matrix E, and the same for $e_{\mathit{im}}$ and $e_{\mathit{mj}}$.

The inference combination layer is to align the result of the encoding layer with the weighted similarity vector. Then, the enhancement vector of the sample address element ${\mathit{M}}_{\mathit{H}}$ and the standard address element ${\mathit{M}}_{\mathit{P}}$ are obtained. The alignment method is shown in Equations (6)(6) $M_H=\overline{H},\tilde{H},\overline{H}-\tilde{H},\overline{H}\odot \tilde{H}$(6) and (7(7) ${\mathit{M}}_{\mathit{P}}=\left[\bar{\mathit{P}},\tilde{\mathit{P}},\bar{\mathit{P}}-\tilde{\mathit{P}},\bar{\mathit{P}}\odot \tilde{\mathit{P}}\right]$(7) ). ${\mathit{M}}_{\mathit{H}}$ and ${\mathit{M}}_{\mathit{P}}$ obtain the corresponding matching $V_H$ and $V_p$ through the bidirectional LSTM network, respectively.

(6) $M_H=\overline{H},\tilde{H},\overline{H}-\tilde{H},\overline{H}\odot \tilde{H}$(6)

(7) ${\mathit{M}}_{\mathit{P}}=\left[\bar{\mathit{P}},\tilde{\mathit{P}},\bar{\mathit{P}}-\tilde{\mathit{P}},\bar{\mathit{P}}\odot \tilde{\mathit{P}}\right]$(7)

The prediction layer pools the results of the inference combination layer, and the pooling computing method is shown in Equation (8)(8) ${\mathit{V}}_{\mathit{k}\mathbf{,}\mathit{avg}}={\sum }_{\mathit{i}=1}^{l_k}\frac{{\mathit{V}}_{\mathit{k}\mathbf{,}\mathit{i}}}{l_k},{\mathit{V}}_{\mathit{k}\mathbf{,}\mathit{m}\mathbf{ax}}=\underset{\mathit{i},\mathit{i}\in l_k}{\max \left({\mathit{V}}_{\mathit{k}\mathbf{,}\mathit{i}}\right)}$(8) . Here ${\mathit{V}}_{\mathit{k}\mathbf{,}\mathit{i}}$ represents the vector of the ith word in the matching vector k. ${\mathit{V}}_{\mathit{k}\mathbf{,}\mathit{avg}}$ represents the average pooling vector of the matching vector ${\mathit{V}}_{\mathit{k}}$. ${\mathit{V}}_{\mathit{k}\mathbf{,}\mathit{max}}$ represents the maximum pooling vector of the matching vector ${\mathit{V}}_{\mathit{k}}$. Then the pooled vectors of the sample elements and the standard elements are input into the fully connected layer, and the final inference result is obtained by using Softmax.

(8) ${\mathit{V}}_{\mathit{k}\mathbf{,}\mathit{avg}}={\sum }_{\mathit{i}=1}^{l_k}\frac{{\mathit{V}}_{\mathit{k}\mathbf{,}\mathit{i}}}{l_k},{\mathit{V}}_{\mathit{k}\mathbf{,}\mathit{m}\mathbf{ax}}=\underset{\mathit{i},\mathit{i}\in l_k}{\max \left({\mathit{V}}_{\mathit{k}\mathbf{,}\mathit{i}}\right)}$(8)

We enter the sample text address pair. The address pair contains a sample address KeyAddr and a standard address StdAddr. The encoding tensor $\mathit{H}1$ of KeyAddr and the encoding tensor $\mathit{H}2$ of StdAddr are obtained by Bert model, respectively. $\mathit{H}1$ and $\mathit{H}2$ pass through the pointer network layer respectively to obtain the region encoding tensor $\mathit{D}1$, $\mathit{D}2$, building encoding tensor $\mathit{B}1$, $\mathit{B}2$, and road number encoding tensor $\mathit{C}1$, $\mathit{C}2$ of $\mathit{H}1$ and $\mathit{H}2$ respectively. We input $\mathit{\hspace{0.17em}}\mathit{D}1$ and $\mathit{D}2$ into area inference network ESIM-d to obtain area inference result res-d. We input $\mathit{B}1$ and $\mathit{B}2$ into building inference network ESIM-b building inference result res-b. We input $\mathit{C}1$ and $\mathit{C}2$ into road number inference network ESIM-c to obtain road number inference result res-c. Finally, the result is obtained by synthesizes res-d, res-b and res-c.

6. Spatio-temporal information integration and knowledge reasoning

Deep learning based geo-semantic address matching method automatically associates the urban governance elements information (people, buildings, objects, events and organizations) to their corresponding geographic entities. The urban elements spatio-temporal data for each geographic entity should be further integrated to support the urban governance.

6.1. Design of data aggregation computing model

To meet the cross-domain information requirements in complex city scenarios, it is necessary to aggregate dispersed and heterogeneous data to provide a unified data view. Through correlation analysis, aggregation and transformation of multi-dimensional information, we build automated pipelines to execute complex data mining and analysis processes in different spatial, temporal and scale aspects to maximize the value of each individual data source.

Based on the design concept of stream and batch integration, a unified resource scheduling model are adopted to realize the collaborative execution of stream and batch tasks. The stream processing is regarded as the processing of unbounded dataset, and the batch processing is regarded as the processing of bounded dataset. Only one unified stream batch system is deployed, which simplifies the deployment, operation and maintenance workload and reduces the system complexity.

The basic framework of data aggregation computing is shown in Figure 10. We divide data processing into two stages: SE and TL. SE is the stage of source connection and extraction, while TL is the stage of transformation and loading. SE establishes connection and access control for different types of data sources, and extracts data into internally recognizable schema. TL converts various computing operators into other schema or formats, and push the connector from sink end to the target end. Since the source and the target may use different storage, and each storage has different single-thread throughput and delay. The separation of SE and TL allows the system to dynamically configure multiple SE or TL nodes to operate in parallel according to the source extraction pressure and target back pressure.

Figure 10. Framework for distributed cross-domain resource aggregation computing.

In the software architecture, data aggregation operators are distributed to different nodes to relieve the throughput pressure brought by the real-time massive heterogeneous data resources in cities. In the specific implementation, SE and TL are distributed to different nodes. The data aggregation operator scheduling methods on each node are divided into three modes:

1. Record-by-record processing, the operator is called for each database record state change, which is time-sensitive but with high I/O.

2. Batch processing, the operators process all data after filtering out the bounded dataset, which reduces I/O but has high throughput.

3. Pipeline processing, operators are dynamic scheduled according to the data change status, number of records, threshold, time window and other different caching methods. It takes both real-time and I/O into consideration.

The specific data aggregation model can be divided into different types according to application scenarios. For each type, different integration operators are formed and dynamically scheduled in the above integration computing framework. Here, only the abstract model of the same type of operators is illustrated.

• Entity data aggregation operator

We use relation extraction to eliminate the ambiguity in the information expression of different entities, and determine whether two entities represent the same entity. We find and associate the same entity between different datasets, or different entities with a certain association relationship. Suppose there are two sets of entities $\mathit{\hspace{0.17em}}\mathit{A}$ and $\mathit{B}$, whose elements are $a_i$ and $b_j$. Entity alignment can be represented as a match matrix $\mathit{M}$. $M_{\mathit{ij}}=1$ represents $a_i$ and $b_j$ matches. $M_{\mathit{ij}}=0$ represents $a_i$ and $b_j$ do not match. We use an objective function to improve the quality of entity alignment:

(9) $\underset{\mathit{M}}{max}\sum _{\mathit{i},\mathit{j}}{M}_{\mathit{ij}}\cdot \mathit{s}\left(a_i,b_j\right)$(9)

Where $\mathit{s}\left(a_i,b_j\right)$ is the similarity function for calculating $a_i$ and $\mathit{\hspace{0.17em}}{b}_j$.

• Set data aggregation operator

The data collected, sorted and cleaned in different data sources are converted (such as ETL) and then loaded into a new data source to provide data consumers with a unified data aggregation.

Suppose set $\mathit{A}\left(a_1,a_2,\dots ,a_n\right)$ and $\mathit{B}\left(b_1,b_2,\dots ,b_m\right)$. They all have a common attribute $\mathit{c}$, which can be the result of entity alignment. Then the integration operation of set $\mathit{\hspace{0.17em}}\mathit{A}$ and $\mathit{B}$ can be expressed as:

(10) $\mathit{A}\bowtie { }_c\mathit{B}=\left\{a_i,b_j|a_i.\mathit{c}=b_j.\mathit{c}\right\}$(10)

Where, $\mathit{A}\bowtie { }_c\mathit{B}$ denotes that the elements in the set $\mathit{A}$ and $\mathit{B}$ are associated on common attribute $\mathit{c}$, and the result is a new set. Each element of this new set has the set attribute value of $\mathit{A}$ and $\mathit{B}$. $a_i.\mathit{c}$ and $\mathit{\hspace{0.17em}}{b}_j.\mathit{c}$ represents the attribute value corresponding to attribute $\mathit{c}$ in the set $\mathit{A}$ and $\mathit{B}$. The resulting new set can be represented as:

(11) $\mathit{AB}\left(a_1,a_2,\dots ,a_n,b_1,b_2,\dots ,b_m\right)$(11)

That is, all the elements of set $\mathit{A}$ and $\mathit{B}$ are merged into a larger, higher-dimensional set.

• Feature data aggregation operator

Data features from different sources are aggregated to improve feature expression and classifier performance.

Suppose there are $\mathit{n}$ set of features $F_1,F_2,\dots ,F_n$, and each set contains $\mathit{m}$ feature vectors $f_{\mathit{i}1},f_{\mathit{i}2},\dots ,f_{\mathit{im}}$ respectively, where $\mathit{i}$ is the index of set. Feature integration can be expressed as the operation of merging feature vectors in $\mathit{n}$ feature sets into a single vector, for example:

(12) ${{\mathit{f}}_{\mathit{ij}}}^{\prime }=\mathrm{\varnothing }\left(f_{1\mathit{j}},f_{2\mathit{j}},\dots ,f_{\mathit{nj}}\right)$(12)

Where $\mathrm{\varnothing }$ is the feature fusion function, which can be a linear weighted or nonlinear neural network model. After the feature integration, a new feature vector $\mathit{\hspace{0.17em}}{f}_{ij}^{\prime }$ is obtained, which can be used for further analysis and processing. For example, in the task of public opinion analysis, it is necessary to integrate the semantic and grammatical features of the text and the geographical environment features to improve the accuracy and robustness of the urban infrastructure location model.

• Model data aggregation operator

We combine the predictions of different models to improve the accuracy and robustness of the model.

Suppose there are $\mathit{k}$ models, and the computing result of each model is $y_k$, then model data aggregation can be expressed as:

(13) $\hat{y}=\frac{1}{k}{\sum }_{k=1}^K{y}_k$(13)

Where $\mathit{\hspace{0.17em}}\hat{y}$ represents the computing result of the model after integration. For example, the time series model established based on historical traffic flow data is integrated with the regression model established based on geographical traffic influencing factors (road attributes, weather, POI distribution, etc.). The prediction accuracy of traffic flow is improved by integrating the prediction results of the two types of models.

• Decision data aggregation operator

We integrate different decision rules or strategies to improve the accuracy and reliability of decisions.

Considering the weight of decision rules corresponding to the computing results of different models, we assume that the weight of decision rules of model $\mathit{k}$ is ${\alpha }_k$. Then decision data aggregation can be expressed as:

(14) $\hat{y}={\sum }_{k=1}^K{\alpha }_k{y}_k$(14)

In special cases, when two datasets $\mathit{A}$ and $\mathit{B}$ contain an attribute at the same time, and the decision uses the attribute of the dataset $\mathit{A}$ as the final attribute after integration, there will be ${\alpha }_A=1,\mathit{\hspace{0.17em}}{\alpha }_B=0$.

• Spatio-temporal data aggregation operator

We integrate spatial information from different sources and data at different time scales to extract historical trends and predict future development trends.

The datasets of different spatio-temporal standards are first aligned and merged. The merged datasets are then used as the judgment basis of other entities’ alignment, so as to realize the spatio-temporal aggregation for data analysis and mining. Suppose there are two spatio-temporal datasets $S_1$ and $S_2$. Their domain (geographical location) is $D_1$ and $D_2$, and the time range on the domain is $T_1$ and $T_2$. Suppose there exist $\mathit{k}$ normalized spatio-temporal objects $\left(D_{\mathit{g}1\mathit{t}1},D_{\mathit{g}2\mathit{t}2},D_{\mathit{g}3\mathit{t}3},\dots ,D_{\mathit{gktk}}\right)$ at the same time, then the process of spatio-temporal data aggregation can be expressed as follows:

The intersection of domain: assume that the domain integration is $\mathit{D}=D_1\cap D_2$, it means that the domain of $S_1\mathit{\hspace{0.17em}}$ and $S_2\mathit{\hspace{0.17em}}$ is intersected.

The intersection of time: assume that the time integration is $\mathit{T}=T_1\cap T_2$, it means that the time axis of $S_1\mathit{\hspace{0.17em}}$ and $S_2\mathit{\hspace{0.17em}}$ is intersected.

The intersection of spatio-temporal objects: all $\mathit{k}$ normalized spatio-temporal objects are intersected with $\mathit{D}\times \mathit{T}$, and the spatio-temporal objects that intersect with $\mathit{D}\times \mathit{T}$ in time interval and spatial range are integrated together, namely:

(15) $D_{\mathit{combine}}=\left\{D_{\mathit{gi}}\cap D_1\cap D_2\right\},T_{\mathit{combine}}=T_1\cap T_2$(15)

Where $D_{\mathit{gi}}$ represents the $\mathit{i}$th normalized spatio-temporal object. $D_{\mathit{combine}}$ represents all sets of spatio-temporal objects intersecting with $\mathit{D}\times \mathit{T}$. $T_{\mathit{combine}}$ represents the time interval existing in the set of spatio-temporal objects. We can intersect the elements in dataset $\mathit{A}$ and $\mathit{B}$ with the objects in $D_{\mathit{combine}}$ respectively, so as to integrate the elements $\mathit{A}$ and $\mathit{B}$ through the spatio-temporal relationship. If $A_x$ and $B_y$ intersect with $D_{\left\{\mathit{gi}\right\}}\mathit{\hspace{0.17em}}$ at the same time, further information integration can be carried out for $A_x$ and $B_y$.

• Multi-level data aggregation operator

Data from different levels are integrated to extract multidimensional information and perform cross-level analysis.

We combine entity features at different levels to achieve comprehensive data analysis and mining. Assume that the level is $\mathit{L}$, the characteristics of the entity $\mathit{i}$ are ${\mathrm{x}}_{\mathrm{i},\mathrm{l}}$, the multi-level data aggregation operator can be expressed as:

(16) $\widehat{x_i}=\left\{\begin{array}{c}{x}_{\mathit{i},\mathit{l}},\mathit{if}\mathit{L}=1\\ f_l\left(x_{\mathit{i},\mathit{l}},\widehat{x_{p_i}}\right),\mathit{if}\mathit{L}>1\end{array}\right.$(16)

Where, $\mathit{L}$ represents the maximum level, $p_i$ represents the parent node of entity $\mathit{i}$, $f_l$ represents the feature fusion function at level $\mathit{\hspace{0.17em}}\mathit{l}$, and $\widehat{x_{p_i}}$ represents the feature representation of the parent node of entity $\mathit{i}$ at the previous level.

For example, the accuracy of climate prediction can be improved by integrating the output data of the global climate model with the ground measured data. Climate models provide global and long-term data on atmospheric motion, but may differ from ground reality. Ground-based measurements provide more precise data, but with limited spatial scope. By integrating the two types of data, gaps in the precision and breadth of data can be filled and the accuracy of climate predictions can be improved.

Data aggregation operators based on entity, set, feature, model, decision set, spatio-temporal information and multi-level are built into the data aggregation computing framework for unified scheduling and operation, supporting the integration of cross-domain data.

6.2. Knowledge construction and knowledge inference model design

Cross-domain data aggregation helps to form spatio-temporal knowledge system. Compared with data, knowledge contains richer semantic information. Data only includes less semantic descriptions of facts, while knowledge uses ontology concepts to express rich semantic relationships, such as “belonging”, “part-whole”, “causation”, and so on. This makes knowledge easier to understand and reason. By standardizing the spatio-temporal knowledge of ontology modeling and representation, spatio-temporal data can be shared and reused among multiple applications (Gagnon 2007; Spyns, Meersman, and Jarrar 2002). Among them, knowledge graph has strong logical reasoning ability, and new knowledge can be deduced automatically based on knowledge graph to realize the expansion and enrichment of knowledge.

Knowledge in smart city includes five types of urban elements (people, buildings, organizations, events, objects) with their attributes, inter-entity relations, which are closely related to spatio-temporal information (X. Chen, Jia, and Xiang 2019; Hmelo-Silver 2003; Israilidis, Odusanya, and Mazhar 2021). For specific domain scenarios, urban elements are classified in more detail, and their attributes can be classified as cross-domain common attributes and domain-specific attributes. The relationship between these elements also has certain differences and extensions in different fields.

Through data governance, the original data of people, buildings, events, objects, organizations and other urban elements obtained from the perception system are transformed into the basic knowledge expression form composed of structured data, spatio-temporal data and unstructured data. These basic knowledge are then fused and calculated to form advanced knowledge forms, such as block database, AI model, knowledge representation, and knowledge graph.

• Knowledge construction

The relationships among block database, AI model, knowledge representation and knowledge graph are as follows:

1. Knowledge representation is the basis of building knowledge base and knowledge graph. Different knowledge representation methods, such as triples, texts, concept trees, etc., can be used to express different types of knowledge and constitute the basic elements of knowledge base and knowledge graph.

2. Knowledge base integrates knowledge from different sources and fields. It integrates knowledge expressed in different knowledge representation ways to realize knowledge organization and storage. Knowledge base provides source knowledge for constructing knowledge graph and AI model.

3. Knowledge graph further organizes knowledge in knowledge base. It establishes the relationship between knowledge, and expresses its logical relationship and internal rules. It has stronger ability of reasoning and new knowledge discovery. Knowledge graph can also provide rich knowledge support for AI models.

4. AI model uses machine learning technology, based on a large number of data and knowledge training model, to realize the prediction, classification and judgment of new data. The knowledge required for training AI models can be derived from knowledge base or knowledge graph.

In practice, the above four elements will promote and optimize each other: the optimized knowledge representation can improve the construction quality of the knowledge base and knowledge graph.

We define the smart city ontology model as the structure and relationship collection of different geographic entities with their corresponding various urban elements. As shown in Figure 11, the smart city knowledge ontology contains the urban elements information of people, buildings, events, objects and events. “People” describes the natural person class in various fields of application. “Buildings” describes the category of urban spatial buildings or locations. “Events” describes the events that occur at a specific time and place in the city. “Objects” describes the material objects that exist outside the human body, which meet people’s social needs and are controlled or dominated by people. “Organizations” describes all kinds of enterprises, institutions and social groups in the city.

Figure 11. Smart city ontology model based on geographic entities with their corresponding 5 urban elements.

The attributes of these five urban elements are shown in Table 1.

Table 1. Urban element types of people, buildings, events, objects and organizations.

Elements	Stats
People	Name, sex, height, weight, age, date of birth, nationality, native place, place of household registration, certificate number, educational background, political appearance, contact information, etc.
Buildings	Name, function, location, address, size, capacity, terrain, etc.
Events	Time, subject, place, type, description, effect, cause, etc.
Objects	Name, function, model, shape, size, color, weight, material, duration, etc.
Organizations	Name, type, duties/business scope, personnel size, previous name, telephone number, email address, website, address, organization code, uniform social credit code, date of establishment, etc.

This smart city ontology model is constructed based on the geographic entity block data structure. Due to the ubiquitous characteristics of spatial information, the unique address of geographic entity is selected to associate multiple urban elements, and finally generate multiple data blocks to form the knowledge graph of smart city. As shown in Figure 12. In the process of ontology-based knowledge construction, we first define the ontology, including the definitions of entities, attributes, relations and events, to express the conceptual system of the knowledge domain. Then, knowledge extraction is carried out to obtain entities, attributes, relationships and events information from multi-source unstructured data through natural language processing and data mining technology. Finally, the knowledge from different sources is integrated. The same entity is merged by entity alignment, and the entity content is enriched by attribute filling to build a structured knowledge graph.

Figure 12. Smart city ontology model construction process, from ontology definition, knowledge extraction to knowledge fusion.

Based on the above construction methods, the construction process of smart city knowledge goes through the re-organization and structuring of the original data to obtain the basic knowledge. Then, the basic knowledge is integrated, computed and extracted to obtain advanced knowledge, forming a smart city knowledge expression framework, as shown in Figure 13.

Figure 13. Smart city knowledge representation framework, from collecting original data to obtain the advanced knowledge.

• Knowledge reasoning

To conduct knowledge reasoning on the ontology model of urban knowledge graph, two methods of rule inference and graph pattern inference can be adopted:

1. Rule reasoning. This is a reasoning method based on if-then rule. A series of reasoning rules are defined in the knowledge map, and then results can be obtained when matching if conditions. For example, if a person lives in a building and the purpose of the building is residence, it can be inferred that the person’s residence type is resident. If an event occurs in a certain location and the location has the function of a hospital, it can be inferred that the type of the event may be a medical event. With a large number of such rules, new knowledge can be obtained by chain reasoning.

2. Graph pattern reasoning. This is a type of reasoning that looks for graph patterns that occur frequently in the knowledge graph. For example, if there is a graph pattern in the knowledge graph: people – live – building – located – place, and the property of the place is a commercial area, then it can be inferred that the person may be engaged in business activities. If there is an event – happening in a – building – use is a – school, and the type of event is an accident, then it can be inferred that the event may be a student-related event. New spatio-temporal knowledge can be obtained by probabilistic inference through frequent spatio-temporal correlator patterns in the statistical graph. The two types of reasoning can be combined, either for deterministic reasoning based on defined rules, or for probabilistic reasoning based on automatically discovered graph patterns to verify and enhance each other.

As the knowledge graph continues to be enriched, more hidden rules and graph patterns can be discovered to achieve continuously optimized and improved knowledge reasoning capabilities. By integrating information from different domains, knowledge graph can promote cross-domain collaboration, resource sharing and collaborative innovation in smart cities.

7. Experiment results and project conclusions

By using the above technical framework, a data governance platform that supports cross-domain spatio-temporal big data fusion can be built efficiently. The platform facilitates the fusion of geographic entities with unified addresses, the aggregation of cross-domain spatio-temporal data, and the integration of cross-domain vector tile services. Therefore, it can quickly build a resource collaboration data center that supports spatio-temporal information management.

In this section, we first show the experiment results of our proposed Bert+PtrNet+ESIM method for geo-semantic address mapping. Then, we present our constructed spatio-temporal data fusion platform and illustrate how this platform supports the Wuhan city urban management.

7.1. Experimental result for Chinese segmentation address mapping

To verify the validity of the Chinese address matching model based on geo-segmentic inference, this paper executes deep learning experiments on the open dataset with about 300,000 samples provided by Lin et al. (2020), and on the real dataset with about 240,000 address samples from real urban governance business in Shenzhen.

We select three Chinese address matching methods for comparison, namely, address matching based on Rouge-L, address matching based on Word2vec+ESIM (Lin et al. 2020), and address matching based on Bert+ESIM. We take accuracy, recall rate and F1-Score as the evaluation criteria.

For the Chinese address matching method based on Rouge-L, we set the multi-paragraph editing threshold, obtain multiple test results, and select the highest F1-score as the result.

For the Word2vec+ESIM Chinese address matching method, we use Word2vec model to convert text addresses into word vectors, and then match the address vectors based on ESIM model.

For the Bert+ESIM Chinese address matching method, we use Bert model to convert text addresses into address vectors, and then use ESIM model to match the address vectors. In the training stage, the fine-tuning method is used to train Bert and ESIM.

The program code is written in Python language, and the deep learning model is implemented by PyTorch framework. The experimental hardware adopted are Xeon 5218 CPU, 32GB memory and Tesla V100 graphics card.

The experimental results of the four methods on the open dataset are shown in Table 2. Compared with Rouge-L method, the F1-Scores of the three deep learning-based address matching methods are more than 15% higher. Bert+ ESIM method performs the best in accuracy, recall rate and F-Score, while Bert+Ptrnet+ESIM method has the highest precision score. The three deep learning address matching methods have a very small overall performance gap, with less than 0.01 difference in the F1-Score.

Table 2. Matching result on open dataset.

	Accuracy	Precision	Recall-Rate	F1-Score
Rouge-L	0.833	0.737	0.913	0.816
Word2vec+ESIM	0.967	0.960	0.976	0.968
Bert+ESIM	0.972	0.963	0.981	0.972
Bert+PtrNet+ESIM	0.969	0.971	0.965	0.968

The experimental results of the four methods on real datasets are shown in Table 3. The F1-Score of Rouge-L method is only 0.52. The F1-Score of Word2vec+ESIM is 0.871, and the F1-Score of Bert+ESIM is 0.956, which is 7% higher than Word2vec+ESIM. The Bert+Ptrnet+ESIM method has the best performance, and the F1-Score is 0.993, improves about 12.2% compared with Word2vec+ESIM.

Table 3. Matching result on real address dataset.

	Accuracy	Precision	Recall-Rate	F1-Score
Rouge-L	0.596	0.500	0.520	0.520
Word2vec+ESIM	0.867	0.847	0.896	0.871
Bert+ESIM	0.955	0.955	0.968	0.956
Bert+PtrNet+ESIM	0.993	0.991	0.996	0.993

The four methods show great differences between the open dataset and the real dataset. The main reason is that the sample text features in the open dataset are relatively simple, and the differences are obvious. Therefore, the model only needs to learn less semantic features to judge whether the address matches.

However, in the real dataset, the address to be matched is quite different from the standard address structure, which leads to the unsatisfactory performance of the Rouge-L method on the real dataset. Bert+ESIM performs better than Word2vec+ESIM since the word vector generated by Word2vec is static, while the word vector generated by Bert is dynamic. For example, in the address of “Shenzhen Avenue, Nanshan District, Shenzhen”, the two “Shenzhen” in the address generate the same word vector in the Word2vec model, while the Bert model generates different word vectors, which better represents the text semantics.

Compared with the other three methods, the Bert+PtrNet+ESIM method proposed in this paper has the best performance in the real dataset, because it uses the pointer network to extract and match different address elements respectively. The road number ESIM model and the building ESIM model are more sensitive to digital information. It improves the discrimination ability of detail information in the address. The regional ESIM model reduces the sensitivity of text address length. It improves the matching ability of a large range of administrative divisions (such as provinces, cities and districts), thus improving the matching accuracy of the model overall. In addition, the model also supports the design of multiple sub-address matching modules according to different scenarios to learn features respectively, to improve the model’s address matching accuracy.

7.2. Urban spatio-temporal data fusion center construction in Wuhan city management

Our constructed spatio-temporal data fusion center aims to solve the problem of data fusion and service integration in the Wuhan city urban governance scenarios. Wuhan is an inland city of China with a population of over 20 million. Since this project was launched in September 2018, this platform has integrated 34 types of urban management elements, established 53 types of relationships among 32.84 million entities, and formed 560 million triple data structures. For stock data, we realized the automatic establishment of the association between 82% entity objects through address normalization and matching methods. In terms of incremental data, we normalized the address information, and provided a unified address search, location, coding and parsing service interface for different business departments, realizing the association ratio of incremental data entity objects as 99.8%.

As shown in Figure 14, we label social governance elements, and generate geographical entities of different regions on the geographic base. Database federation synchronizes the governance elements data from the remote database of different departments to the local data center. It realizes cross-domain and cross-departmental data association, and forms a spatio-temporal data analysis framework with physical separation and logical unity. Geographical entity addresses are used to associate the urban information, such as personnel and their address information, enterprise and registration place information, building geometry and address information, event description information, urban management unit information, etc. Cross-domain data aggregation operators are used to automatically complete the address matching and spatial relationship matching, and associate the government entity with the geographical entity. The knowledge map is used to identify the potential patterns between different geographical entities. It is combined with the government knowledge base and rule base to provide data support for different AI models, such as event assignment, similar disposal process recommendation, importance and urgency judgment, etc. Through the cross-domain service integration function access, we dynamically integrate the business information related to geographical entity topics with the published vector tile services, and form a unified vector tile service for government management.

Figure 14. Wuhan spatio-temporal data fusion center framework, from cross-domain geographic entity fusion, cross-domain spatio-temporal data aggregation to cross-domain vector tile service integration.

Our spatio-temporal data fusion platform supports the intelligent urban operation in Wuhan city. When faced with cross-domain urban governance problems, this platform supports data governance teams integrate different data formats, semantics and interface protocols to achieve the convergence, integration and optimization of cross-domain resources. It helps enterprises and organizations share data and services among multiple systems and platforms, optimize business processes and improve operational efficiency. Since this platform was launched, the daily average number of calls to the related information across departments peaked at more than 1 million times, and the number of calls during the whole COVID-19 epidemic exceeded 1 billion times. We use a real complaint case to illustrate the intelligent handling process of urban events:

Take a telephone work order scene as an example. When the telephone operator listens to the citizens’ complaints, he records the whole conversation process through voice transcription, and uses the above technical framework to automatically analyze the context and intelligently extract the key information:

1. Match the colloquial location description with the unified address, and associate the information of residents, enterprises, events that have happened and building environment related to the location.

2. Match the characteristics of these information with the rule model in the knowledge base, and actively recommend the disposal process of similar events in the past.

3. According to the characteristics of people and enterprises involved in the incident, as well as the complainant’s recent reporting behavior, reporting frequency and the intensity of appeal, the importance and urgency of incident handling are comprehensively predicted.

4. Automatically assign the handling department through matching the characteristics of the incident.

5. After the above information is structured, it will be automatically filled into the work order form, and the work order will be pushed through manual review.

This spatio-temporal data fusion platform greatly improves the efficiency of urban event governance. Before the intelligent system is adopted, the operator needs to read the incoming content repeatedly and rely on experience to analyze and understand the problem. Moreover, the operator’s knowledge background is limited, which often leads to misjudgment of the severity of the complicated events, or assigns the complicated events to the wrong departments. It significantly affects the efficiency of government affairs implementation. After using this system, the automatic form filling and confirmation can be completed within 30s after the end of the call, which improves the efficiency of cross-department resource integration by 10–30 times. Multi-dimensional association of urban events with specific regions, specific people and specific organizations can greatly expand the dimension of information analysis for complex urban governance and provide basic information support for various application scenarios.

8. Summary

In this research, we propose a new spatio-temporal data fusion framework to support complex urban governance. It involves frontier technologies in big data, artificial intelligence, spatial computing, cloud computing, API management, database and other directions. It realizes the sharing and collaboration of cross-domain spatial data resources, and provides basic data support for the complex urban governance scenarios. The key points of building a spatio-temporal data fusion center are:

1. A new address-based multi-source data fusion method: We proposed a data association fusion method based on unstructured identification-address. Address integrates the related urban elements data of people, buildings, things, events and organizations for each single entity, thus reflecting the running state of the city more truly and comprehensively. Using the Bert+PtrNet+ESIM based geo-sematic address mapping method, we align the geographic entities with different urban elements with high F1-Score of 0.993. It lays the foundation for cross-domain data aggregation and service integration.

2. Inter-departmental data federation and heterogeneous data management: By building a database federation, the social governance data dimensions can be dynamically expanded through online integration and management. This method not only ensures the independence and security of the data for each department, but also enables the effective analysis of inter-departmental data to provide more comprehensive decision support. Through continuous data updating and quality control, the platform can ensure the timeliness and accuracy of information, thus better reflecting the real-time situation of the city.

3. Spatial information analysis based on knowledge mapping: Through address aggregation and unique ID index generation based on spatial semantics, the platform can accurately locate and analyze all kinds of data related to geographical location and generate knowledge map model. These data processing modes can dig and express the complex relationships between entities in a deeper level, provide richer and more accurate spatial information analysis, and assist urban operation and management decision-making.

4. Intelligent reasoning and automated decision support: The AI reasoning service quickly extracts key information from urban governance data according to advanced algorithm models for intelligent analysis and prediction. It not only improves the processing speed, but also makes the response to complex urban management events accurately and timely.

The real spatio-temporal data center construction experience in Wuhan indicates that our method has significantly improved the efficiency of urban management and services. By associating the block data of different geographic entities with unified address, and using the technologies of database federation and knowledge map, we not only optimized the event handling process, shortened the handling time, but also improved the accuracy of event handling and the intelligent level of decision-making to support urban planning and social governance. This study not only confirms the validity of our proposed concepts and methods, but also provides experience and methodological guidance for the future research and practice of smart cities.

At present, the research on this technical system has made some progress, but the related theories and technologies need to be further strengthened and innovated. In the future, we will further explore the application of spatio-temporal data fusion technology in other urban management fields, and how to further improve the intelligence and automation level of the system. We will devote ourselves to carrying out deep semantic understanding and correlation, and realizing knowledge-based reuse. We will build a flexible integration framework and engine based on micro-service architecture and DevOps concept, which can quickly respond to changes in integration requirements. We will also adopt privacy computing technology and blockchain technology to ensure data security and intellectual property protection, and realize more efficient and credible system operation.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work is supported by Shenzhen Institute of Artificial Intelligence and Robotics for Society (AIRS) [Grant numbers BN00202309021 and AC01202201003-05], the National Natural Science Foundation of China [Grant number 62202410], and Shenzhen Science and Technology Program [Grant number JCYJ20220530143808019].

Notes on contributors

Yuejin Li

Yuejin Li received BS degree in Economics from Wuhan University, China in 2016. He received two master’s degrees from London School of Economics and Political Science in 2017, and University of St. Andrews in 2018 respectively. He received the PhD degree from the Chinese University of Hong Kong, Shenzhen. His research interests include cloud/edge computing, Internet of things, and artificial intelligence.

Shengpeng Chen

Shengpeng Chen received his BS degree in Information Engineering from Wuhan University, China, in 2003. He received his master’s degree in Cartography and Geographic Information Systems from the same university in 2006. His research interests include governmental big data, spatio-temporal big data, artificial intelligence, and platform-level product development. He is currently employed in Wuhan, China, at Wuda Geoinformatics Co., Ltd, serving as the Technical Director for the Urban Governance Business Group.

Kai Hwang

Kai Hwang is an IEEE life fellow, and a Presidential Chair Professor at the Chinese University of Hong Kong (CUHK), Shenzhen, China. He received the PhD in EECS from the University of California at Berkeley. He has worked at Purdue University and University of Southern California for 45 years prior to joining the CUHK in 2018. He has published 10 books and over 300 papers with a Google citation of 26,000 times and h index of 69. He received the first CFC Outstanding Achievement Award and Lifetime Achievement Award from IEEE CloudCom. In 2020, he received the Wu Wenjun AI Natural Science Award for his recent work on AI-oriented cloud computing. Dr. Hwang is currently ranked among the top 2% mostly cited scientists in the world.

Xiaoqiang Ji

Xiaoqiang Ji received his PhD degree from Columbia University, New York, at the department of mechanical engineering. He is a research assistant professor with the School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen. His research interests include intelligent control theory and control systems.

Zhen Lei

Zhen Lei received his Bachelor of Business Administration from Wuhan University in 2004. His research interests include Geographic Information Systems, spatial data governance, and big data operations. He is currently employed in Wuhan, China, at Wuda Geoinformatics Co., Ltd, serving as the Head of the Industry Market Department for the Urban Governance Business Group.

Yi Zhu

Yi Zhu received his BMgt from Wuhan University of Technology in 2004 and his MEng from Wuhan University in 2014. His research interests include spatial-temporal information management, government data governance, urban comprehensive management, and social governance and services. He is currently employed in Wuhan, China, at Wuda Geoinformatics Co., Ltd, serving as the Deputy General Manager for the Urban Governance Business Group.

Feng Ye

Feng Ye received his BS degree in Geographic Information Systems from Shandong Jiaotong University in 2008. His research interests include spatiotemporal big data, government big data, unified addresses, urban governance, and artificial intelligence. He is currently employed in Wuhan, China, at Wuda Geoinformatics Co., Ltd, serving as the Deputy Director of the Urban Governance Business Group’s Industry Market Department.

Mengjun Liu

Mengjun Liu received his BS degree in Computer Science from South-Central Minzu University, China, in 2006. His research interests include governmental big data, spatial-temporal big data, the Internet of Things, and artificial intelligence. He is currently employed in Wuhan, China, at Wuda Geoinformatics Co., Ltd, serving as the Engineering Director of the Wuhan Business Center for the Urban Governance Business Group.