Urban Visual Intelligence: Studying Cities with Artificial Intelligence and Street-Level Imagery

Abstract

The visual dimension of cities has been a fundamental subject in urban studies since the pioneering work of late-nineteenth- to mid-twentieth-century scholars such as Camillo Sitte, Kevin Lynch, Rudolf Arnheim, and Jane Jacobs. Several decades later, big data and artificial intelligence (AI) are revolutionizing how people move, sense, and interact with cities. This article reviews the literature on the appearance and function of cities to illustrate how visual information has been used to understand them. A conceptual framework, urban visual intelligence, is introduced to systematically elaborate on how new image data sources and AI techniques are reshaping the way researchers perceive and measure cities, enabling the study of the physical environment and its interactions with the socioeconomic environment at various scales. The article argues that these new approaches would allow researchers to revisit the classic urban theories and themes and potentially help cities create environments that align with human behaviors and aspirations in today’s AI-driven and data-centric era.

自19世纪末至20世纪中期卡米洛·西特、凯文·林奇、鲁道夫·阿恩海姆和简·雅各布斯等学者开展开创性研究以来, 城市的视觉维度一直是城市研究的重要主题。几十年后, 大数据和人工智能正在彻底改变人们的活动、感知及其与城市的互动。本文回顾了城市外观和功能的文献, 以阐述如何运用视觉信息来理解城市外观和功能。引入了一个概念性框架——城市视觉智能, 系统地阐述了新的图像数据和人工智能技术如何重塑研究人员感知和度量城市的方法, 从而在不同尺度上研究自然环境及其与社会经济环境的相互作用。这些新方法使得研究人员重新审视经典的城市理论和主题, 并在人工智能驱动和数据为中心的当今时代, 帮助城市建设符合人类行为和愿望的环境。

La dimensión visual de las ciudades ha sido materia fundamental en los estudios urbanos desde el trabajo pionero de estudiosos que actuaron desde finales del siglo XIX hasta mediados del siglo XX, tales como Camillo Sitte, Kevin Lynch, Rudolf Arnheim y Jame Jacobs. Algunas décadas después, big data y la inteligencia artificial (IA) están revolucionando el modo como la gente se mueve, siente e interactúa con las ciudades. Este artículo hace una revisión de la literatura sobre el surgimiento y la función de las ciudades para ilustrar cómo la información visual ha sido utilizada para comprenderlas. Se presenta un marco conceptual, la inteligencia visual urbana, para elaborar sistemáticamente en el modo como una nueva imagen de las fuentes de datos y las técnicas de IA están reconfigurando la manera como los investigadores perciben y miden las ciudades, permitiendo el estudio del entorno físico y sus interacciones con el entorno socioeconómico a varias escalas. En el artículo se arguye que estos nuevos enfoques podrían permitir a los investigadores volver sobre las teorías y temas urbanos clásicos, y potencialmente ayudar a las ciudades a crear escenarios geográficos que se alineen con los comportamientos y aspiraciones humanos en esta era actual impulsada por la IA y centrada en los datos.

关键词:

• 深度学习
• 人与环境关系
• 位置
• 街道图像
• 城市视觉智能

Palabras clave:

• aprendizaje a fondo
• interacciones humano-ambientales
• imágenes a nivel callejero
• inteligencia visual urbana
• lugar

Key Words:

• deep learning
• human–environment interactions
• place
• street-level imagery
• urban visual intelligence

Images have played a crucial and enduring role in the study of cities, providing valuable insights into the physical environment and influencing urban design theories. This historical importance has been essential in shaping our understanding of urban spaces, their characteristics, and their impact on individuals and communities.

From the nineteenth century onward, images facilitated the aesthetic evaluation of urban spaces, allowing researchers to observe, document, and assess the design, beauty, and workings of urban settings (Freestone 2011). Images also played a key role in shaping and informing urban design theories that emphasized the significance of aesthetic value in urban environments and their impact on human behavior (C. Mulford 1899; Cullen 1961; Nasar 1990; Robinson 1903).

Traditionally, the use of images for studying cities relied on manual collection and careful observation and analysis. Pioneers of the field, such as Lynch (1960), used firsthand observations and photographs from selected parts of a city to create perceptual maps. Whyte (1980) hand-collected and manually analyzed hours of video footage to study human behavior in urban settings. Although these traditional methods provide valuable insights, they are limited in terms of scalability and feasibility over large areas or extended periods (Sampson and Raudenbush 1999; Clifton, Livi Smith, and Rodriguez 2007).

Recent advances in sensing technologies and the growing availability of geo-tagged imagery have begun to address these limitations. Social media platforms and advanced sensing technologies enable real-time image capture, vastly expanding the scope and scale of urban research (Biljecki and Ito 2021). The advent of powerful computer vision and machine learning algorithms has further revolutionized the analysis of these vast image data sets (Ibrahim, Haworth, and Cheng 2020). This transformation in data collection and analysis techniques marks a significant revival and advancement in the field of urban studies.

Despite the wealth of available visual data, there is still a challenge in distinguishing and effectively using the different characteristics of each type of visual data. Moreover, developing standardized methods to extract and interpret this information consistently remains a challenge. Although numerous studies have used visual data to analyze neighborhood appearance, the methodologies for conceptually quantifying the physical environment of urban spaces are not yet fully established. Furthermore, it is unclear how this quantification can be systematically applied to deepen our understanding of the relationship between humans and place and how it might integrate or enhance classical theories and practices.

In this article, we examine theories and recent empirical studies on the use of visual information to understand cities. We introduce a conceptual framework termed urban visual intelligence, which illustrates how the integration of images and artificial intelligence (AI) can be used to (1) observe, (2) interpret, (3) measure the physical environment of cities, and (4) discover how they interact with human behavior and the socioeconomic environment.

Historical Overview of Visual Information in Urban Studies

The tradition of incorporating visual information has been integral to modern urban studies since the early days of city planning. Historically, planners documented and measured physical environmental attributes—such as shape, proportion, rhythm, scale, complexity, color, order, elements, and hierarchy—through photographs or sketches along the streets and sidewalks (Wohlwill 1976). This practice of using the formal attributes of the physical environment to create a pleasing sensory experience for citizens traces back to Camilo Sitte. He advocated for interpreting cities through visual art and architecture, emphasizing the value of irregularity in urban form over rigid symmetry, and proposing aesthetic considerations as primary in city design (Sitte 1889).

This focus on designing the physical environment to influence citizen behavior peaked with planning utopias such as the Garden City and the City Beautiful movements. Proponents of these philosophies believe in the power of beauty, order, and cleanliness in the public realm to shape civic spirit and enhance the quality of life (Talen and Ellis 2002). Designers and theorists, including Frederick Law Olmsted, Sr., Phillip Mackintosh, and F. W. Fitzpatrick, recognized the aesthetic experience of urban spaces as fundamental in urban design and planning. They argued that visually pleasing cities contribute to citizens’ satisfaction, comfort, and pride (L. S. Mulford 1899; Wilson 1964; Nasar 1994; Ahlfeldt and Mastro 2012). Despite a consensus on the need for aesthetic appeal in cities, the debate about what constitutes beauty in urban spaces persists. Is beauty subjective, or can it be measured to design spaces appealing to many?

In the twentieth century, urban planners shifted focus from aesthetics to understanding the subjective experiences evoked by urban environments. This approach sought a deeper understanding of how humans visually perceive and evaluate urban scenes (Arnheim 1965). Researchers attempted to capture how a city’s physical environment can elicit emotions, informing our understanding of attractive and unattractive environments. For example, Nasar (1998) proposed a model explaining aesthetic responses from human interaction with the surrounding environment. Similarly, Rapoport (1990) identified thirty-six characteristics of aesthetically pleasing urban environments. These studies focused on the interplay between people shaping their environment and the physical environment’s impact on them. Their influence on urban studies theories and practices was limited, however, due to challenges in quantifying and representing the physical environment on a large scale.

To overcome this limitation, Lynch (1960) introduced “imageability” as a criterion for quantifying and representing a place’s physical environment. This concept, developed by Lynch, shifted the focus from a purely aesthetic perspective to one emphasizing human cognition, underscoring the importance of meaning in understanding how people navigate and comprehend urban environments (Lynch 1984). In his seminal work, The Image of the City, Lynch proposed a framework consisting of three categories: identity (distinct visual objects), structure (recognizable patterns and relationships between objects), and meaning (emotional values and character of a place; Lynch 1960). Although Lynch initially concentrated on identity and structure, he acknowledged the importance of “meaning.” This aspect received less emphasis in his early analysis, however.

It was Nasar (1998) who further explored the “meaning” aspect of urban environments, enriching Lynch’s framework. Nasar’s work provided a more nuanced understanding of urban spaces, emphasizing the cognitive impacts alongside physical attributes. This evolution in urban studies was also mirrored in Milgram’s (1970) approach to mapping New York City. His method of measuring city recognizability through small-scale experiments, as discussed in Milgram (1970), paralleled this shift toward integrating cognitive experiences with urban environments. Collectively, these advancements marked a significant evolution in the field, offering a more holistic understanding of urban spaces.

In the late twentieth century, there was a philosophical shift toward people-centered and place-based urban design. Designers and planners started prioritizing the performance, vitality, and usage of spaces as alternative measures of urban design quality (Gehl 1971). Scholars typically collected information on urban space utilization using simple techniques like pen and paper and photographic images. For example, William H. Whyte’s “The Street Life Project” used conversations, photographs, and video analysis to observe public space usage (Whyte 1980). Similarly, Gehl (1971) documented elements fostering liveliness and social interaction in urban spaces in his influential book, Life Between Buildings. These studies have profoundly influenced twenty-first-century urban design practices.

These works have been pivotal in shaping urban studies and design, but it is crucial to recognize the dynamic nature of our world. The rapid changes in urban environments need a reexamination of these studies in diverse contexts and times to effectively address contemporary issues. Moreover, there is a growing concern among researchers about the limitations of these studies, particularly regarding small sample sizes and subject selection biases. Such limitations could lead to a skewed understanding of urban spaces, reflecting varying preferences across different populations and over time (Nasar 1998).

Framework of Urban Visual Intelligence

Cities generate an enormous amount of data through hybrid sensing techniques such as crowdsensing and ad hoc sensor deployments. With the advent of urban big data and AI-driven approaches, researchers can now analyze and quantify the physical environment, socioeconomic conditions, and human dynamics in unprecedented ways. These tools enable observation of the interaction between human behavior and the physical environment across spatial and temporal scales.

To leverage these opportunities, we propose the framework of urban visual intelligence. This framework is designed to assess and integrate current data and AI-based approaches, specifically targeting the challenges that traditional methods in urban studies have faced. At the heart of this framework is the strategic employment of AI tools for the analysis of large data sets. This approach not only offers a novel perspective but also seamlessly merges modern methodologies with both long-standing and emerging issues in urban studies, thereby transforming our understanding and analysis of urban environments.

As illustrated in Figure 1, the urban visual intelligence framework proposes four levels for studying urban environments using street-level imagery: vista, scene, place, and city. Each hierarchical level addresses a different challenge: (1) observing the urban physical environment at a human scale; (2) deriving semantic information from street-level imagery; (3) quantifying the physical environment of a place; and (4) understanding the fine-grained interactions between physical and socioeconomic environments.

Figure 1. Framework of urban visual intelligence. The framework outlines key aspects related to the urban physical environment. It focuses on using visual intelligence technologies to observe, measure, and represent physical environments and explore their interactions with socioeconomic dimensions at various levels and scales.

Vista. The first level emphasizes the importance of street-level imagery for observing the physical environment. It focuses on individual images or vistas, capturing the urban environment from a human perspective.

Scene. The second level builds on the vista level by applying advanced techniques, including deep learning and computer vision, to analyze semantic information derived from street-level imagery. This involves extracting details about the physical environment, such as measuring tree coverage or sky visibility in a given scene.

Place. The third level focuses on localized areas within a city, using collections of images to create comprehensive, quantitative representations of a place. This is crucial for characterizing the structure and perception of a specific area.

City. The final level extends the study to a broader scale, integrating insights from the preceding levels. It focuses on a comprehensive understanding of the interactions between the physical environment of places and broader human dynamics and socioeconomic characteristics.

The framework starts with the focused observation of individual images (vistas), progresses to understanding scenes, delves into a detailed analysis of specific areas or places within a city, and finally expands to encompass the city as a whole. This structure ensures a comprehensive understanding of urban environments at various scales, from the most detailed to the most expansive.

Observing the Urban Physical Environment at a Human Scale

Street-level imagery, as highlighted in the first level of the urban visual intelligence framework, is increasingly becoming an important data source for studying cities. With the rapid adoption of mobile Internet technologies and the growing use of Web mapping services and crowdsourcing platforms, geo-tagged images are being produced at an unprecedented rate, covering every corner of cities (Goodchild 2007). This data source, with its extensive spatial coverage, has become invaluable for observing large-scale urban environments (Ibrahim, Haworth, and Cheng 2020; Biljecki and Ito 2021; Duarte and Ratti 2021; Liang, Zhao, and Biljecki 2023). In 2023, the Camera and Imaging Products Association reported that 1.72 trillion photos were taken using digital cameras on mobile phones.¹

Figure 2 illustrates three primary sources of street-level imagery for city analysis. The first category includes map service images, such as those from Google Street View, which offer stable update frequency, broad coverage (spanning more than 200 countries), and uniform standards, enabling comparative analysis across locations (Anguelov et al. 2010; Goel et al. 2018). The second category consists of crowdsourced photos from platforms like Flickr and Mapillary, a form of volunteered geographic information (VGI; Goodchild 2007). As the volume of crowdsourced photos increases, it is anticipated to overtake map services as the primary source of street-level imagery due to its denser spatiotemporal coverage. The third category features custom image collections captured by individuals or researchers for specific studies. These custom collections, which can include a time series of images, are valuable for tracking changes in the physical environment and individual activities, complementing mapping services and crowdsourced imagery.

Figure 2. Sources of street-level imagery.

Compared to traditional urban study data sources, street-level imagery offers several advantages, including easy accessibility, extensive spatiotemporal coverage, and objective, standardized views from embedded vantage points (Rzotkiewicz et al. 2018; Ibrahim, Haworth, and Cheng 2020; Biljecki and Ito 2021). Additionally, street view imagery provides a distinct perspective compared to satellite imagery. Whereas satellite imagery presents an aerial view, street view imagery captures a human-level perspective, more closely aligned with people’s experiences and perceptions of cities. This perspective is particularly useful for examining visual cues at the human scale, offering valuable insights for urban planning and design. The recent standardization of images across cities and the advancement of visual analytic methodologies have further enhanced the utility of street-level imagery.

Street-level imagery has become one of the most valuable data sources for studying physical environments (Cinnamon and Jahiu 2021; N. He and Li 2021). Its applications extend across various research fields, including physical environment auditing (L. Zhang et al. 2020; S. Chen and Biljecki 2023; T. Zhao et al. 2023), public health (Nguyen et al. 2018; H. He et al. 2020; Keralis et al. 2020), urban mobility and transportation (Lu et al. 2019; Hong, McArthur, and Raturi 2020; Mooney et al. 2020), energy estimation (Z. Liu et al. 2019; Sun, Han, et al. 2022; K. Zhang et al. 2022), and real estate (Law, Paige, and Russell 2019; E. B. Johnson, Tidwell, and Villupuram 2020; Kang, Zhang, Peng, et al. 2020), among others.

Deriving Semantic Information from Street-Level Imagery

How can semantic information be derived from street-level imagery in a scene? The second level of the urban visual intelligence framework focuses on using deep learning and computer vision techniques to interpret semantic information from street-level imagery.

Deep Learning and Computer Vision.Traditionally, field surveys have relied on manual visual data analysis, a time-consuming and labor-intensive process that limits the scope of research. Although image processing techniques have been developed for mass analysis, they have predominantly focused on basic features such as color and texture, not complex information like objects and styles. Understanding high-level information is essential in linking a city’s appearance to human behavior.

Recent advancements in deep learning and computer vision have enabled the extraction of high-level features like semantic objects and scene understanding from images, overcoming traditional image processing limitations. Deep neural networks have empowered models with learning capabilities, interpretation, and decision-making akin to the human brain. The influence of deep learning spans various domains, including speech recognition (e.g., Wav2Vec; Baevski et al. 2020), natural language understanding (e.g., generative pre-trained transformer [GPT] models; Brown et al. 2020), strategic game playing (e.g., AlphaGo; Silver et al. 2016), and sophisticated image recognition and segmentation tasks (Radford et al. 2021).

A significant area of improvement is image segmentation, where models now delineate and classify different parts of an image with remarkable accuracy, crucial for autonomous driving (Kirillov et al. 2023). Additionally, large language models (LLMs) like ChatGPT have broadened natural language processing capabilities (Radford et al. 2021), enabling more intuitive, human-like AI interactions. These LLMs can generate coherent text, perform complex language tasks, and engage in meaningful conversations, showcasing deep learning’s potential in understanding both visual and textual content.

Geospatial Artificial Intelligence. Leveraging the progress in AI, geospatial artificial intelligence (GeoAI) technologies enrich classical methods by integrating spatial effects, focusing on factors such as spatial dependency, distance decay, and scale effects (Gao, Hu, and Li 2024; Zhu et al. 2020). Such integration shifts the focus from viewing samples in isolation to understanding their complex interrelations within a spatial context (Zhu et al. 2020). GeoAI integrates these spatial dimensions into analytical models, enhancing the capacity to interpret and predict geographic phenomena (Mai et al. 2023). By accounting for the spatial arrangement of objects and their interactions, it provides a more holistic understanding of urban environments. This spatial awareness is crucial when studying cities, where the arrangement of buildings, roads, and public spaces significantly affects human behavior and environmental dynamics.

Large-Scale Image Data Sets. Large-scale image data sets play a crucial role in deep learning, offering vast, labeled images to help models comprehend complex input–label relationships. A prime data set should cover a wide range of categories with extensive variety and depth, providing numerous images per category to capture its diversity (B. Zhou, Lapedriza, et al. 2017).

Creating a training set involves manual annotations, data association, and data generation. Manual annotation, streamlined by online platforms like LabelMe and Amazon Mechanical Turk, requires labeling images or marking object outlines (Russell et al. 2008; Sorokin and Forsyth 2008). Emerging AI-assisted labeling methods are simplifying this process (L. Chen et al. 2020). Data association connects images with labels based on patterns like occurrence or location. For example, house photos from real estate sites can be correlated with prices. In situations where direct label matching is not feasible, an alternative approach involves extracting information directly from the images. This method is particularly useful in tasks such as analyzing street scenes to identify font types on signage (Ma et al. 2019).

Key urban image data sets in urban studies and geospatial analytics include Places2 (B. Zhou, Lapedriza, et al. 2017) and ADE20K (B. Zhou, Zhao, et al. 2017). The Places2 data set, with approximately 10 million labeled images, represents various place types like residential neighborhoods, highways, and parks, aiding deep learning models in classifying scene types from street view images. The ADE20K data set, containing more than 20,000 labeled images, covers a diverse range of visual object categories such as plants, sky, vehicles, and buildings. Researchers are also compiling additional data sources to link images with ground-truth data for specific applications, like describing scene attributes (G. Patterson and Hays 2012), classifying architectural styles (Xu et al. 2014; Sun, Han, et al. 2022), tracking neighborhood change (Naik et al. 2017), and detecting informal settlements (Ibrahim, Haworth, and Cheng 2021).

In the era of large-scale models, the sheer volume of data required for training presents a challenge for manual image annotation. An emerging solution for large-scale visual tasks is leveraging LLMs to acquire knowledge, followed by applying multimodal transfer to adapt this knowledge to the visual domain, thus facilitating the execution of visual tasks.

Computer Vision Tasks for Urban Image Inference. The availability of large-scale image data sets has enabled the development of complex computer vision tasks with greater depth. In urban contexts, these models are employed for scene classification, object detection, and semantic segmentation, each differentiated by their final model layers, as illustrated in Figure 3. Scene classification (Figure 3A) assigns a label to the overall image, using architectures like ResNet (K. He et al. 2016), GoogLeNet (Szegedy et al. 2015), DenseNet (G. Huang et al. 2017), and the Vision-Transformer series (Z. Liu et al. 2021). Object detection (Figure 3B) identifies and classifies objects, marking their positions with bounding boxes, using models such as Faster R-CNN (Ren et al. 2015), SSD (W. Liu et al. 2016), and the YOLO series (Ge et al. 2021). Semantic segmentation (Figure 3C) divides the image into segments, creating pixel-level masks for each object, employing PSPNet (H. Zhao et al. 2017), Mask RCNN (K. He et al. 2017), HRNet (J. Wang et al. 2020), and the Deeplab series (L.-C. Chen et al. 2018).

Figure 3. Three typical computer vision tasks in urban applications: (A) scene classification, (B) object detection, and (C) semantic segmentation.

Scene Analysis and Understanding. Extracting elements from a scene is a common method used in analyzing physical environments. This analysis can be achieved through object detection or semantic segmentation. Object detection models provide details about detected objects along with their bounding boxes, facilitating the counting of different objects within an image. Conversely, semantic segmentation models predict the categories of objects for each pixel, allowing for the calculation of object proportions within a scene. Both types of models offer quantitative methods for scene measurement.

A notable example of semantic segmentation is the Treepedia project,² which uses Google Street View to map tree canopies. This method offers a scalable alternative to manual surveys across thirty global cities (Seiferling et al. 2017). Other studies have integrated green canopy measurements with satellite imagery-derived green indexes to analyze how perceptions of the physical environment are influenced by different camera angles (R. Wang, Helbich, et al. 2019; Laumer et al. 2020; Biljecki et al. 2023).

Deep learning models have also been used to classify various street elements, including the sky, road, buildings, vegetation, vehicles, and pedestrians (F. Zhang, Zhang, et al. 2018; B. Zhou et al. 2019). High-quality imagery can be used to classify intricate aspects such as street signs, abandoned houses, sidewalk cracks, broken windows, and deteriorating walls (Less et al. 2015; Zou and Wang 2021). For example, Miranda et al. (2021) used Google Street View to assess urban features attractive to pedestrians in Boston, quantifying elements like urban furniture, sidewalk dimensions, facade complexity, and visual enclosure.³ These metrics aid urbanists in identifying pedestrian-friendly environments (Zünd and Bettencourt 2021).

Computer vision also extends to image classification. Using 10 million social media photos and semantic labels, B. Zhou, Lapedriza, et al. (2017) developed a model to classify place types (e.g., neighborhoods or bus stations) and attributes (e.g., man-made or sunny), setting a standard for understanding place functionality (Xiao, Fang, and Lin 2020; Ye et al. 2020). Similarly, Hu et al. (2020) demonstrated that deep learning could classify street canyons from Google Street View images, eliminating the need for manual measurement and streamlining environmental analysis.

Scene inference models are capable of extracting nonvisible details like crime rates (Khosla et al. 2014), property values (Law, Paige, and Russell 2019), and changes in human activity (F. Zhang, Wu, et al. 2019). For instance, a model by Khosla et al. (2014) predicts unseen aspects of a scene, such as proximity to grocery stores or hospitals. The models are trained using an end-to-end learning process, where a single model directly maps inputs to outputs without intermediate steps or feature engineering. This process allows the model to autonomously learn the complex relationships between the initial input image and the final output labels, such as crime rates and housing prices, based on a nonlinear correlation between built environments and socioeconomic factors.

Quantifying the Physical Environment of a Place

The term place encompasses a broad spectrum of elements, ranging from individual cognition to perceptual dimensions. In this study, we focus on the aspects of a place’s physical environment. We define the physical environment of a place as the aggregate of all visible natural and built elements within an urban landscape. This includes natural features such as trees, water bodies, and parks, as well as built structures like buildings, roads, and bridges. Dynamic elements such as human activity and vehicular movement are also integral to this definition, contributing to the character and functionality of urban spaces.

Quantifying the physical environment of a place poses challenges due to its complexity and inherent subjectivity. Although AI has been instrumental in objectively evaluating a place’s visual aspects, there remains a gap between algorithmic measurements and the human-centric essence of places (Tuan 1979). Recent research has, however, made strides in modeling various dimensions of a place, including human activities, cognitive regions, and semantics (Gao et al. 2017; Purves, Winter, and Kuhn 2019). This has resulted in the development of formal computational representations of a place, essential for interdisciplinary research (Janowicz et al. 2022).

In this context, our focus is on the quantitative analysis and representation of a place’s physical environment. We explore this through three perspectives: place identity, place structure, and place perception. These dimensions play are pivotal in determining a place’s “imageability,” a concept crucial in geography and urban planning (Lynch 1960; Morison 2002; M. E. Patterson and Williams 2005). Our aim is to shed new light on classic theories, such as those of Lynch, by incorporating recent AI and big-data-driven urban studies.

Place Identity. We use place identity to denote a place’s unique or common attributes, distinct from the psychological construct of place identity (Lewicka 2008). This focuses on aspects of “imageability” in urban spaces as proposed by Lynch (1960).

Assessing the identity of a place is important to understand how people interact with their surroundings. Visual identity, representing a place’s unique or common attributes, plays a key role in its recognition. Traditionally, measuring place identity and similarity relied on subjective, qualitative insights, which lacked uniformity and scalability. AI, however, has revolutionized this process through representation learning (RL), a data-driven, quantitative approach. RL allows machines to autonomously identify essential features of a place, often encoded in numerical vectors.

For example, in RL, visual identity and similarity can be assessed using a classification task handled by a computer vision model. The model learns to recognize unique features of street-level images and classify the origin of the place. The misclassification rate essentially indicates similarity between places, with high misclassification suggesting similarity and high accuracy indicating distinctiveness. Confidence scores produced by the model for each image help rank the most representative scenes of a place. Higher scores reflect greater certainty and a more robust representation of place identity.

Several studies have used this approach to measure place identity and similarity. For instance, Doersch et al. (2012) used an automated approach to identify the unique architectural features of cities. F. Zhang, Zhou, et al. (2019) trained a deep learning model to recognize places among eighteen cities globally, measuring the visual similarity and distinctiveness of these cities and identifying unique visual cues. For indoor spaces, F. Zhang et al. (2016) analyzed corridors and spaces in large interconnected buildings on the MIT campus to understand visual elements of indoor design and human cognition.

This method has also proven useful in routing, as shown by Z. Wang et al. (2019), who evaluated the legibility of two train stations in Paris using a computer vision model. Furthermore, L. Liu et al. (2016) used 2 million geotagged photos of twenty-six cities to reproduce the Image of the City, demonstrating how digital techniques can enhance our understanding of places across cities. Other studies have extended this digital approach to many other cities in recent years (Salesses, Schechtner, and Hidalgo 2013; Filomena, Verstegen, and Manley 2019; Y. Huang et al. 2023).

Place Structure. Place structure, as conceptualized by Lynch (1960) in The Image of the City, refers to the physical layout and organization of urban spaces, which is crucial in shaping how people perceive and remember their environment. This concept includes identifying and understanding key elements such as paths, edges, districts, nodes, and landmarks, forming a coherent and memorable image of the city (Lynch 1960).

Recent advancements, as highlighted in works by G. Patterson and Hays (2012) and F. Zhang, Zhang, et al. (2018), offer complementary perspectives on organizing scene elements into hierarchical categories. For instance, F. Zhang, Zhang, et al. (2018) categorized common urban objects into a hierarchical tree based on conceptual relationships, grouping “tree,” “flower,” and “grass” under “vegetation.” This, along with categories like “waterbody” and “sky,” forms a “natural” supercategory. This structure allows for qualitative and quantitative analysis of a place’s “structure” through element presence and abundance measured using a deep learning model.

Place Perception. Understanding how humans perceive their surrounding environment is crucial for assessing urban design quality. This topic, central to disciplines like human geography, urban planning, and environmental psychology, has been enriched by the use of street-level imagery and deep learning techniques (Lynch 1960; Tuan 1977; Nasar and Jones 1997). These technologies have opened new avenues for quantifying human perception, particularly through crowdsourced information, which allows for the analysis of preferences and perceptions on a massive scale. A prominent example is the Place Pulse platform, which collects online ratings to assess human perception (Salesses, Schechtner, and Hidalgo 2013). This platform has collected more than 1 million ratings on various aspects of Google Street Views from 80,000 volunteers across fifty-six cities in twenty-eight countries, surpassing the scope of traditional survey methods.

Platforms like Place Pulse have facilitated research on how humans visually evaluate their environments (Ordonez and Berg 2014; Dubey et al. 2016). Such studies revisit and extend classic urban theories about the relationship between the physical environment and human perceptions, which were previously constrained by the limited scale of sample sizes and geography. For example, F. Zhang, Zhou, et al. (2018) explored the impact of street features on human perceptions in Beijing and Shanghai by analyzing 1 million street views through image segmentation. Their work evaluated the effects of physical disorder, such as litter, graffiti, and building conditions, on people’s feelings, providing a method to measure the “sense of place” in expansive urban areas. Similarly, Saiz, Salazar, and Bernard (2018) harnessed the widespread sharing of photographs online to gauge how people value the aesthetic aspects of their environment. They demonstrated that street-level imagery could be a scalable tool for measuring subjective attractiveness, enhancing our comprehension of how individuals perceive urban spaces.

Moreover, researchers have used human perceptions derived from deep learning to assess the social and economic dynamics of cities. Studies have leveraged street-level imagery from Google Street View to measure changes in the physical appearance of neighborhoods. Naik et al. (2017) connected changes in the physical appearance of five U.S. cities with economic and demographic data to document the underlying factors that predict neighborhood improvement. Additionally, F. Zhang et al. (2020) characterized places by their physical appearance and popularity, uncovering many inconspicuous yet frequented restaurants in Beijing. These establishments, often situated in less visually appealing alleyways of old neighborhoods, are popular social areas, highlighting the complex relationship between appearance and social activity in urban settings.

Understanding the Fine-Grained Interactions Between Physical and Socioeconomic Environments

Exploring the interplay between the physical environment and the socioeconomic context is crucial for a variety of disciplines. This fourth level of the urban visual intelligence framework focuses on studying these interactions in detail. It underscores fields such as public health, transportation, and urban economics. These are not the only domains explored using visual information, but they represent significant areas where the framework’s applications are particularly innovative.

Public Health. Environmental health studies have traditionally relied on field surveys and questionnaires to gather data about the physical environment (Takano, Nakamura, and Watanabe 2002; Lawlor et al. 2003; Gullón et al. 2015). Although useful, these methods often face limitations in cross-place comparisons and capturing a human perspective. Street-level imagery and visual intelligence offer a complementary view, enhancing these established methods.

The physical environment significantly affects health outcomes, encompassing both physical aspects such as obesity and psychological factors like mental health (Mehrabian and Russell 1974; Ulrich 1984; Lee et al. 2012). Street-level imagery has been instrumental in quantifying visual elements associated with health, such as green spaces, sidewalks, and urban infrastructure, alongside food advertisements and indicators of physical disorder (J. Chen et al. 2023). For instance, the presence of greenery has been linked to active behaviors like walking and cycling, its effects on children’s body weight, mental well-being, soundscapes, and perceived safety (James et al. 2015; X. Li, Zhang, and Li 2015; Lu, Sarkar, and Xiao 2018; Svoray et al. 2018; Kang et al. 2019; Lu et al. 2019; Kruse et al. 2021; J. Huang et al. 2023; T. Zhao et al. 2023). Comparisons between greenery metrics from street view and remote sensing imagery have highlighted the unique advantage of street-level perspectives in capturing the eye-level greenery experience (Villeneuve et al. 2018; Helbich et al. 2019; Larkin and Hystad 2019; Lu et al. 2019; Kang, Zhang, Gao, et al. 2020).

Empirical studies have also uncovered correlations between visual aspects derived from street imagery and health outcomes. Streets with visual enclosure have been associated with higher quality environments, and the presence of pedestrian infrastructure like sidewalks and crosswalks correlates with increased walkability and enhanced mental health (Vargo, Stone, and Glanz 2012; Yin and Wang 2016; Nguyen et al. 2018; R. Wang, Lu, et al. 2019). Features such as food and beverage advertisements have been used to identify environments conducive to obesity (Feuillet et al. 2016; Roda et al. 2016; Egli et al. 2019), and other visual cues have served as proxies for various health-related metrics.

Transportation and Mobility. Street-level imagery serves as a valuable asset for comprehending the transportation context and its implications, informed by physical environmental features. These features facilitate virtual audits, aiding the identification of traffic hot spots and potential congestion areas (Qin et al. 2020; Tanprasert et al. 2020). This subsection concentrates on how attributes of roads and their surroundings extracted from street imagery inform transportation behavior.

Research using image-derived features has investigated transportation patterns. Characteristics such as traffic lights, speed bump density, and pedestrian crossings have been related to traffic volumes and route preferences (Verhoeven et al. 2018; den Braver et al. 2020). Additionally, attributes like the presence and condition of bicycle lanes, sidewalks, and road surfaces have been analyzed in relation to pedestrian safety incidents (N. S. Johnson and Gabler 2015; Isola et al. 2019; Kwon and Cho 2020; Mooney et al. 2020).

Deep learning models offer a nuanced approach to studying the connections between the physical environment and urban mobility. The visual appearance captured in images can reveal insights into function and land use (Y. Liu et al. 2012; Yuan, Zheng, and Xie 2012; Fan, Zhang, and Loo 2022). These models, through end-to-end training, can discern nonlinear associations between environmental aspects and movement patterns. For instance, F. Zhang, Wu, et al. (2019) inferred hourly human activity levels from street view images, effectively predicting urban mobility fluctuations. Other studies have predicted spatial trends in cycling and walking through points of interest and street imagery (L. Chen et al. 2020; Hankey et al. 2021).

Computer vision and deep learning are promising avenues for researchers aiming to design physical environments that influence urban navigation. Mirowski et al. (2018), for example, applied deep reinforcement learning to enable agents to navigate cities solely based on street view images. This research demonstrates the potential of visual cues in the physical environment to aid individuals in navigating the city.

Additionally, Salazar-Miranda et al. (2023) presented a framework for real-time analysis of street activities using computer vision to classify pedestrian behaviors and transportation modes. This study leverages camera-equipped buses in Paris to capture diverse street activities, assisting in city planning and testing safety interventions.

Urban Economics. The physical appearance of a city offers insights into its socioeconomic fabric. Advances in street-level imagery and deep learning have led to a surge in studies characterizing the physical environment’s nuanced relationship with social and economic outcomes like income, housing prices, and crime (Ibrahim, Haworth, and Cheng 2020; Fan et al. 2023).

Crime is a prominent socioeconomic dimension that has been extensively studied using street-level imagery and deep learning techniques (H. Zhou et al. 2021). Research has sought to align community safety perceptions with actual crime statistics. F. Zhang et al. (2021) introduced a “perception bias” metric to measure the discrepancy between perceived safety from Google Street View images and actual violent crime rates while also considering socioeconomic influences. Kang et al. (2023) expanded this measure by surveying residents’ sense of safety in their neighborhoods.

The visual quality of neighborhoods has proven to be an effective predictor of real estate values and housing appreciation (Kang et al. 2021; Yang et al. 2021; Qiu et al. 2022). Elements captured in images, such as vehicle types, can predict demographics and political leanings (Gebru et al. 2017), and business amenity typefaces can be proxies for neighborhood income (Ma et al. 2019). Time-sequenced street view images allowed Naik et al. (2017) to track urban changes, linking infrastructure improvements to educational and density metrics, with better looking neighborhoods showing more substantial progress.

Computer vision models trained on street-level imagery can predict socioeconomic characteristics, capturing the intricate interplay between the physical environment and socioeconomic factors. Such models have been applied to study job–housing patterns, social and environmental inequities, and urban deprivation (Suel et al. 2019; Suel et al. 2021; Yao et al. 2021). A comprehensive study by Fan et al. (2023) used a computer vision model on 27 million street view images from U.S. counties, proving that visual features can precisely estimate neighborhood socioeconomic profiles. These estimations, which provide more explanatory power than traditional demographics or point of interest data, have shown consistency across different regions, confirming their reliability and wide applicability in diverse urban settings.

Discussion

Urban Visual Intelligence: Navigating the Edges of Sight and Insight

Although our proposed framework aims to address several fundamental issues, the adoption of the new data sources and methods introduces its own set of emerging challenges. Visual data alone might not fully encapsulate the complexity of urban life (Batty 2021). The following subsections discuss the challenges in this process, from the detailed analysis within single locations to the application of findings across diverse urban contexts, and the need to consider subjective interpretations and cultural aspects in the analyses.

Challenges in Within-Place and Between-Place Inference. In urban analysis, within-place inference is the process of extracting and interpreting information from images to understand various aspects of a specific place (Batty 2021). Traditional modeling approaches often struggle to capture the complex, nonlinear relationships inherent in a place. In contrast, deep learning and computer vision models excel in these environments, offering nonlinear modeling capabilities.

Conversely, between-place inference concerns the applicability of a model trained in one place and applied to another. This type of inference questions the adaptability and transferability of trained models across different urban settings, presenting a multifaceted challenge involving the heterogeneity of place, cross-domain generalizability, and spatial variability. From the perspective of human geography, the heterogeneous nature of place and varying cultural perceptions can significantly affect model performance. For example, a model trained on visual appearance from Asian countries might underperform in Western settings, owing to differences in urban design and human activity patterns (F. Zhang, Wu, et al. 2019). In GIScience, the challenge of between-place inference is compounded by spatial heterogeneity and nonstationarity. Such spatial variability often limits the ability of models to make consistent predictions across different locations (Goodchild et al. 2020; Goodchild and Li 2021; Kedron et al. 2021; W. Li, Hsu, and Hu 2021). Furthermore, in computer science, ensuring that a model’s performance is consistent across different domains involves addressing the domain shift challenge. Domain shift refers to the discrepancies in data distribution when a model, trained in one context, is applied to data from another context (Quiñonero-Candela et al. 2009; Neyshabur et al. 2017; M. Wang and Deng 2018). Such discrepancies can lead to reduced model accuracy because the patterns recognized in the training data might not be present or might manifest differently in the new data set.

These factors underline the importance of developing models that can adapt to new, varied data distributions to maintain their efficacy in diverse applications.

Challenges in Interpreting the Subjective and Culturally Significant Aspects of Place. The essence of a place transcends its physical attributes to include cultural significance and subjective interpretation, which together cultivate a unique cultural landscape. This is manifested in the varied experiences one might have when comparing visits to the Eiffel Tower in Paris with its replicas, or in the differing cultural narratives attached to architecturally similar rows of brick houses in Edinburgh and Boston.

Such uniqueness of place aligns with the principles of nomothetic geography and idiographic science (Warntz 1989), emphasizing the singularity of each location. This perspective challenges the generalizing nature of deep learning models, which typically infer broad rules from specific instances. Consequently, AI models often encounter difficulties in capturing the full scope of complex cultural landscapes through visual data alone.

Moreover, perceptions of place are deeply personal, shaped by an individual’s experiences, life stages, and preferences, contributing to a distinctive “sense of place” (Tuan 1977). The Temple Mount in Jerusalem, for instance, embodies various religious significances for different groups, illustrating subjective interpretation. Additionally, the practice of spatial ethnography, such as the examination of “time-sharing” within busy sidewalk environments (Kim 2015), illustrates how a place’s meaning can shift with time. A deep learning model might count the number of people on a sidewalk, but it cannot grasp the subtle variations in how these spaces are perceived and used by different demographic groups throughout the day or year.

Addressing these interpretative challenges requires innovative methods that integrate personal experiences, cultural context, and subjective perception. LLMs offer a promising solution. Trained on extensive textual data, LLMs acquire a broad understanding of cultural nuances and can tailor outputs based on individual input. This ability to apply knowledge from linguistic contexts to visual data interpretation promises to deepen AI models’ grasp of complex cultural landscapes, leading to richer and more nuanced decision-making.

Challenges in Analyzing Uncertainty in Street-Level Imagery. Street-level imagery is subject to various uncertainties that can affect the accuracy of the results (Biljecki et al. 2023). These uncertainties range from the modifiable areal unit problem (MAUP) and ecological fallacy to measurement issues and temporal changes.

The MAUP arises during the aggregation of point-based measurements into larger spatial units (Fotheringham and Wong 1991). In the context of street-level imagery, this issue is prevalent due to the nonuniform distribution of images, often constrained by road networks and the geographic spread of social media contributions. As a result, the aggregation process might obscure the true diversity of an area by presenting a misleadingly uniform characteristic along a street.

Another known challenge is the ecological fallacy, which occurs when aggregated data conclusions are applied to individual elements. For example, deducing that every segment of a street is equally aesthetically pleasing based on its overall high beauty rating ignores the potential for significant variations among individual segments.

Whereas ecological fallacy highlights errors in overgeneralizing group data to individuals, measurement uncertainty presents a different challenge that affects the scale at which urban features are perceived. This type of uncertainty arises from the variability in camera angles and distances in street-level imagery, which can alter the apparent proportions of visual elements, leading to potential misinterpretations in subsequent computer vision analyses. For example, images that capture only the lower sections of tall buildings fail to convey their full architectural context, resulting in a partial and potentially misleading representation.

Temporal changes introduce yet another layer of complexity. Just as measurement uncertainty can skew the perception of static images, the dynamic nature of urban environments means that features such as vegetation and pedestrian activity can vary widely over time. Most existing studies rely on data sets like Google Street View, which are updated infrequently, thus limiting the ability to analyze changes that occur over shorter periods. The advent of more granular and frequently updated data sources, like LiDAR point clouds and platforms such as Mapillary, is beginning to provide researchers with the tools needed for a more detailed and current exploration of urban spaces (Miranda et al. 2022).

Promising Avenues of Inquiry and Future Work

The integration of street-level imagery with deep learning presents exciting opportunities for urban studies. This section outlines potential research directions.

Discovering Hidden Visual Cues. Street-level imagery is rich in visual cues that offer insights into urban life. It captures elements such as written language in street names, business signs, and advertisements, which help identify points of interest and indicate linguistic or ethnic compositions. These images can also reveal subtle indicators of social disorder and psychosocial stress, such as the presence of graffiti, litter, and broken windows, which have been linked to neighborhood social disorder (Sampson and Raudenbush 2004).

Integrating Visual with Multimodal Information. Recent advances in LLMs have significantly improved AI’s capacity for understanding and organizing knowledge. Initially focused on text, these models are expanding to include visual and other modalities. For example, integrating visual information with text, sound, location, and other multimodal data can enhance AI’s understanding of urban environments. This integration allows AI systems to draw deeper connections between visual aspects and the broader cultural, social, and geographical contexts of cities, leading to more informed urban analysis and decision-making.

AI-Driven Generation of Urban Scenes. AI techniques in scene generation, such as variational autoencoders (VAEs), generative adversarial networks (GANs), and diffusion models, offer exciting possibilities for creating new urban environments. These models can generate realistic urban scenes based on both objective features and subjective perceptions, such as attractiveness and safety (Bau et al. 2020). Practical applications include urban design, scenario planning, and conceptualizing future cities (B. Zhao et al. 2021; Wu and Biljecki 2022). A notable example is a platform that allows users to create custom street scenes, integrating elements like land use and building density (Noyman and Larson 2020).

Development of Interpretable and Reliable AI Models. Developing interpretable and reliable AI models is crucial for urban studies. Such models can enhance scientific research by providing better predictions of human activities and socioeconomic profiles. For practitioners and policymakers, these models can identify confounding factors in causal analysis, aiding in the formulation of more informed policies. The interpretability of these models is essential for understanding and applying their findings in practical urban planning contexts.

Understanding the Hidden Laws of Cities. Investigating the fine-grained characteristics of urban environments through street-level imagery could shed light on spatial laws governing cities. Research could explore the existence of a fundamental spatial unit in urban spaces, examining how the computational representation of physical features varies with spatial scales. Understanding these patterns might reveal a consistent spatial scale that effectively represents the physical aspects of urban environments.

Conclusion

This article examined the role of AI and visual data in urban studies, highlighting their impact on enhancing the analysis of urban environments. The integration of these technologies facilitates a more detailed understanding of urban dynamics, both physical and socioeconomic, thus contributing to the evolution of urban theory and practice. The urban visual intelligence framework introduced serves as a conceptual model for guiding research in this domain. It advocates for the analysis of urban spaces through a lens that incorporates technological advancements in AI and visual data analysis, also drawing on established urban theories.

The overview of current work underscores the potential of AI to augment traditional urban analysis methods, enabling a deeper and more granular examination of urban phenomena. It also identifies challenges, though, particularly in ensuring that the interpretation of data adequately reflects the cultural and subjective nuances of urban life.

Looking ahead, the successful application of urban visual intelligence will depend on its ability to harmonize detailed, data-driven analysis with an understanding of the qualitative, human aspects of urban spaces. This approach promises to broaden the scope and depth of urban studies, offering insights that are both technically robust and contextually rich.

Disclosure Statement

No potential conflict of interest was reported by the authors.

Notes

Additional information

Funding

This work was supported by the National Natural Science Foundation of China under Grant 42371468.

Notes on contributors

Fan Zhang

FAN ZHANG is an Assistant Professor at the Institute of Remote Sensing and Geographical Information System, School of Earth and Space Sciences, Peking University, Beijing 100871, China. E-mail: [email protected]. His research interests include geospatial AI and data-driven urban studies.

Arianna Salazar-Miranda

ARIANNA SALAZAR-MIRANDA is a researcher at the Senseable City Lab at MIT and an incoming Assistant Professor of Urban Planning and Data Science at the School of the Environment at Yale University, New Haven, CT 06511. E-mail: [email protected]. Her research focuses on the relationship between urban planning, the built environment, human behavior, and sustainability using computational methods and new sources of data.

Fábio Duarte

FÁBIO DUARTE is a Principal Research Scientist at the Massachusetts Institute of Technology Senseable City Lab, Cambridge, MA 02139. E-mail: [email protected]. His research work involves urban technologies and the social construction of technologies.

Lawrence Vale

LAWRENCE VALE is Ford Professor of Urban Design and Planning and Associate Dean of the School of Architecture and Planning at the Massachusetts Institute of Technology, Cambridge, MA 02139. E-mail: [email protected]. His research interests include urban and regional planning, applied economics, and public administration and policy.

Gary Hack

GARY HACK is a Professor Emeritus of Urban Design at the Massachusetts Institute of Technology, where he headed the Department of Urban Studies and Planning, Professor Emeritus and former Dean of the School of Design at the University of Pennsylvania. E-mail: [email protected]. He has studied, taught, and practiced site planning for more than forty years in the United States, Canada, and other countries.

Min Chen

MIN CHEN is a Professor and Vice Director of the Key Laboratory of Virtual Geographic Environment, Ministry of Education of PRC, School of Geography, Nanjing Normal University, Nanjing, 210023, PR China. E-mail: [email protected]. His research interests include geographic modeling and simulation, and virtual geographic environments.

Yu Liu

YU LIU is a Professor of GIScience at the Institute of Remote Sensing and Geographical Information Systems, School of Earth and Space Sciences, Peking University, Beijing 100871, China. E-mail: [email protected]. His research concentration is in GIScience and big geodata.

Michael Batty

MICHAEL BATTY is Bartlett Professor of Planning and Chair of the Centre for Advanced Spatial Analysis at University College London, Gower Street, London WC1E 6BT, UK. E-mail: [email protected]. His research work involves the development of computer models of cities and regions, big data, and smart cities, and simulating long-term structural change and dynamics in cities as well as their visualization.

Carlo Ratti

CARLO RATTI is a Professor and Director of the MIT Senseable City Lab at the Massachusetts Institute of Technology, Cambridge, MA 02139. E-mail: [email protected]. He is also a founding partner of the international design office Carlo Ratti Associati.

Notes

1 See https://www.cipa.jp/e/stats/dc.html.

2 See http://senseable.mit.edu/treepedia.

3 See https://senseable.mit.edu/desirable-streets/.