Review of counterfactual land change modeling for causal inference in land system science

ABSTRACT

Land change models are important tools for land systems analysis, but their potential for causal inference using a counterfactual approach remains underdeveloped. This paper reviews the state of counterfactual land change modeling with the intent of causal inference. All of the reviewed studies promoted the value of creating ‘counterfactual worlds’ via simulation modeling to untangle complex causation in land systems in order to assess the effects of specific interventions and/or historical events. Several models used counterfactual analysis to challenge prevailing assumptions motivating past policy interventions, while others isolated the spatial heterogeneity of policy effects. The review also highlights methodological limitations and proposes best practices specific to counterfactual land change modeling for causal inference, such as ensemble approaches and multiple calibration-validation iterations. Counterfactual modeling is still underdeveloped in land system science, but it holds promise for advancing causal inference for some of the most challenging to study land change phenomenon.

KEYWORDS:

• Literature review
• land-use change
• model development cycle
• causal inference

Introduction

Land-use change (LUC) lies at the intersection of a number of global change challenges, such as mitigating and adapting to climate change through carbon sequestration, food-energy-water-land nexus security concerns, and a means for or limitation to sustainable livelihoods (Verburg et al., 2013, 2015; Ehrensperger et al., 2019; Meyfroidt et al., 2022; Turner et al., 2020). LUC is the emergent result of myriad interactions and feedbacks among diverse actors, technologies, institutions, cultural practices, and the associated demands and motivations for using land, which collectively constitute land systems (Meyfroidt, 2016). Land systems evolve through these interactions which act across diverse spatial and temporal scales and can be highly contingent (Meyfroidt et al., 2022; Radosavljevic et al., 2021; Schlüter et al., 2022). A core tenet of land system science (LSS) is the understanding of the causes of LUC, but the multi-factor and multi-scale nature of LUC often entails complex causal pathways that challenge the identification and analysis of causal effects and mechanisms (Magliocca et al., 2018; Meyfroidt, 2016; Turner et al., 2020). Because of this complexity, land change simulation models are important tools for enabling holistic analysis of land systems (Dearing et al., 2010; Verburg et al., 2019) and advancing land system theory (Turner et al., 2020). These attributes also make simulation models powerful tools for counterfactual analysis. This systematic review focuses on the use of land change simulation modeling for counterfactual analysis as a particularly promising, but underdeveloped, causal inference approach in land system science (LSS).

Causal factors in land systems are best characterized as INUS causes (Mahoney, 2008). An INUS cause is ‘an insufficient but necessary part of a combination of causes, which [the combination] is itself unnecessary but sufficient for the outcome’ (Meyfroidt, 2016, p. 3). In other words, single factors are rarely the sole cause of LUCs, but rather many factors act in combination to produce LUC. Causal inference in LSS must therefore contend with many interacting factors that together produce observed LUCs. However, the situation often arises in LSS research that the causal effect of a particular factor must be understood, such as when designing an intervention or policy to manage undesirable LUC outcomes and contribute to sustainability (Ehrensperger et al., 2019; Munroe et al., 2019; Turner et al., 2020). For these reasons, the Rubin Causal Model (RCM) has gained traction in LSS and more broadly among social sciences and economics. According to the RCM, the causal effect of a given factor can be established by comparing the differences in an outcome variable under conditions when the outcome is affected by the causal variable of interest (i.e. the causal variable is a ‘treatment’) and when that causal variable is absent (i.e. ‘control’ conditions) (Holland, 1986; Rubin, 1974). This conceptualization is the foundation for counterfactual analysis through non-experimental causal inference.

Causal inference studies utilizing counterfactual reasoning have proliferated across diverse disciplines and methodologies, ranging from fixed effects and cross-lagged panel regression in environmental and earth sciences (Jain, 2020) as well as sociology (Halaby, 2004); difference-in-difference (DiD) in econometrics (Roth et al., 2022); before-after-control-impact (BACI) designs (Larsen et al., 2019) and quasi-experimental matching (Butsic et al., 2017) in ecology; and non-randomized controlled trials in medicine (Deaton & Cartwright, 2018) and criminology (Nagin & Sampson, 2019). Within the LSS domain, several of these methods were recently applied to investigate the causes of deforestation. Quasi-experimental matching methods were applied to quantifying the acceleration and/or displacement of deforestation caused by large-scale land acquisitions in Cambodia (Davis et al., 2015; N.R. Magliocca et al., 2019, 2020), and to estimate the effectiveness of decentralized forest governance in slowing deforestation in the Argentinian Dry Chaco (Nolte et al., 2017, 2018). The influence of illicit economic activities on the rate and area of deforestation was estimated using satellite imagery and fixed effects panel regression (Tellman et al., 2020) and BACI design (Sesnie et al., 2017).

Although these examples have significantly advanced inferential capabilities into the causes of deforestation, they were all challenged by two limitations inherent to statistical approaches to counterfactual analysis (Blackman et al., 2015; Ferraro, 2009; Meyfroidt, 2016). First, robust estimates of treatment effects depend on the availability of sufficient ‘control’ observations against which ‘treatments’ can be compared. Sufficient ‘control’ and/or ‘treatment’ observations that vary only in the causal factor of interest are often unavailable because of other confounding factors simultaneously affecting one or the other unit. Extensive sensitivity testing to unobserved confounding factors is required to obtain reliable estimates of causal effects of the treatment variable, but there is no widely accepted sensitivity threshold (Davis et al., 2015; N.R. Magliocca et al., 2019). In some cases, counterfactual observations may not exist. For example, to estimate potential land sparing due to the adoption of Green Revolution agricultural technologies and subsequent intensification, Stevenson et al. (2013) generated a counterfactual scenario without agricultural intensification using a global land use simulation model that was then compared to observed historical land use trends. The difference between the counterfactual simulation and historical outcomes could then be attributed to land sparing caused by Green Revolution agricultural intensification. Second, statistical counterfactual approaches are not well suited to studying endogenous interactions among causal factors. For example, agricultural frontier expansion and road construction have each been cited as causal agents of deforestation in the Amazon, although the sequence of causality is often context-dependent and difficult to determine (Arima et al., 2011; Dávalos, 2018). Thus, statistical approaches to counterfactual analysis in the context of LUC can estimate causal effects, but do not allow for an explanation of how the causal process occurred (Meyfroidt, 2016).

Establishing a causal mechanism is needed to fully explain how a cause or combination of causes produces causal effects (Meyfroidt, 2016). Identifying causal mechanisms is particularly important when designing policy interventions intended to target, manage, and modify unsustainable land-use processes and avoid undesirable trade-offs. This is due to equifinality in land systems: a causal effect (e.g. deforestation) can be produced through different causal mechanisms [e.g. population growth causing forest extraction (Coomes et al., 2011); commodity price spikes causing agricultural frontier expansion (Junquera et al., 2020); land control facilitating illegal economic activities causing pasture expansion (Tellman et al., 2021)] and each mechanism may require a different intervention (Meyfroidt, 2016). Causal effects and/or consequences of LUCs often vary spatially as well depending on local biophysical, socioeconomic, cultural, and demographic contexts (Verburg et al., 2019). Additionally, the effects of a casual mechanism within complex, highly connected land systems will likely be non-linear and have both direct and indirect effects due to cascading, rebounding, and/or spillover effects (Lambin et al., 2014; Meyfroidt et al., 2014, 2022). Isolating the causal mechanisms generating spatially and temporally varying outcomes from a sample of control and treatment observations may be infeasible, because such observations are a necessarily partial view of the full state of a land system.

In such situations, the analytical units shift from a sample of control and counterfactual observations to comparison of ‘counterfactual worlds’ (Nagin & Sampson, 2019). Counterfactual worlds ‘emerge as a consequence of their being subjected to different treatment regimes applied to all eligible population members over a sustained period of time’ (Nagin & Sampson, 2019, p. 123). Consistent with scenario modeling, counterfactual modeling explores the spatial and temporal implications of hypothetical alternative realities (Brown et al., 2013), but counterfactual analysis is distinctly backward-looking (i.e. counterfactual histories) and anchored to an observable time point beyond which the counterfactual world diverges from the observable world. Similarly, targeted modeling techniques, such as computational experiments (e.g. Murray-Rust et al., 2014; Sun et al., 2014) or sensitivity and uncertainty analysis (Ligmann-Zielinska et al., 2020), can isolate the direct effects of particular model parameters on land change outcomes, but they lack the temporal and spatial continuity with observed history (i.e. event sequencing) necessary for empirically grounded causal attribution and inference (Bennett, 2010; Waldner, 2012). This is consistent with the approach of process tracing, which examines event sequences and their links to outcomes hypothesized by the unfolding of alternative causal mechanisms (Bennett, 2010). Finally, counterfactual modeling relies on the RCM approach to distinguish the effects of a counterfactual treatment from the observed control. Unlike statistical counterfactual analyses, however, counterfactual modeling does not rely on discrete units of analysis (i.e. sample treatment and control observations or locations). Different locations or segments of a population might be affected by an intervention unevenly – directly, indirectly, or not at all – which is not easily captured by discretizing analytical units within land systems characterized by endogenous spatial, temporal, and/or cross-scale interactions (e.g. spillover, cascading, or rebound LUC). Spatially explicit land change simulation modeling can accommodate such complex causation through neighborhood contingencies and dynamic feedbacks. Thus, counterfactual land change modeling for causal inference combines the ability of land change simulation models to address complex causation in land systems with principles of spatial and temporal continuity of counterfactual analysis to enable empirically grounded causal attribution and inference.

Simulation models are essential tools for creating counterfactual worlds and enabling system-wide and spatially explicit causal inference, but their use in counterfactual analysis is relatively rare (He et al., 2013). The paradigm of counterfactual worlds has been adopted in fields adjacent to LSS, such as earth system science, to contend with the effects of LUCs and other human activities. For example, detection and attribution of weather and climate-related events is a growing subfield in climate science (Hannart et al., 2016; Rosenzweig & Neofotis, 2013; Runge et al., 2019) due in part to the demands in establishing causal attribution in climate-related litigation (Minnerop & Otto, 2019). Conceptually, causal attribution of extreme events or long-term changes must isolate the contributions of multiple anthropogenic and natural forcings from internal variability. Since long-term changes in climate forcings or the frequency of extreme events often exceed available observational data, counterfactual worlds simulated with climate models are required to generate likely outcomes in the absence of those forcings (Runge et al., 2019). Although there is a robust literature developed over two decades supporting land change modeling within LSS (Verburg et al., 2019), the extent to which land change simulation models have been leveraged for counterfactual analysis and the effectiveness of doing so remains unclear.

The objective of this systematic review is to assess the state-of-the-art of counterfactual land change modeling within and beyond LSS, and to evaluate the level of causal inference achieved. Specifically, we address the following research questions:

• What LUC phenomena and contexts (e.g. type of land system, world region, spatial and temporal scale and resolution) are most commonly studied with counterfactual simulation approaches?

• What land change modeling methodologies have been used for counterfactual simulation, and how do those vary with the phenomena and contexts being studied?

• What additional methodologies (e.g. household surveys, remote-sensing analyses) have been combined with counterfactual simulation to enable causal inference?

• To what extent has process tracing leading to the identification of causal mechanisms been achieved through counterfactual simulation? Are there conceptual, empirical, or methodological factors associated with more or less successful applications of counterfactual simulation for causal inference?

The remainder of the article proceeds as follows. The next section describes the search, selection, and coding procedures used in the systematic review. Trends in counterfactual land change modeling are then presented with attention to contexts in which counterfactual modeling has been applied as an indication of whether some LUC phenomena are better suited to counterfactual analysis than others. The implications of methodological and model design choices for achieving casual inferences (e.g. causal effects versus mechanisms) are also traced. Findings are then synthesized to identify current research gaps and promising avenues for future research.

Methods

We began the systematic review process by determining the way in which we would collect, select, and evaluate existing counterfactual LSS literature. As counterfactual analysis is an established methodology in multiple other fields of research, it was imperative that we determined a process to target literature from the LSS field. Our search process used relatively broad search terms in multiple peer-review literature databases to produce an inclusive pool of relevant articles, which was then filtered using relatively restrictive criteria as described below.

The first step in the literature collection process was the establishment of a ‘target set’ of five articles with which to test our search keywords. These five articles were identified based on expert opinion and were assembled to represent the expected diversity of counterfactual modeling topics, approaches, and disciplines applicable to academic literature in the LSS domain. The target set was assembled with articles in journals from a wide range of disciplines and included four articles considered representative of ‘true positive’ search results as well as one article considered a ‘false positive’ (i.e. closely related but ultimately excluded based on the particular modeling application). This was done to maximize the breadth of the literature search, allowing for the inclusion of potential false-positive results to ensure the inclusion of potential true positive results that would have otherwise been missed. Kroeger et al. (2019) served as a false positive. In this case, a counterfactual land change model was applied, but the authors implemented the counterfactual model as a scenario for future LUC and counterfactual scenario applications were ultimately excluded (described below).

The literature search was conducted in Web of Science (WoS) and Scopus databases. Search keywords¹ were selected from those of the target set and applied to the entire text of the articles. The search was constrained to articles written in English and published in peer-reviewed journals. No restrictions on publication date were imposed. Search terms were iteratively adjusted until all articles in the target set were returned in search results. The same set of search terms was used for both databases and returned 45 and 50 results in WoS and Scopus, respectively. After removing duplicates, a total corpus of 62 unique articles was created. The steps taken in the search process are presented in Figure 1.

Figure 1. Search and screening process to assemble the article corpus for review.

The next step in the selection process involved screening the abstracts of the 62 remaining articles. Only article abstracts that referenced explicitly modeling the causes or consequences of LUC and had an explicitly stated research goal of observing causal inference through counterfactual modeling were selected. Studies that were spatially explicit but not focused on LUC phenomena were excluded. Studies that applied only empirical or statistical methodology, such as quasi-experimental matching, or focused on land-cover change alone were excluded. This process removed 27 articles from consideration.

The next step of the review process was completing full article reviews of the remaining 35 articles. The criteria for an article’s inclusion in the final review was two-tiered. The first inclusion criteria was that an eligible article’s study must have produced or have been primarily concerned with spatially disaggregated counterfactual outcomes. Spatial variation in LUCs must have been studied at some disaggregated level relative to the study area’s total extent. For example, many studies used spatially continuous and gridded or rasterized landscapes (e.g. Bradley et al., 2017) that covered the entire study area, while Ma and Jin (2019) modeled change within the larger Beijing region by disaggregating results to the many smaller and irregular districts within the metropolitan area. Both examples were included as spatially disaggregated models.

The second inclusion criterion was that a study’s model uncertainty must have been assessed through comparison of simulated and observed spatial patterns of LUC for control locations and/or prior to the implementation time of the counterfactual. This process was essential for evaluating whether the differences between a model’s ‘business-as-usual’ (BAU) (i.e. control) outcomes and its counterfactual outcomes were due to the counterfactual treatment or model error. A particular strength of counterfactual modeling for causal inference is its ability to quantify the divergence between the empirically observed and counterfactually constructed outcomes over space and/or time. However, if a study did not conduct an assessment of model accuracy against observed LUCs, then there was no means for quantitative causal inference based on differences between BAU and counterfactual outcomes. This uncertainty assessment process is particularly important in the context of rapid LUC phenomena, which may violate assumptions of stationary causes of LUC and produce prediction errors equal to or greater than the effects of a counterfactual. Accordingly, we excluded studies that were only concerned with predictive counterfactual scenario modeling as their inherently forward-looking nature prevented quantification of model error (i.e. no empirical outcomes against which model accuracy could be assessed). Such model applications offered no ability to distinguish differences between BAU and counterfactual outcomes from variations in LUCs due to model uncertainty.

The final corpus of selected articles was then evaluated and analyzed through an iterative, consensus-based coding process among the authors. First, all 35 articles that received full-text review were read, analyzed, and coded independently by each author and our results were compared to identify coding and decision discrepancies, revise variable definitions and inclusion/exclusion criteria, and clarify the coding procedure. After the first round of coding, intercoder reliability was relatively high with a consensus among authors on the inclusion or exclusion of 30 of the 35 (86%) articles, while a consensus could not be made on the remaining five articles. The coding process that was of particular concern was deciding what constituted counterfactual modeling for causal inference versus scenario analysis. This was an area of coding ambiguity that, once clarified and resolved, was the decisive criteria in categorizing the remaining five articles. The resulting coding procedure for each variable is described in Table 2. The final corpus of articles after this process totaled 13, which are summarized by their study locations, land-use/cover contexts, research questions, and modeling frameworks in Table 1.

Table 1. Summary characteristics of reviewed counterfactual land change modeling articles.

Author	Specific Location	Land Use/Cover Context	Research Question	Modeling Method
Assaf et al (2021)	Itinguçu State Park, São Paulo, Brazil	Effect of shifting cultivation in tropical forest	How much land-use change would have occurred without the protected area?	Cellular automata and Markov models
Bradley et al (2017)	Paragominas, Brazil	Deforestation in tropical Forest	Does an ensemble modeling approach improve assessment of the impact of forest conservation policies?	Four spatial land change modeling frameworks
Denning et al. (2010)	Silicon Valley, California, USA	Effect of forest protection in housing stock	How did the establishment of protected areas affect housing stock?	Logistic regression
He et al. (2013)	Huangmei County of Hubei Province, China	Urban sprawl over arable land	What would have happened without the intervention of farmland preservation? What is the impact of farmland preservation policies in China?	Constrained cellular automata with logistic regression
Huang et al. (2018)	Nanjing City, China	Effect of urban pattern in habitat connectivity	How would landscape connectivity change with a monocentric urban structure?	Constrained cellular automata with logistic regression
Killeen et al. (2011)	(A Pará – Brazil; (B) East Kalimantan – Indonesia; (C) Madagascar; (D) Colombia; and (E) Liberia	Forest and Biofuel	How effective REDD+biofuel, RDL+Biofuel and most-likely scenarios will be in reducing GHG emission?	Spatial analysis and suitability modeling, demands calculated based on policy model
le Noe et al. (2021)	France	Multiple land-use and -cover conversions	What is the effects of land conversion, changes in land-use intensity and climate change on carbon stock accumulation in France?	A LUCCA model incorporating the CRAFT, GRAFS, AMG, SOC models
Lehtonen and Rankinen (2015)	Yläneenjoki River Basin Catchment, Finland	Agri-environmental policy regarding nutrient leaching	What were the economic and hydrological impacts of an agri-environmental scheme on agricultural production, land use, and nutrient leaching?	INCA-N model using DREMFIA modeling inputs
Ma and Jin (2019)	Beijing, China	Urban sprawl	What would have been the impact of either stringent or no greenbelt policy enforcement in the Beijing municipality on consumers’ utility, producers’ productivity, and their locational choices compared to the reality?	Cross-sector, spatial equilibrium models
Mondal and Southworth (2010)	Pench Tiger Reserve, India	Biodiversity Conservation	How would the absence of certain conservation intervention methods have affected biodiversity in the Pench Tiger Reserve in central India?	Cellular automata and Markov models
Teter et al. (2018)	Waterways, USA	Biofuel Policy/Production	What are the impacts of biofuel policy on crop selection, land use and, by extension, the water and soil management methods introduced by the land-use and land-cover?	BEPAM – CropWatR
Visconti et al. (2015)	Multiple Protected Areas around the world	Protected Areas	What impacts would the expansion of protected areas around the world have on species conservation, biodiversity preservation, and socio-economic development?	IMAGE, Extent of Suitable Habitat models
Xiong et al. (2018)	Hangzhou, China	Land-use and Land-cover change	How did construction of the Hangzhou international airport affect land-use and land-cover change in the area?	Cellular automata and Markov models

Table 2. Description of the variables coded through the iterative coding process.

Variable	Coding Description
Context	The category of land system in which the study was conducted (e.g. tropical forest, agricultural landscapes, urban growth areas).
Focal/Secondary land use	The land-use change (LUC) that is the main focus of the modeling (e.g. agricultural expansion) and any secondary land uses affected by those changes (e.g. forest).
Spatial scale	Spatial extent (e.g. administrative boundary, watershed) and resolution (e.g. granularity) of LUC analysis.
Temporal scale	Time span of modeling analysis and frequency of simulation/analysis (e.g. calibration intervals, years of remote sensing imagery within study period).
Outcome variable	The LUC variable or its effects compared between observed (i.e. control or ‘business as usual’) and counterfactual models (e.g. forest loss, sequestered carbon, housing stock).
Calibration	Calibration information included one or more of the following: (1) number of model calibrations against empirical data prior to implementing the counterfactual model; (2) frequency of calibrations within calibration period; and (3) calibration metrics (e.g. pixel-based comparisons of land use maps, regional LUC totals).
Uncertainty assessment	Evaluation technique for quantifying uncertainty in modeled outcomes independent of calibration period/data.
Spatial/Temporal counterfactual	Comparing ‘business-as-usual’ or observed with counterfactual LUC patterns based on location (spatial), trajectories (temporal), or both.
Primary/Secondary modeling approach	Methodology used to model LUC (primary) and its consequences (secondary).
Additional methods/data	Additional data or analytical techniques used to implement or evaluate control and counterfactual model outcomes (e.g. remote sensing imagery).
Causal inference level	Type of causal inference achieved through counterfactual modeling: causal effects, causal mechanisms, or process tracing.

Results

The articles reviewed generally conformed to established land change modeling efforts with respect to thematic applications and modeling methodologies. However, the different and distinct types of problems and research questions motivating a counterfactual approach illustrated the shortcomings of statistical counterfactual analyses for addressing complex land system dynamics or established land change modeling techniques for achieving causal attribution of observed LUCs.

Thematic trends

Distinct thematic trends were observed among the reviewed articles. Overall, the focus of counterfactual modeling studies was the quantification of the consequences, rather than the causes, of LUC in response to observed versus counterfactual policy interventions, climate change, and/or continued or divergent historical LUC trends. Eight studies (Denning et al., 2010; He et al., 2013; Lehtonen & Rankinen, 2015; Ma & Jin, 2019; Assaf et al., 2021; Mondal & Southworth, 2010; Teter et al., 2018; Visconti et al., 2015) explored the LUC outcomes of different land preservation or conservation policies. These varied in scale from the global, such as Visconti et al. (2015) which explored the impacts of expanding Wildlife Protection Areas on a global scale, to the local, such as Mondal and Southworth (2010) which investigated the effects of conservation intervention methods on species diversity within a roughly 300 km² tiger reserve. Other policies for which counterfactual evaluations were conducted included biofuel mandates or subsidies (Teter et al., 2018), agri-environmental schemes within the EU’s Common Agricultural Policy (Lehtonen & Rankinen, 2015), and greenbelt development restrictions in Beijing (Ma & Jin, 2019). Killeen et al. (2011) did not investigate the impacts of a particular policy intervention, but rather explored the possible greenhouse gas (GHG) emissions implications for linking different hypothetical combinations of REDD and biofuel policies. The other four studies were not specifically focused on policy interventions but instead examined the counterfactual outcomes of specific events, such as the construction of the Hangzhou international airport (Xiong et al., 2018), or of alternative LUC trends of deforestation (Bradley et al., 2017), carbon sequestration (le Noe et al., 2021), or urban structure shift (Huang et al., 2018).

There was a distinct land use context and problem domain to which most of the counterfactual modeling studies were applied. Ten of the studies performed environmental impact analyses of the consequences of LUCs in the contexts of either ecological/biodiversity conservation or biogeochemical cycling. Forest preservation and associated forest loss were investigated in moist tropical forest settings (Killeen et al., 2011; Bradley et al., 2017) and coastal oak woodlands (Denning et al., 2010). Evaluation of the effectiveness of protected areas for biodiversity conservation, which is a frequent focus of policy evaluation and counterfactual studies more broadly (Andam et al., 2008; Blackman et al., 2015; Ferraro, 2009), was conducted in three of the reviewed articles. There were two based in tropical settings (Mondal & Southworth, 2010; Assaf et al., 2021) and one that spanned the entire globe (Visconti et al., 2015). Other studies focused on impacts to biogeochemical cycling, such as water quality impacts of biofuel policies (Teter et al., 2018) or conservation agriculture policies (Lehtonen & Rankinen, 2015). le Noe et al. (2021) examined the long-term impacts of LUC trajectories on carbon stocks throughout all of continental France. Urban growth studies from China were also well represented with four studies (He et al., 2013; Huang et al., 2018; Ma & Jin, 2019; Xiong et al., 2018) investigating different effects of urban growth constraints on urban sprawl patterns as well as the trade-offs created by replacing urban land uses (e.g. agriculture). Notably, Ma and Jin (2019) and Huang et al. (2018) investigated urban growth impacts on wildlife habitat connectivity and ecological sensitivity, respectively, which reflected the relatively greater uptake of counterfactual modeling approaches in the conservation field.

Motivations for conducting counterfactual modeling analyses

A review of the authors’ stated motivations for choosing a counterfactual modeling approach provided insights into the nature of the problems currently considered best suited to the use of such an approach. Several studies cited the need to challenge prevailing assumptions leading to support for or resistance to a particular policy or intervention or a need to resolve conflicting empirical evidence in the existing research literature. For policy interventions, a priori assumptions about their effectiveness often served as the motivations for action, such as the establishment of protected areas and presumed effectiveness of exclusionary protected areas to limit impacts of LUC. In many of the contexts in which protected area effectiveness was studied with counterfactual modeling, authors asserted that such a causal relationship between exclusion and preservation was not critically examined from a historical perspective. For example, Assaf et al. (2021) observed that the removal of local populations for the establishment of protected areas in Brazil’s Atlantic Forest was based on the perception that the shifting cultivation agriculture practiced by local communities degraded forest ecosystems. However, those perceptions did not take into account historical trends in land-use and land-cover change prior to the protected area establishment. A dynamic, counterfactual land change modeling approach enabled the simulated evolution of spatial patterns of land change prior to the establishment of the protected area and the ability to compare future scenarios of LUC both with and without the protected area. The authors found little difference in LUC impacts between the observed and counterfactual outcomes. Denning et al. (2010) used counterfactual land change modeling to examine two competing hypotheses about the effects of protected areas on housing stock in the San Francisco Bay Area in the state California. Land protection, in the form of open space conservation, was argued by opponents of the policy to create scarcity in the housing market due to the constraints it applied to the overall housing supply. To interrogate this argument, the authors’ counterfactual modeling study simulated what the residential development patterns would have been like in the Sane Francisco Bay Area without the land conversation policies. They found minimal impacts on housing supply because land protection was implemented mainly on steep slopes and in areas with poor drainage where substantial housing was unlikely to occur regardless of policy.

Similarly, Killeen et al. (2011) used a counterfactual scenario approach as an ‘illustrative’ model to demonstrate the benefits of linking forest conversation and biofuel production policy goals as well as to address the resistance to both policy instruments. Existing research has demonstrated the extent of forested area that has been converted for biofuel production despite policy instruments like REDD that aim to incentivize forest conservation and reforestation (Fargione et al., 2008). The authors implemented three counterfactual models: one with REDD policy only, one with biofuel expansion without REDD (i.e. business-as-usual), and one using both REDD and biofuel expansion but without regulatory coordination. These counterfactuals were then compared with a model that ‘postulates that a ratio of 4:1 forest conservation to biofuel cultivation be linked to proposals for reducing emissions from deforestation and forest degradation (REDD + Biofuel), while a ratio of 9:1 biofuel cultivation to reforestation on degraded landscape (RDL + Biofuel) be linked to the afforestation/reforestation component of the Clean Development Mechanism’ (Killeen et al., 2011, 4815). Their ‘illustrative’ modeling approach was implemented in five case study regions that experienced forest loss from 2000 to 2010, which enabled an assessment of hypothetical biofuel production which avoided deforestation and GHG emissions.

Policy-oriented counterfactual modeling efforts were also in consensus that the approach could uniquely improve policy evaluation studies by isolating the effects of a particular policy intervention from the complex influence of other factors on LUC. Due to the multi-factor and multi-scale nature of LUC process, some outcomes of interest are likely to be influenced by a policy’s implementation, while other outcomes may not be influenced to the same degree or at all. This significantly complicates the evaluation of the actual impact of the policy from observational data alone (Ferraro, 2009; Lehtonen & Rankinen, 2015; Mondal & Southworth, 2010). In other words, the policy of interest can and likely will interact with factors beyond those specifically targeted by the policy, and the effects of these external drivers are difficult to disentangle from the effects of the policy or treatment. Mondal and Southworth (2010) were among the earliest to assert the value of counterfactual modeling combined with remote sensing time series analyses as an effective policy evaluation tool in response to the lack of well-designed empirical studies to estimate effectiveness of conservation intervention, particularly in the context of developing countries where data deficiencies may exist. Mondal and Southworth argue that conservation in this context ‘is often the product of complex interactions between law enforcement, local livelihood and different forest management activities, thus making it difficult to adapt a conventional counterfactual approach’ (Mondal & Southworth, 2010, 1717). Similarly, in some highly biodiverse settings, there can be multiple conservation interventions occurring inside and outside of the protected areas. Thus ‘it is difficult to choose “control” and “treatment” units, since “control” units, even though apparently unaffected by the protected area establishment, are subjected to other, simultaneous, conservation efforts’ (Mondal & Southworth, 2010, 1717). A simulation modeling approach was thus adopted to overcome the challenges of empirical counterfactual analysis.

In other examples, Lehtonen and Rankinen (2015) motivated the use of counterfactual modeling based on the limitations of previous research evaluating the effects of agri-environmental schemes using field monitoring data, which were able to demonstrate the ‘development in selected key variables, but not the actual impact of the agri-environmental scheme’ (Aakkula et al., 2012; in Lehtonen & Rankinen, 2015, p. 131). A specific objective of the agri-environmental scheme was to lower nutrient loading from agriculture but other factors, such as decreasing real crop prices and increasing fertilizer costs, likely influenced observed trends of reduced nutrient loading independent of the scheme. Because of this complex causation, it was ‘important for policy design to evaluate how agri-environmental schemes perform under other drivers of agricultural development’ (Lehtonen & Rankinen, 2015, p. 131), which was made possible with the use of counterfactual scenarios. In the context of urban growth and the implementation of greenbelts in Beijing, Ma and Jin (2019) stated that ‘empirical [spatial and temporal] comparisons cannot fully differentiate which impacts are due to the greenbelt and which are due to other factors, for example market incentives and institutional backgrounds’ (Ma & Jin, 2019, p. 80). To address this limitation, the authors produced historical counterfactuals comparing a fully enforced greenbelts policy, a fully absent policy, and the observed greenbelt development outcomes around Beijing. They found that a relaxation of the first greenbelt would increase economic productivity, and key transportation improvements would improve the benefits of the later greenbelts. Similarly, Xiong et al. (2018) argued that a key limitation of previous studies of the influence of infrastructure on LUC was that they did not consider the ‘dynamics of land use change regardless of the intervention of infrastructure. Thus, the traditional way of evaluating the effect of infrastructure on LUCC through before and after comparison may be biased’ (Xiong et al., 2018, p. 2). The authors used counterfactual modeling to describe the positive association between infrastructure investments and socioeconomic development, specifically the positive spillover effects such investments had on overall employment, incomes, traffic patterns, and economic investment patterns – all of which led to additional LUC beyond the infrastructure itself.

In addition to the advantages over empirical counterfactual study designs, counterfactual modeling approaches were deemed necessary to distinguish spatially varying effects of policies or alternative LUC patterns. Importantly, for the context of LUC analysis, different locations and/or portions of populations likely experience the effects of the counterfactual treatments differently; some are impacted directly, while others may only be affected indirectly through spillover or cascading LUCs. Others may not be impacted at all and represent ‘controls’ embedded within a larger landscape or community experiencing varying levels of treatment effects. Statistical counterfactual approaches designed for causal inference estimate the overall impact of a policy or treatment, but do not adequately consider spatially varying impacts (He et al., 2013).

This was a consistent refrain across sectors and LUC contexts. He et al. (2013) asserted that ‘farmland preservation policies did not prevent urban expansion encroachment on the arable land but rather controlled the rates, locations, and densities of this land development’ (He et al., 2013, p. 137). Teter et al. (2018) cited related research about the environmental impacts of biofuel production as a motivation for using spatially explicit counterfactual modeling. ‘Geographically explicit water footprint (WF) assessments find that the WF of cultivating biofuel feedstocks such as corn stover and grain, and wheat straw vary by more than an order of magnitude across US counties due to spatial heterogeneity in water use and due to variations in the methods used for the assessment’ (Chiu & Wu, 2012; in Teter et al., 2018, p. 2). To address this gap, Teter et al. (2018) coupled biophysical and economic models to simulate mixes of biofuel and other crops, land uses, and crop management choices and their spatially varying impacts on water quality and soil management under alternative policies. Lehtonen and Rankinen (2015) found that nitrogen loading of intensive agriculture increased in their counterfactual scenario of no agri-environmental support scheme as expected, but in more extensive agricultural regions nitrogen loading was actually lower than in the presence of the agri-environmental support scheme. This revealed spatially varying effects of the policy. ‘It appears that this opportunity for nutrient loading reduction has been missed in the realised FAEP which seems effective in inhibiting nutrient balances in intensive production regions while contributing to maintaining production [in] less favoured areas’ (Lehtonen & Rankinen, 2015, p. 142). In the urban growth context, Huang et al. (2018) measured changes in landscape connectivity under observed and counterfactual urban expansion scenarios and their impacts on four wildlife species of interest. They found that a shift from a monocentric to a polycentric urban form had differential impacts based on how those species utilized space (e.g. dispersal ranges and territorial size). Overcoming the challenges of statistical counterfactual approaches in a spatially explicit manner can better support spatially targeted policy interventions as compared to a uniform policy inattentive to spatially vary effects.

Perhaps, the most ambitious response to the importance of spatially vary effects of policy interventions was provided by Visconti et al. (2015). The authors sought to maximize the effect or effectiveness of protected areas by targeting optimal and spatially varying implementation. They noted that previous studies ‘have been useful for identifying the most efficient networks of sites that contain under-protected species and ecosystems. However, none of them estimated or sought to maximize the impact of these Pas in reducing biodiversity loss, i.e. they did not investigate how the proposed Pas would reduce habitat loss under future scenarios of anthropogenic threats. … Global conservation planning studies ignoring future socio-economic conditions, and in particular, expected agricultural production, give a false sense of feasibility based on the small amount of additional protection required to achieve all targets’ (Visconti et al., 2015, p. 2). Visconti et al. (2015) used a counterfactual modeling approach to compare the possible effects of expanding protected areas versus doing nothing under different socio-economic and consequent LUC scenarios, and how protected area expansion interacted with other LUC dynamics outside of the protected areas. In particular, the expansion of protected areas may displace human activities, such as agriculture, to habitats outside of the protected areas important for species persistence. Due to these potential interactions, the study found that expanding protected areas under the BAU scenario resulted in a worse prognosis than doing nothing, while a more spatially targeted approach towards threatened species could increase available habitat to over 60% of terrestrial mammals.

The possibility of assessing time-varying effects with counterfactual modeling was also identified as a strength of the approach. Ma and Jin (2019) noted the importance of being able to study time-varying effects of policy interventions (i.e. producing land use outcomes for multiple calibrated and/or counterfactual time points), particularly as relative to a comparative statics or spatial-only analysis. ‘Compared to independent cross-sectional predictions, the recursive modelling structure is capable of showing the effects of chronic and large-scale land use changes. A temporal dimension enables it to reveal the growth inertia and path dependency of greenbelt evolution from the previous decade’ (Ma & Jin, 2019, p. 90). Additionally, land use legacies (Barton et al., 2016; N.R. Magliocca & Ellis, 2016; Ramankutty & Coomes, 2016; Schulp et al., 2019) can explain contemporary impacts to biogeochemical processes from LUCs. For example, le Noe et al. (2021) studied the long-term carbon budget associated with LUC dynamics in France. Noting the complexity of land-use dynamics that evolve over both time and space, they argued that a long-term socio-ecological perspective was needed to identify drivers of changes to landscape carbon budgets. Moreover, previous studies have used a static coefficient of carbon content in ecosystems which does not capture the legacy effect of past land use. ‘Reliable, robust and dynamic modelling approaches spanning across all land-uses are greatly needed to assess the drivers of long-term changes in [carbon] stocks in land ecosystems and encompass both their natural and socioeconomic components’ (le Noe et al., 2021, p. 2). A counterfactual modeling approach applied to simulate long-term LUCs estimated that carbon emissions induced by fossil fuel consumption in France from 1850 to 2015 were 9 times that of the carbon sink in terrestrial ecosystems.

Methodological considerations

Land change simulation modeling takes many forms, ranging from top-down (e.g. computable general equilibrium) to bottom-up (e.g. cellular automata, or CA) and pattern-based (e.g. pixel-based) to process-based (e.g. agent-based) models (Brown et al., 2013; Verburg et al., 2019). However, not all modeling approaches are suitable for counterfactual analysis and causal inference. The chosen modeling approach and efforts to calibrate and validate model outcomes against observable LUC histories determined whether control simulations could be anchored to an observed reality, and whether causal effects were estimated from different spatial patterns of LUC or if causal mechanisms were established through event sequencing and/or process tracing.

Diversity of modeling approaches

Spatially explicit counterfactual modeling was commonly conducted over continuous, pixel-based landscapes or aggregated to discrete bounded areas corresponding to socially or biophysically relevant units (e.g. administrative units, watersheds). Pixel-based modeling approaches consistently used Markov chain and/or logistic regression techniques. Given the process-based nature of ABMs and their applications in modeling complex system dynamics, one might consider them to be more suitable for causal inference than CA and Markov models. Lack of other well-known framework such as CLUE (Verburg & Overmars, 2009) or SLEUTH (Jantz et al., 2004) frameworks for modeling was also notable. Mondal and Southworth (2010) pointed out that the stationarity assumption of Markov models, i.e. past trends of land use transitions will remain the same in the future, make it a suitable approach for simulating counterfactual scenarios concerned with what would have happened if the trend continued or some intervention did not take place. Assaf et al. (2021) used the typical methodology of a Markov model to extrapolate land use demands under observed and counterfactual scenarios based on transition matrices derived from past LUCs. CA models, due to their simplicity, were often combined with the Markov model (CA-Markov model) to spatially allocate land use demands based on regional land use priorities and local neighborhood factors. Xiong et al. (2018) also used a CA-Markov model to construct a counterfactual scenario that reveled construction of an international airport actually had a slowing effect on development in the short term, which was not apparent from a simple before-and-after comparison of remotely sensed LUC patterns. Bradley et al. (2017) used an ensemble modeling approach combining the Lulcc‑R (LUCR), StocModLCC (SM), TerrSet land‑change modeler: Imazon (TS‑IM) and TerrSet land‑change modeler: Clark Labs (TS‑CL) models to produce 21 counterfactual outcomes for the same study area, time period, and set of predictor variables. The comparison of results from the different models showed that LUC predictions could vary widely based on different structural assumptions and calibration periods. This emphasized the challenge of uncertainty assessment for LUC simulation generally and the generation of counterfactual scenarios specifically.

Logistic regression was also frequently used to predict transition probabilities either independently or in combination with CA models to allocate counterfactual LUCs over space. A benefit of logistic regression cited by several studies was its ability to quantify the effect of underlying drivers of observed LUC as well as to compute transition rules under different scenarios when constrained by empirically derived suitability metrics (He et al., 2013; Huang et al., 2018). He et al. (2013) combined transition rules derived from logistic regression with a constrained CA model to construct a counterfactual scenario without the intervention of farmland preservation policies. Huang et al. (2018) combined logistic regression and CA modeling with a landscape graph modeling approach to simulate wildlife habitat connectivity under counterfactual monocentric urban development patterns. Denning et al. (2010) used logistic and linear regression methods to identify driving factors that significantly influenced the probability of development and housing density, respectively.

Beyond these conventional LUC modeling techniques, a diversity of other modeling techniques were used to address specific consequences of LUC. For example, Killeen et al. (2011) were interested in the potential coordination of REDD and biofuel policies. They used a spatial suitability modeling approach based on land use conversion criteria to identify likely areas of land conversion to fulfill each policy objective. Outcomes of the suitability model were then used in a spreadsheet model to compute GHG emissions in each scenario. Lehtonen and Rankinen (2015) used the dynamic, recursive economic agriculture sector model (DREMFIA) to simulate annual agricultural production, which served as inputs to the INCA model to compute nitrogen leaching. Ma and Jin (2019) used a recursive spatial equilibrium model to simulate counterfactual urban development patterns and subsequent housing rents, industrial productivity, and average household utility across 130 disaggregated zones in Beijing. To determine the water impact of biofuel policies, Teter et al. (2018) used the economic model BEMAP to predict changes in land use, cropping mixes, and management choices under different biofuel policies. In particular, the impacts of changes in irrigation on water flows between the soil, crop, and atmosphere were additionally model with CropWateR. Visconti et al. (2015) used the Integrated Model to Assess the Global Environment (IMAGE) to simulate likely locations of agricultural expansion, crop, pasture, and bioenergy production in the context of expanding protected areas. The diversity of modeling techniques observed illustrated the potential of counterfactual modeling’s application to a wide range of social, economic, and environmental research questions.

Model calibration, validation, and sensitivity analysis

Among the modeling efforts reviewed, there was also a clear divide based on the modeling technique used and attention to model calibration and validation. The modeling studies that used CA-Markov and/or logistic regression methods (Denning et al., 2010; He et al., 2013; Bradley et al., 2017; Huang et al., 2018; Mondal & Southworth, 2010; Xiong et al., 2018; Assaf et al., 2021) explicitly addressed calibration and validation of land use outcomes, which were motivated by acknowledgements of the uncertainties present in LUC processes. Calibration was based on the transition probabilities between at least two land use maps from time points prior to the implementation of the counterfactual. For example, Mondal and Southworth (2010) used land cover maps from 1977 to 1989 to estimate LUC transition probabilities for a CA-Markov model in the region of the Pench Tiger Reserve. The calibrated model was then used to predict counterfactual LUC patterns for 2000 and 2007 in the absence of a revised forest policy implemented in 1988. Bradley et al. (2017) explored the performance of an ensemble modeling approach for spatially explicit land change modeling. Four different spatial modeling frameworks were calibrated with the same set of predictor variables but over varying intervals within the same calibration period prior to policy intervention to generate 21 counterfactual business-as-usual scenarios. An ensemble approach was then used to compare individual and collective model outcomes to observed land change outcomes to improve prediction accuracy of the ensemble model, quantify uncertainty in land change predictions, and increase support for predicted policy impacts.

Validation, and the associated uncertainty assessment, was typically done through comparison of simulated and observed land use outcomes for a given time period after calibration but prior to implementation of the counterfactual. For pixel-based models, pixel-by-pixel comparison was typically used with either crisp kappa that penalized for each incorrect pixel (Assaf et al. 2021, Mondal & Southworth, 2010; Xiong et al., 2018) or fuzzy kappa (He et al., 2013; Huang et al., 2018) that allowed for small displacement of predicted land use classes from their actual position. It was also common to perform multiple replicate runs of the model (Mondal & Southworth, 2010; Assaf et al., 2021) or Monte Carlo methods (He et al., 2013; Bradley et al., 2017; le Noe et al., 2021) for uncertainty assessment on model prediction. The other six studies reviewed either did not address calibration and validation explicitly or performed calibration or validation on outcomes other than LUC, such as the number of zonal employed residents and jobs (Ma & Jin, 2019) or stream flow statistics (Teter et al., 2018).

Several studies took additional steps to assess the plausibility of their counterfactual outcomes beyond validation based on map comparisons. Bradley et al. (2017) created alternative counterfactuals using different calibration methods against observed deforestation trends prior to policy intervention. Since the counterfactual model outcomes were intended to predict deforestation rates in the absence of forest protection polices, there was no way to validate the predicted land change outcomes against observed changes. Alternatively, the counterfactual outcomes were compared against paired random models for each individual model prediction using the same number of change pixels (Bradley et al., 2017: 1220). The plausibility of each counterfactual outcome was assessed relative to its paired random model using the number of ‘hits’ (predicted change where there was observed change) versus the sum of ‘hits’, ‘misses’ (observed change which was not predicted), and ‘false alarms’ (prediction of change where there was no observed change). Counterfactual scenarios were considered plausible if they performed better than their random counterparts.

Other studies took a different approach of implementing ex ante constraints on counterfactual outcomes. Huang et al. (2018) ensured that their counterfactual scenario had a realistic starting point by enforcing equal quantities of each land use type for the last year of calibration so that the counterfactual and factual scenarios had the same urban size and land-use intensities to isolate the difference in urban structure (i.e. monocentric vs. polycentric). Xiong et al. (2018) noted that the assumption of stationarity in the implementation of Markov chain models can lead to unrealistic trends in counterfactual simulations. For example, if past trends of decreasing cultivated land and increasing developed land continued, the study area would be dominated by developed land and lack cultivated land, which would be contrary to the land management system enacted in China. The authors suggested two approaches to address this issue. ‘One way is to shorten the time interval of the simulation, as LUCC is approximately constant in the short term. Another way is to identify the main factors that affect land use change in different periods and take them into account correspondingly by combining a CA – Markov model with a multi-criteria evaluation’ (Xiong et al., 2018, p. 14). Finally, Killeen et al. (2011) implemented counterfactual scenarios to evaluate the potential of linking forest conservation and biofuel production regulations. To improve the reliability of their scenarios, spatial suitability analysis was first conducted to identify likely locations within each of the case study regions for two woody perennial biofuel feedstock species with ‘known climatic requirements, cultivation practices and existing global markets: eucalyptus and oil palm’ (Killeen et al. 2011: 4817). Conservative and previously published values for biomass accumulation and biofuel production were also used to ground estimates.

Sensitivity analyses were also performed by several of the reviewed studies. Alternative counterfactual outcomes were produced by implementing ranges of key parameters underlying counterfactual assumptions to investigate the sensitivity of policy design and/or implementation and resulting deviations from what was observed. Lehtonen and Rankinen (2015) consider this possibility for sensitivity analysis of the uncertain parameters, such as specific policy conditions or future commodity prices, ‘a significant advantage of this kind of model-based counterfactual analysis’ (Lehtonen & Rankinen, 2015, p. 143). Ma and Jin (2019) used a similar justification for their analysis of greenbelt implementation and enforcement counterfactuals. While previous research suggested that weak governance and planning regulation was primarily to blame for the failure of greenbelts around Beijing, the authors quantified economic impacts by varying assumptions between fully enforced greenbelt policy and no greenbelt policy counterfactuals to identify economic impact as a more likely explanation for hindering greenbelt implementation. le Noe et al. (2021) implemented a scenario-based approach similar to a local sensitivity analysis (Ligmann-Zielinska et al., 2020; N.R. Magliocca et al., 2018; ten Broeke et al., 2016). The authors used five counterfactual scenarios designed to isolate the influence of land conversion (all land use areas held constant), management (land-use intensity held constant), biomass accumulation (rates remained constant), and temperature and precipitation (held constant at 1850 values) on long-term carbon stocks.

(Un)Successful causal inference with counterfactual modeling

One of the main strengths of the counterfactual approach is the potential for causal inference beyond that of purely empirical and statistical techniques (Ferraro, 2009; Nagin & Sampson, 2019). The ability of a land change model to make causal inferences through counterfactual analysis relies on rigorously anchoring baseline simulations to observable LUC histories prior to implementing a counterfactual. If this is accomplished, the modeling methodology chosen will determine the level of causal inference that can be achieved. Pattern-based land change models were limited to estimating causal effects from differences in observed and counterfactual spatial patterns of LUC, whereas modeling approaches that targeted the processes generating LUC were able to assess the effects of causal mechanisms.

Accordingly, the level of causal inference achieved varied with the targets of the counterfactual outcomes in a given study (e.g. model parameter, policy scenario, or underlying assumption). Of the 13 studies reviewed, only one (Teter et al., 2018) achieved process tracing, while three provided insights into causal mechanisms (Huang et al., 2018; le Noe et al., 2021; Ma & Jin, 2019), and the remainder were limited to causal effects. Process tracing was achieved by Teter et al. (2018) through the integration of biophysical and economic models that enabled the linkage of different farmers’ crop selections and land uses under alternative biofuel policies to alternations in surface water and groundwater use and availability. The use of multiple models allowed the researchers to not only investigate the causal effects of different biofuel policies on land use outcomes, in which the only differences between observed and counterfactual outcomes were the policy objectives, but also to isolate the causal mechanism of farmers’ decisions responding to each alternative policy’s incentives or constraints. The spatial and temporal implications of farmers’ decisions could then be traced to alternative land uses and water demands, which were then linked to impacts on water availability. In this case, the causal mechanism producing emergent LUCs and their consequences was isolated by the counterfactual, and the spatial and temporal implications of the causal mechanism could be traced.

The three additional models that enabled inferences into the causal mechanisms isolated processes producing alternative LUCs and their consequences but did so assuming that different processes were operating rather than modeling their emergence. Huang et al. (2018) used a monocentric urban growth structure as a counterfactual to observed polycentric urban growth patterns to investigate consequences for wildlife habitat. They were able to isolate the effects of urban structure on ecological networks and differentiated impacts on specific species. However, changes in the urban growth process were imposed rather than explicitly modeled, which prohibited process tracing between spatial and temporal changes in urban form to specific ecological impacts. Ma and Jin (2019) used a spatial equilibrium model to modify underlying factors producing alternative land-use and regional economic patterns, specifically distance from work to residence (transportation cost), household utility, floor space demand, and rent. However, while the spatial equilibrium approach can effectively investigate economic structural changes that produce different levels of economic productivity, the top-down and aggregated nature of the spatial equilibrium models did not allow causal linkages between regional-level outcomes and micro-level (e.g. households, businesses) responses to counterfactual conditions. le Noe et al. (2021) isolated the individual and combined effects of LUC extent and intensity and climate on long-term carbon stocks. Similar to Huang et al. (2018), the authors implemented changes in factors that dynamically affected the outcome of interest (stored carbon in this case) but imposed, rather than explicitly modeled, the LUC processes generating those changes. This precluded the possibility of dynamically linking changes in these factors to spatial and/or temporal changes in carbon stocks.

Discussion

Overall, the thematic foci and modeling methodologies of counterfactual land change modeling were similar to those found in the broader land change modeling literature. In contrast, the authors of the reviewed studies expressed motivations for land change modeling distinct from broader land change modeling because of the emphasis on causal inference. Typically, land change modeling is promoted as a method to cope with the multi-dimensional and multi-scale causation of land systems (Verburg et al., 2019) and this was certainly echoed in the reviewed studies. In addition, the emphasis on causal attribution brought into focus the limitations of statistical counterfactual approaches for characterizing complex land system responses to interventions, which is unique to the counterfactual modeling context. Despite this attention to causal inference, the reviewed models shared many of the same problems of broader land change modeling efforts.

The limitations reported in the previous section generally have their roots in three areas. First, the majority of articles reviewed lacked methodological details about one or more steps of the standard model development cycle: model selection/building, calibration, and validation (van Vliet et al., 2016). This was likely because most articles used pre-existing modeling frameworks or software, and those that used novel models presented the development details in separate articles independent of the counterfactual application. The lack of model details pertaining specifically to the counterfactual application, particularly about the first two stages of model development, made it difficult to assess the spatial and temporal bounds of the counterfactual analysis, and thus the applicability of the causal inferences drawn from the counterfactual analysis (N.R. Magliocca et al., 2018). For instance, insufficient information about the temporal depth or spatial resolution of the input data used to calibrate LUC transitions prohibits an assessment of whether any scale-sensitivities in the underlying LUC process were captured in transition matrices or other statistical relationships (Assaf et al., 2021).

Second, except for Bradley et al. (2017) and Assaf et al. (2021), little attention was given to the realism and/or robustness of the counterfactual model outcomes. Generally, the modeling methodologies resembled those used for simulation of future land change scenario assessments, and thus no consistent approach to validating the historical continuity of model outcomes was apparent. Conventional applications of empirically calibrated, pattern-based land change modeling typically use all available LUC observations for calibration prior to the time period of the model simulation, and validation is consequently limited to a single time period, if it is conducted at all. Such applications, particularly those using Markov chain models, rely on an assumption of stationarity in the process generating future land-use outcomes. This assumption has long been debated in the land change modeling literature (Bell & Hinojosa, 1977; Baker, 1989), and is often violated in rapidly changing environments where historical LUC trends are not good predictors of subsequent changes. If the purpose of the simulated future LUC scenarios is merely heuristic, then this is not necessarily a problem, as qualitative differences between scenarios can be informative and traced to model assumptions (Baker, 1989; Assaf et al., 2021). However, if the purpose is causal inference based on comparisons of simulated and observed LUC trends, then the assumption of stationary – while key to the counterfactual approach – needs to be bounded as rigorously as possible. Without a realistic counterfactual scenario, differences between simulated and observed LUC outcomes may be driven more by flawed calibration than the real effects of the intervention of interest.

Third, and related to the previous two issues, there is a lack of coherent or accepted best practices for counterfactual land change modeling. This review revealed that the workflows employed for counterfactual modeling varied widely and only partially followed established practices of conventional land change modeling. This was due in part to the diversity of disciplinary perspectives, problem domains, and different modeling platforms used, which is indicative of a relatively new methodological approach. If spatially explicit counterfactual land change modeling is to reach its potential for causal inference, then a set of codified practices is needed.

Based on the reviewed modeling efforts and the broader spatially explicit land change modeling literature (Brown et al., 2013; Camacho Olmedo et al., 2015; Messina et al., 2008; van Vliet et al., 2016; Verburg & Overmars, 2009; Verburg et al., 2019), we address this need by proposing a set of best practices and modeling workflow (Figure 2) for counterfactual land change modeling for causal inference. Our recommendations follow the established practices of conventional land change modeling (van Vliet et al., 2016), but place emphasis on the unique requirements of the calibration and validation phases to ensure robust counterfactual simulation that are anchored to an observable reality.

Figure 2. Best practice modeling workflow for counterfactual land change modeling. Best practice for a single model would be to calibrate and validate over at least three times of observed land change data (top model) before generating a counterfactual prediction. An ensemble modeling approach would calibrate and validate over different combinations of times and time spans of observed data (top three models). Finally, using as much observed data before the counterfactual time period as possible for calibration, counterfactual model predictions can be compared to observed trends and effect sizes estimated (inset plot).

Model selection depends on the purpose, computational demands, and data availability of each research context. Not all land change modeling methodologies are suitable for counterfactual analysis and causal inference. The chosen modeling approaches were in large part determined by the need to anchor baseline simulations to an observed reality through calibration and validation. Cellular automata models are used extensively in spatially explicit land change model (Tong & Feng, 2020) and were the mostly frequently used approach in our review. Their simplicity in use and established calibration methods make them suitable for creating spatially explicit counterfactuals, and they can be used in cases where there are inadequate data available for characterizing driving factors of LUC. Additional modeling approaches were used (see section 3.3.1), but calibration of LUC trends was either not performed or insufficient methodological details were provided (i.e. LUC was not the focus of the counterfactual application).

After model selection, calibration should be conducted against empirical LUC trends in locations unaffected by and/or times prior to the intervention of interest. Standard model calibration techniques, either automated (e.g. model fitting) or statistical (e.g. logistic regression) (van Vliet et al., 2016), suffice for most applications. Validation best practices follow those used in standard land change modeling applications, including location and pattern accuracy assessments and, to a lesser extent, sensitivity and uncertainty analyses (van Vliet et al., 2016). The type of validation technique used must be responsive to the goals of the counterfactual analysis. For example, if the encroachment of land use into sensitive habitat is analyzed with (observed) and without (counterfactual) a protected area boundary, the exact locations of counterfactual LUC are of interest and a location accuracy assessment using pixel-to-pixel comparison might be most appropriate. Pixel-based accuracy assessments have been criticized for underestimating model accuracy by exaggerating the effects of near-misses. Modified pixel-based assessments, such as multi-resolution approaches (Pontius & Millones, 2011; Pontius et al., 2011) and methods based on fuzzy membership (Hagen, 2003; van Vliet et al., 2013), differentiate between small errors and complete misses. If, however, the goal of the counterfactual analysis is to identify spatial patterns or classes of land-use change and/or stability, rather than specific pixels, then allocation validation techniques may be more appropriate. For example, a land change model can be assessed by comparing the allocation of (in)correctly predicted pixels in persistently changing or unchanging regions within a landscape relative to a random model (Brown et al., 2013). This provides a better assessment of the accuracy of the simulated process for allocating LUC across space.

At this point, however, the best practices and workflow for counterfactual modeling diverge slightly from standard practices and place greater value on validating model predictions in order to create robust counterfactual simulations (Figure 2). This is an area of some weakness in conventional land change modeling, as van Vliet et al. (2016) found that 31% of land change models reviewed did not report any validation. Among the 13 articles we reviewed, four articles (~30%) did not mention validation or uncertainty assessment of counterfactual outcome. We argue that validation prior to counterfactual simulation, i.e. anchored to an observed reality, is essential if one is to causally attribute differences between observed and ‘counterfactual worlds’ to a given intervention or change rather than model uncertainty and error. In other words, the researcher must be able to establish that the effect size of the intervention or change is greater than the model’s prediction error in locations and/or times prior to the counterfactual. This is a unique requirement for counterfactual land change modeling.

Consequently, causal inference with counterfactual modeling requires more data than conventional land change modeling to maximize the realism of the counterfactual simulation. Increased data demands subsequently constrain the modeling methodologies used in practice, which in turn limit the level of causal inference that can be achieved. Revisiting the discussion of causal effects versus causal mechanisms (Meyfroidt, 2016), the models reviewed predominately demonstrated causal effects by comparing spatial patterns of LUC between observed and ‘counterfactual worlds’. Only one, Teter et al. (2018), was able to achieve the deepest level of causal inference in the form of process tracing, and multiple modeling paradigms were integrated to do so. Notably, none of the reviewed studies used agent-based modeling (ABM) for counterfactual analysis even though ABMs are widely used in the land change modeling community due to their process-based nature capable of representing complex land system dynamics (Filatova et al., 2013; O’sullivan et al., 2016). There are to our knowledge no examples of ABM applications of counterfactual modeling approach for causal inference. The input data required for ABMs to achieve the continuity with spatial LUC patterns and historical event sequencing required for casual attribution would be substantial. ABMs that have achieved such empirical validity are typically applied for prediction or scenario analysis rather than historical counterfactual analysis (Polhill et al., 2021).

The challenges of rigorous model calibration and validation were thus a determining factor in the choice of pattern-based modeling methodologies reviewed. Best practice for creating a robust counterfactual simulation requires partitioning available data into calibration and validation datasets, so that a full model development cycle can be completed before a counterfactual map is generated. This establishes a minimum data availability threshold of LUC maps from at least four time periods; i.e. calibration based on the first and second maps; validation of LUC predicted from the second map against observed LUC in the third; and comparison of the counterfactual map against observed LUC in the fourth time period. This requirement for historical data can be daunting and may exclude counterfactual modeling as an available tool in particularly data poor settings. Fortunately, the increasing availability of annual LUC remote sensing products (e.g. Fagan et al., 2022; Hansen et al., 2013) is making this more feasible. Increased data availability can improve model performance through multiple calibration-validation iterations at short time intervals, such as used by Assaf et al. (2021) and Bradley et al. (2017), to mitigate the influence of non-stationary driving forces.

The most rigorous calibration-validation process we reviewed was presented by Bradley et al. (2017). Borrowing from developments in climate sciences, ensemble modeling is a promising technique to ensure robust counterfactual simulations by rigorously managing model uncertainty. Ensemble modeling enables the researcher to maximize data use and implement multiple model versions calibrated over different time spans in an effort to capture both fast and slow drivers of LUC (Figure 2). For example, with 10 years of annual LUC maps, one could calibrate separate versions of the same land change model over the last 2, 4, 6, 8, and/or 10 years. Differences in the calibration period will lead to divergent LUCC trend predictions, which will then vary in their accuracy compared to the validation year (Bradley et al.,). With the possibility of calibrating a land change model over multiple time spans, each model version can be inversely weighted by its prediction error, for example using Bayesian network (Abdulkareem et al., 2019) or Bayesian Model Averaging (Hoeting et al., 1999; Hsu et al., 2009) approaches, and combined to create one ‘best’ prediction. Not only does such an approach improve the accuracy of model predictions, but uncertainty is systematically quantified and easily comparable to the magnitude of effect sizes estimated based on differences between observed and counterfactual maps.

Conclusions

What advances are still needed to realize the potential of counterfactual land change modeling for causal inference? This approach is promising but still underutilized and underdeveloped for causal inference in LSS. The best practices we have outlined are a first step to improving the application of land change modeling for counterfactual analysis. In particular, quantifying uncertainty through multiple calibration-validation iterations should be prioritized in order to assess the robustness of any effect sizes estimated by comparing counterfactual and observed outcomes. Additional advances can be made by better targeting and explicitly modeling the drivers or processes producing LUC. One way to do this is to leverage external models of the phenomena driving LUC in combination with rigorously calibrated and validated land changes models. For example, agricultural commodity supply chains connect production regions with distant consumption markets, and thus translate market signals into local land-use and livelihood changes (Liu et al., 2018; Millington et al., 2017). Spatially disaggregated supply chain modeling has advanced considerably (e.g. Godar et al., 2015) but has had little application with spatially explicit land change modeling. Integrating such a modeling approach with counterfactual land change modeling can open new research directions, such as unpacking how disruptions in regional or global supply chains cause local LUCs (e.g. Millington et al., 2017) or how alternative governance arrangements of transnational supply chains (e.g. zero deforestation commitments; Zu Ermgassen et al., 2020) have affected LUC patterns. Such cross-scale interactions, frequently associated with spillover or indirect LUCs, are often invoked as research priorities in LSS (Carlson et al., 2018; Friis et al., 2016; Turner et al., 2020). Counterfactual land change modeling has the potential to provide new insights into the causes of such complex and opaque LUC processes.

Acknowledgments

This work was supported by an award from the NASA Land-Cover Land-Use Change program (#80NSSC21K0297).

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

The work was supported by the National Aeronautics and Space Administration [80NSSC21K0297]

Notes

1. Search terms: (‘land-use change’ OR ‘land system’ OR ‘landscape’) AND (‘model*’ OR ‘simulation’) AND ‘counterfactual’.