Does validity matter for policymakers? Evidence from choice experiments on urban green

ABSTRACT

Stated preference methods such as choice experiments are frequently used for the valuation of environmental goods. Studies suggest that the impact of valuation results on policymaking is rare. How the validity of stated preference results is perceived by policymakers may be a neglected barrier to use in policymaking. The study investigates (1) how valuation results are used by policymakers, (2) how policymakers perceive their validity, and (3) how these perceptions matter for the use of the results. We conduct choice experiments on urban green, directly involving local policymakers in the process. The policymakers, who were interviewed later, report frequent informative use of the results. Although concerns regarding validity exist, they are not a major barrier for informative use but maybe for decisive use. Our findings provide new insights on the use of valuation results by policymakers, as our study is the first to focus on stated preference results and on the role of perceived validity and enables an in-depth analysis by interviewing policymakers involved in a transdisciplinary process. We derive recommendations for researchers on how to design and communicate stated preference studies to increase their use in environmental policy-making.

KEYWORDS:

• Climate adaptation
• decision-making
• economic valuation
• nature-based solutions
• stated preferences
• urban ecosystem services

1. Introduction

Stated preference methods such as choice experiments are frequently used for the economic valuation of environmental goods. In contrast to other valuation methods based on observed behaviour, such as hedonic pricing or the travel cost method, they allow for deriving values for goods that do not currently exist. Also, they allow not only for assessing the use value of environmental good, for example, the benefit people derive from using a park, but also the non-use value, for example, the benefit people derive from knowing that urban green promotes biodiversity. Because of these advantages, stated preference methods may provide valuable additional information for policymakers in a wide range of settings (Hanley and Czajkowski 2019).

However, evidence suggests that economic valuation studies, in general, rarely have a significant impact on policymaking (Gowan, Stephenson, and Shabman 2006; Stephenson and Shabman 2019; Waite et al. 2015; IPBES 2022). Existing studies rarely analyse the use of stated preference results in particular. There has been little research on the role that concerns about the validity of stated preference results, as they are prominent in methodological literature, may play for policymakers’ decisions about their use. If they also perceive stated preference results as invalid, this might be an important barrier to the use of stated preferences in policy-making.

Our study aims to contribute to the yet limited understanding of the use of stated preference results in policy-making and perceived validity as a potential barrier. We carried out choice experiments on urban green in four German cities. Following the choice experiments, we conducted eight semi-structured interviews with local policymakers who had been involved in the transdisciplinary choice experiment process, to investigate (1) how the choice experiments results are used by policymakers, (2) how policymakers perceive the validity of the results, and (3) how these perceptions of validity matter for the use of the choice experiment results. Because the characteristics that lead to validity concerns are similar between choice experiments and other stated preference methods, our study can improve understanding of the role of perceived validity for the use of stated preference results in general. This provides valuable insights for researchers in designing stated preference studies and communicating the results in order to increase their use in environmental policy-making.

Our study is, to our knowledge, the first to exclusively focus on the use of stated preference results by policymakers. The specific characteristics of stated preference methods and the prominent debate on their validity may lead to different uses compared to other valuation methods. Our study connects the methodological literature on the validity of stated preference results and the research on the use of economic valuation results by policymakers by examining concerns about the validity as a potential barrier to their use. Previous studies investigated the role of perceived validity as a barrier to a very limited extent. Marre et al. (2016) and Rogers et al. (2013) find doubts about the validity of economic valuation results in general as a constraining factor for their use. However, their analyses do not allow for identifying the role of the specific validity concerns regarding stated preferences. Rogers et al. (2013) suspect that a lack of familiarity with the method among the policymakers may have caused the responses. By interviewing policymakers who were involved in the transdisciplinary choice experiment process, our research design enables a more in-depth analysis of the role of perceived validity than previous studies and allows for investigating whether validity concerns remain a barrier among policy makers that are familiar with the method.

The remainder of this paper is divided into five sections. Section 2 reviews the literature on the validity of stated preferences, the use of economic valuation by policymakers, and the role of perceived validity for the use of stated preference results. Section 3 describes the research design consisting of the transdisciplinary choice experiment process and the interviews with policymakers. Section 4 reports the interview results, and Section 5 discusses them and concludes.

2. Literature review

2.1. Validity of stated preference results

The willingness to pay estimated in a stated preference survey is valid if it is equal to the true willingness to pay of the target population. This can be evaluated by comparing the stated willingness to pay to predictions from economic theory (construct validity), to results of other valuation methods (convergent validity), to real market transactions (criterion validity) and by focusing on the procedure of the study as a condition for validity (content validity) (Bishop and Boyle 2019).

The question whether stated preference surveys produce valid estimates has been discussed in economic literature for many years. The debate gained momentum after Alaska sued Exxon for the total economic value of lost ecosystems in Prince William Sound after an oil tanker ran aground in 1989 with catastrophic consequences for local flora and fauna (Kling, Phaneuf, and Zhao 2012). During the trial stated preference methods received a lot of attention as the only way to generate monetary value estimates for non-use values (Mariel 2010). The large value estimates sparked many research articles debating the validity of stated preferences, and several guidelines on how to design stated preference surveys that produce valid results have been published since then (Arrow et al. 1993; Johnston et al. 2017; Kanninen 2007; Mariel et al. 2021).

Prominent critics of stated preference methods are Hausman (1996, 2012) and McFadden and Train (2017). They doubt that applications of the method produce valid results. On the one hand, they point to evidence that the results of contingent valuation can be inconsistent with economic theory in several ways (construct validity). On the other hand, they point to numerous laboratory and field experiment studies that have found the willingness to pay to be larger in a hypothetical context compared to real market transactions (criterion validity).

However, a number of authors object to this critique and emphasize the methodological progress (Carson 2012; Kling, Phaneuf, and Zhao 2012; Haab et al. 2013; Bishop and Boyle 2019). Bishop and Boyle (2019) suggest that validity should rather be seen as a matter of degree than as binary and how much validity is sufficient depends on the context of use. They assert that many stated preference studies pass tests for construct, convergent and criterion validity. They argue that individual studies failing these tests raise questions about the validity of these individual studies rather than the method itself.

Criticisms of stated preference methods are frequently based on concerns about hypothetical bias, the divergence between willingness to pay and willingness to accept, scope insensitivity and the warm glow effect (e.g. Hausman 2012). In this study, we focus on all of these concerns except the divergence between willingness to pay and willingness to accept since progress in behavioural economics indicates that this is a property of true preferences rather than an issue of the validity of stated preferences (Kling, Phaneuf, and Zhao 2012). Additionally, we consider potential sample bias. Being of importance in any survey research, a few studies investigate sample bias as a potential threat to validity of stated preferences (e.g. Whitehead et al. 1994; Messonier et al. 2000).

Hypothetical bias describes the phenomenon that people tend to overvalue public goods in the hypothetical context of stated preference surveys. That is, respondents state a larger willingness to pay than they would actually be willing to pay in a real-world situation (Carson 2012). Comparing stated preferences with real payments, a majority of studies found this upward bias in the stated preference results (Kling, Phaneuf, and Zhao 2012). Foster and Burrows (2017) report that the extent of the bias varies greatly, with little explainable pattern. Stated preference research has established best practices to mitigate the effect. In particular, the questionnaire needs to create a realistic and comprehensible conception of the natural good that is valued (Kanninen 2007) and create the perception in respondents that their answers can have consequences on real decisions (Mariel et al. 2021).

Scope insensitivity occurs when respondents’ willingness to pay for a good does not increase adequately as the amount of the good increases (McFadden and Train 2017, 83). A frequently cited example is the study by Desvousges et al. (1993), who found no difference in the willingness to pay for saving 2000, 20,000 or 200,000 migratory birds. Scope insensitivity is closely related to the embedding effect, where the willingness to pay for a good is lower when valued as part of a package (Venkatachalam 2004). As a response, the literature suggests using tests for scope insensitivity. However, there is disagreement about which test is preferable and whether stated preference studies usually pass it (Carson 2012; Desvousges, Mathews, and Train 2012). It is found that the sensitivity to scope can be improved with images or maps for better visualization (Carson 2012) and the use of absolute rather than relative measures (Haab et al. 2013).

Warm glow describes the good feeling that might arise from giving money for a good cause per se – beyond the actual value of the good (Kling, Phaneuf, and Zhao 2012). Hausman (1996) sees this ‘pleasure of giving’ as a threat to the validity of stated preferences because the estimated willingness to pay based on a warm glow does not reflect true preferences for the good. It could be one possible cause for scope insensitivity or hypothetical bias (Hausman 1996). Bishop (2018), however, claims that Hausman and others confuse the warm glow effect, which is part of true preferences and thus not a threat to validity, with a different concept of ‘good feelings’, which lacks a theoretical foundation and empirical support.

A sample bias can occur in a survey if respondents are not representative of the target population. A likely cause might be self-selection or non-response by participants. For example, individuals who are more interested in the survey topic respond to the survey with greater probability, while also having different preferences than the general population (Messonier et al. 2000). Consequently, estimated willingness to pay would be different from the target population’s true preferences. To mitigate this bias, guidelines recommend to collect and assess the sociodemographic information of the respondents and let the survey topic appear more general to reduce self-selection (Johnston et al. 2017; Mariel et al. 2021).

2.2. Use of economic valuation results in policymaking

Research on the use of stated preference methods, or predominantly of economic valuation in general, for policymaking has gained increased attention only in recent years. The growing body of academic literature that scrutinizes the actual use and influence of economic valuation in real-life policy shows ambiguous results. Overall, however, it suggests that the contribution to policymaking is limited and far behind the potential use (Laurans et al. 2013; Marre et al. 2016, 2015; Rogers et al. 2013). In its Assessment Report on Diverse Values and Valuation of Nature, the Intergovernmental Science-Policy Platform on Biodiversity and Ecosystem Services (IPBES 2022) declares the uptake of valuation results in decision-making as one of the most pressing issues and suggests a more extensive documentation of uptake as well as barriers and enablers.

An early study by Gowan, Stephenson, and Shabman (2006) examines the contribution of benefit estimates on public policy-making and finds no evidence that monetary values are useful in the context of hydropower relicensing, findings that were corroborated later by Stephenson and Shabman (2019). Rogers et al. (2013) surveyed policymakers as well as researchers about the impact of non-market valuation approaches in environmental policy and management. They find a discrepancy between researchers’ perceptions of the usefulness of their results and the actual use by policymakers. This is supported by Waite et al. (2015), who find that, while there is some use of valuation in coastal policy-making, the majority of valuation studies did not have a clear influence. Findings of Primmer, Saarikoski, and Vatn (2018) suggest that perceptions of the usefulness of valuation results vary between policy actors: Those who operate on a local level turned out to be more skeptical about an operational use of valuation results than national level policy actors.

Bergstrom and Loomis (2017) distinguish between the type and scope of use and the influence of valuation results. They state that the role of non-market valuation in river restoration contexts is predominantly to provide more information. Thus, taking a closer look on use categories might reveal a more differentiated picture. The influential review by Laurans et al. (2013) describes the main categories of economic valuation use as (1) decisive, (2) technical or (3) informative. (1) A decisive use is a direct and ex-ante influence of the valuation estimates on a specific decision, which can happen in three different ways. First, the valuation results can be integrated into a cost–benefit analysis by assessing trade-offs of projects. Second, the integration may be used to help prioritise different options given a limited budget. Third, valuation can be used in a participative manner, serving as a basis for discussion about valuation parameters and assumptions while negotiating projects. (2) When the valuation is used to adjust the chosen policy instrument after a decision took place, Laurans et al. categorize it as a technical use, for the purpose of either price setting or damage compensation levels. (3) If the use is considered as informative, the valuation contributes mainly indirectly to the decision-making process. The first subtype of informative use is awareness-raising, meaning that the valuation estimates are used to change the attitude toward conservation measures. Second, as justification, stakeholders use valuation to support and justify their positions. Thirdly, valuation can be used informatively for accounting purposes.

Laurans et al. find that only a few out of several hundred valuation studies address the use of valuation results more extensively than a cursory reference. Among these, the most frequent type of use is enabling policymakers to make trade-offs, followed by justifying decisions, awareness-raising and prioritization. The classification by Laurans et al. is commonly used by research on the use of economic valuation in environmental policy and planning (Forkink 2019; Primmer, Saarikoski, and Vatn 2018; Rogers et al. 2013; Marre et al. 2016; Bergstrom and Loomis 2017). Others propose alternative classifications of the types of use (e.g. Tinch et al. 2019; Beery et al. 2016; Mouter 2017; Vargas, Sarmiento Erazo, and Diaz 2020). All classifications similarly distinguish between informing the policy-making process (by demonstrating value, informing planning, or providing strategic or conceptual knowledge) and decisively determining the final decision (which is what some studies call instrumental use).

Current empirical research challenges the earlier assessment that valuation results have little influence by demonstrating their usefulness as informative use. According to Forkink (2019) and Marre et al. (2015), economic valuation appears to contribute to policymaking ‘ … as a basis for discussion in decision-making processes, and for communication and advocacy’ (Marre et al. 2015, 26). Findings by Mouter (2017) also suggest that policymakers do not base their decisions solely on valuation results but use the information in the process of forming their opinion on policy options. Overall, however, the results are still ambiguous. Valuation results are rarely ignored but also rarely have a significant impact on policy decisions (Marre et al. 2016).

The reviewed studies do not differentiate between economic valuation methods explicitly. To our knowledge, no study exclusively focuses on the use of stated preference results. Rogers et al. (2013) refer in their empirical study to stated as well as revealed preferences, while others do not address specifically which valuation methods are used (Forkink 2019; Marre et al. 2016; Primmer, Saarikoski, and Vatn 2018). Although Bergstrom and Loomis (2017) mainly integrate stated preference methods in their meta-analysis of river restoration valuations, they do not ask policymakers about the actual use of existing estimates.

2.3. The role of perceived validity for the use of stated preference results

As illustrated above, the economic valuation of non-market goods is not used extensively in evidence-based environmental policymaking. Given this, some studies analyse barriers that might hinder a more extensive integration in practice. Empirical studies reveal that conceptual issues appear to be an important barrier to use. According to Dehnhardt (2013), German water policymakers generally question the need of economic valuation in environmental policymaking because they perceive the complexity of ecological interlinkages as too high, but also raise moral concerns about putting a monetary value on nature (constraints that are also emphasized by Rogers et al. 2013 and Marre et al. 2016). Further important barriers are the lack of knowledge and understanding of the method and high expenditure of costs, time and effort for valuation approaches (Rogers et al. 2013; Dehnhardt 2013; Laurans et al. 2013; Forkink 2019). Additionally, formal institutions, i.e. existing rules and regulations, can hamper the integration (Dehnhardt 2013; Marre et al. 2016) as well as interests of policymakers that prevent decisions from being based on explicit cost–benefit considerations (Dehnhardt 2013; Mouter 2017).

Also, methodological concerns appear to be an important constraining factor for the use of valuation, including doubts about the validity of valuation results (Marre et al. 2016; Rogers et al. 2013). The role of perceived validity as a barrier for valuation use is investigated to a very limited extent. Marre et al. (2016) surveyed 88 policymakers about perceived limits to the use of economic valuation. 64% of respondents recognise a lack of validity as a limit, indicating that it is considered as an important barrier for extended use. However, Marre et al. suspect that a lack of familiarity with the method among the policymakers could be a reason for the responses. Rogers et al. (2013) found that researchers perceive validity concerns to be one of the two most important barriers (next to rejecting monetary values for the environment), but policymakers only agree to some extent with that statement. They conclude that researchers overestimate the role of validity issues since policymakers often do not have a sufficiently in-depth insight into the method to raise such concerns. The small number of studies that investigate perceived validity shows that it has not yet achieved the same prominence in the analysis of the use of results as validity has in the methodological debate on stated preferences.

3. Research design and methodology

Our empirical approach consists of four choice experiments about the extension of urban green that were developed in a transdisciplinary process in four German cities and semi-structured interviews about the use and perceived validity of the results that were subsequently conducted with policymakers from the cities.

3.1. Transdisciplinary choice experiment process

In each of the four German cities Augsburg, Bremen, Karlsruhe and Leipzig, a choice experiment was developed in collaboration with local policy-makers from city administration departments responsible for decisions and planning regarding urban green. Researchers and policymakers discussed the questionnaire, including the definitions of attributes and levels in the choice experiment, in two to three joint workshops in each city. The workshops enabled learning about the choice experiment method for the policymakers and transdisciplinary discussion on questionnaires and definitions. The four choice experiments were conducted from summer 2018 to spring 2019 (see Table 1 for details).

Table 1. Timing and design of the choice experiments.

City	Field phase of survey	Delivery of results	Number of respondents	Attributes
				Street trees	Green spaces	Green roofs	Near-natural spaces	Green-ways
Augsburg	July–Nov 18	Jan 19	559	x	x		x	x
Bremen	April–May 19	May 19	1276	x	x	x
Karlsruhe	July–Nov 18	Jan 19	479	x	x		x	x
Leipzig	July–Aug 18	Sep 18	1130	x	x		x	x

The choice experiments presented citywide scenarios of extending urban green, characterized by several urban green attributes as shown in Table 2 and a cost attribute defined as a compulsory yearly payment that the city would collect and spend exclusively on the development and maintenance of the urban green. In all cities, the final attributes were jointly chosen in the workshops from a larger set of potential attributes provided by the researchers. The respondents received descriptions of all attributes in the questionnaire, which were revised based on comments from the policymakers. Images and maps were used to support respondents’ understanding. The aims of this process were to ensure policymakers understand the choice experiment in their city and to improve the questionnaire based on their practical knowledge.

Table 2. Attributes and levels in the choice experiments.

Attribute	Description	Levels	Augsburg	Bremen	Karlsruhe	Leipzig
Street trees	Average number of trees per 100	as today 2 trees more 4 trees more	X	X	X	X
		as today 1 tree more 2 trees more
Green Areas	Share of green spaces of the total	as today 5 percentage points more 10 percentage points more	X	X	X	X
		as today 1 percentage point more 2 percentage points more
Near natural Spaces	Share of a city’s green spaces that are near-natural (i.e. mowed at most twice a year)	as today 10 percentage points more 20 percentage points more	X		X	X
Greenways	Share of pedestrian and cycling ways that are adjacent to lawns and bushes	as today 10 percentage points more 20 percentage points more	X		X	X
Extensive green roofs	Share of roofs that are extensively greened (i.e. those with thin substratum)	as today An increase by 5 of 100 roofs An increase of 10 by 100 roofs		X
Intensive green roofs	Share of roofs that are intensively greened (i.e. those with thick substratum)	as today An increase by 1 of 100 roofs An increase by 2 of 100 roofs		X
Cost	Compulsory yearly payment per individual (in Euro)	0; 5; 10; 50; 200; 300/400	X	X	X	X

The final surveys were implemented online by professional polling agencies between July 2018 and May 2019 with samples of between 479 and 1276 respondents per city. In a workshop in September 2018, final results were presented to policymakers from Leipzig and preliminary results to policymakers from Augsburg and Karlsruhe. All three cities received written reports of the final results in January 2019. In Bremen, preliminary results were presented in a workshop in May 2019 and delivered in the form of a draft fact sheet in April 2020. The results of the choice experiments are described by (Welling et al., 2022).

3.2. Interviews with policymakers

We conducted semi-structured interviews with eight policymakers from administrations of the four cities between April and June 2020. Two main criteria were used for the selection of interview partners. First, they needed to be involved in the choice experiment process in their city and have participated in the workshops during questionnaire development. This ensures some knowledge about the method and the application in the city, enabling more detailed discussion about the results and their perceived validity. Second, the interviewees needed to be decision-makers regarding the provision of urban green in the city, using the definition from Rogers et al. (2013, 5) as actors who are ‘either involved in making decisions themselves or who provided advice to higher-level decision-makers’. In Augsburg, Karlsruhe and Leipzig, we contacted two potential interviewees from the city administration department responsible for urban green in each city. In Bremen, we contacted two interviewees from departments responsible for urban green and two from departments responsible for water management because the effects of urban green on rainwater management were a focus of this research project. Of these 10 contacted persons, 8 accepted to be interviewed, while one from Karlsruhe and one Leipzig refused due to lack of time. Contacting alternative interview partners in these cities was not possible due to the strict selection criteria.

The first section of the questionnaire inquired about the use of the choice experiment results in the city. Interviewees were first asked a general open question on how the results had been used. This was followed by questions on specific types of use based on Laurans et al. (2013). The interviewees were only asked about the two types of decisive use (trade-off, prioritization) and two types of informative use (awareness-raising, justification), because they are the most relevant to our case study since the choice experiments primarily provide an ex-ante analysis of different policy options pertinent to decisive and informative rather than technical use. The second section of the questionnaire inquired about the interviewees’ perceptions of the validity of the results, again using a general open question followed by specific questions operationalizing the four potential validity concerns described in section 2.1. The third section asked about the role of perceived validity for the use of the results. Participants got a summary version of the questionnaire, holding only the general questions and not the specific ones based on categories from the literature, in advance to enable them preparing their answers. An English translation of the questionnaire is available in Table 3 in the appendix.

Table 3. Questionnaire, translated from German by the authors. Respondents received the questions with grey background in advance for preparation.

Interview Question	Corresponding Category
Section 0: Background
Before we start talking about the choice experiment, could you tell me something about your position as well as your current and previous duties and responsibilities?
Before this research project, have you already got in touch with methods of economic valuation, cost-benefits analysis or choice experiments?
Section 1: Use
Have the results of the choice experiment already been used in the city in any way?
How do you think the results will be used in the future?
Do you think results will be used to prioritise between different measures?	Prioritization
Do you think the results will be used to trade off costs and benefits of a measure?	Trade-off
Do you think results will be used to raise awareness of the public or actors in politics and administration for the benefits of a measure?	Awareness-raising
Do you think results will be used to justify policy decisions?	Justification
Section 2: Perceived Validity
Do you think that the average willingness to pay estimated by the choice experiment equals the true willingness to pay of the population in your city? If not, what are the reasons?
Do you think that the willingness to pay of the survey sample deviates from the willingness to pay of the population of the whole city?	Sample bias
Do you think that respondents stated a different willingness to pay because their choice responses are hypothetical?	Hypothetical bias
Do you think that the respondents exaggerated their willingness to pay because they want to feel good by giving morally right responses?	Warm glow
Do you think that respondents understood the scope of the measure? For example, if we had proposed 5.000 instead of 10.000 additional trees in their city, would they state a roughly half as large willingness to pay?	Scope insensitivity
Are there further reasons why you think the willingness to pay estimates of the choice experiment might not match the true willingness to pay of the population?
Section 3: Perceived Validity as a potential barrier
Are such concerns a barrier for the use of the choice experiment results in your city?
If you could trust in the estimated willingness to pay being equal to the real willingness to pay of the city population, would you see the possibility for additional or different types of use in your city?

All interviews were conducted via phone or video conference software. The full interviews were recorded and later transcribed with the approval of the interviewees. The transcripts were coded in MAXQDA 2020. The codes were based on the categories from the literature described in Section 2, while being open for new categories that had not been identified in the literature. In an iterative coding process, some categories were further differentiated.

4. Interview results

4.1. Use of the choice experiment results

Three interviewees from all cities except Bremen reported that they had already used the results of the choice experiments. The four interviewees from Bremen had not used the results yet, though saw a potential for some types of use in the future.

All eight interviewees reported or considered an informative use of the choice experiment. When asked generally whether the results were already used in any way, three interviewees from the three cities except Bremen reported awareness-raising about the benefits of urban green in their city (Interview 5, 6, 7). All interviewees from Bremen could imagine using the results for awareness-raising in the future when explicitly asked (Interview 1, 2, 3, 4). It was frequently stated that the willingness to pay estimates are a valuable additional argument when communicating the value of urban green.

I think this willingness to pay is an approximation and supports us as a new argument that we can use if needed for communicating with target groups.’¹ (Interview 7)

The most frequently reported target of awareness-raising was other policymakers. Two interviewees reported that they had presented the results in committees of the city council (Interview 5, 6). A further four saw this as a promising potential future use (Interview 2, 3, 4, 7). They emphasized that the citizens’ valuation was very relevant for politicians and that such ‘hard numbers are important when communicating within politics’ (Interview 3). Beyond policymakers as a target, one interviewee had already presented the results in events with citizens (Interview 7).

The other type of informative use, justifying positions or decisions with the results of the choice experiment, was already practiced by two and considered for the future by a further five out of eight interviewees. Interviewee 5 reported that she had already used the willingness to pay values in an a priori justification (that is, as an argument for their position before a decision is made), as arguments in budget negotiations within the city and in responses to inquiries by the city council about the potential conversion of green spaces into building land. Interviewee 7 had used the results to legitimise decisions towards the citizens (a posteriori justification). The willingness to pay values for street trees had been communicated as one of several arguments in press releases and on the city’s website when new trees are planted on former parking spots. Five of the remaining six interviewees considered a use as an a priori justification in the future (Interview 1, 2, 3, 4, 6), for example, in future planning processes of new building projects (Interview 1). It was emphasized that in these cases the willingness to pay of the citizens only serves as one among several arguments. Interviewee 1 also expected the results to be used in Bremen as an a posteriori justification to support urban green measures in the city’s climate change adaptation strategy that had already been adopted before the choice experiments.

[We often get] inquiries from citizens or the city council [ … ], in these cases we can integrate it as arguments. [ … ] Arguments may include the provision of the surrounding population with green spaces [ … ], and then with the willingness to pay one can argue how important such green spaces can be for the people. (Interview 5)

In contrast, decisive use of the results plays a minor role. Even after being asked, no interviewee reported a past or potential future use for assessing the trade-offs between costs and benefits of the urban green measures. One interviewee gives as a reason that social costs and benefits are traded off in the political process instead and ‘because the benefits [as estimated in the choice experiment] pertain to society as a whole and not necessarily to the bearer of the costs’ (Interview 4).

The prioritization between several measures has also not been a way of use yet. Two of the interviewees from Bremen saw a potential to prioritise between the urban green measures within the available budget in the future (Interview, 2, 4).

If one looks at it in the context of the climate adaption strategy and asks: Where should the political focus be? What political measures should be prioritised? I would of course consider those with the largest benefit and the lowest costs. (Interview 4)

In Augsburg, Karlsruhe and Leipzig, interviewees did not expect the results to be used for prioritizing the urban green measures. One interviewee stated as a reason that the citizens answering the choice experiment might not have had sufficient knowledge about the impacts and local context dependence of the green measures, so that policymakers will rely on the expertise of city administration specialists rather than on the willingness to pay estimates (Interview 5).

4.2. Perceived validity of the choice experiment results

Five of eight interviewees generally perceive the willingness to pay derived from the choice experiment to be equal to the true willingness to pay of the population. Three were unsure or critical, of which two immediately expressed concerns related to the city population’s heterogeneity and potential sample bias. No other concerns were voiced without being specifically asked. When directly inquiring about hypothetical bias, scope insensitivity, warm glow and sampling bias, for each some interviewees expressed concerns and some others did not believe it is a threat to the validity of the results. No interviewee stated specific concerns beyond the four issues addressed.

When asked whether they believe that the estimated willingness to pay might deviate from the true willingness to pay because of the hypothetical choice situation, the perceptions of the interviewees were mixed. Three believed that there was no hypothetical bias (Interview 4, 5, 6), four supposed that the true willingness to pay deviated (Interview 2, 3, 7, 8), and one interviewee believed that it deviated only for measures that do not directly affect the citizens (Interview 1). In both cities where more than one person was interviewed, there was no agreement about a potential hypothetical bias. To explain her belief in a hypothetical bias, one interviewee described a lack of awareness for the importance of urban green among many people. She believed that the true willingness to pay would be as large as in the choice experiment only if people would intensely engage with information, as had been the case in the choice experiment (Interview 8). Two other interviewees believed that the willingness to pay might be different from the choice experiment if politics actually established a cost that was mandatory to pay (Interview 2, 3).

If it is not binding, that is it will not directly affect the wallet, then people are quite willing: Yes, I want urban green, I want more green spaces, I want more trees, I want, I want, I want. However, I can imagine that the willingness to actually pay might later be different. (Interview 2)

When asked whether they believe respondents were sensitive to the scope of additional urban green, only one interviewee unconditionally said so (interview 6) and two believed they were insensitive (interview 4, 8). Two interviewees from different cities supposed respondents are sensitive to the number of street trees, but less so for green measures that are harder to imagine or less part of everyday lives like green roofs (Interview 3, 5). Another interviewee expected respondents only to be sensitive if they had been provided with extensive information ‘about which current stock of trees there is, about which potential, that is which additional possibilities there are’ for extending the number of trees (Interview 2). One interviewee believed that some respondents who are good with numbers might have been fully sensitive to the extent of additional urban green, but some respondents who are worse with numbers might have been less sensitive and would have required more images (Interview 7).

When asked whether they believe respondents stated larger than their true willingness to pay because of the warm glow they derived from a morally desirable answer, three interviewees did not believe this had been the case (Interview 3, 5, 6), while five think it might had been a bias in the respondents’ choices (Interview 1, 2, 4, 7, 8). One interviewee compared the choice situation to being intercepted on the street by a fundraiser for a social project (Interview 1). Another interviewee who was less concerned thought the alternatives consisting of combined attributes in the choice experiment made it ‘difficult to recognise which choice alternative is the most politically correct’ and therefore to ‘consciously choose the expensive alternative’ because of warm glow (interview 6).

Notably, two interviewees already expressed concerns about sample bias before explicitly asked (Interview 2, 4). When the other six were asked whether they believed the willingness to pay of the survey participants was equal to the willingness to pay of the general population of the city, four of the interviewees agreed and two additional interviewees expressed concern. One of the initially doubtful interviewees concluded later that he believed sampling was not an issue, when he remembered a map of the survey participants’ residence locations widely distributed across the city that was shown in an earlier workshop (Interview 2). Several interviewees who expressed concern believed that there was a socioeconomically disadvantaged share of the population who did not participate in such surveys and had no ability or willingness to pay. In consequence, well-educated citizens with a stronger affinity to urban green might have been overrepresented and the willingness to pay of the general population overestimated (Interviews 1, 4, 7).

We know the people who take part in those surveys. It is always the usual suspects. Thus, I would say, it does not necessarily reflect the opinion of the whole population. You can’t extrapolate to the whole city. I think, people who are distant to this topic did not participate and do not have a willingness to pay or possibility to pay. (Interview 7)

4.3. The role of perceived validity for the use of the results

When directly asked whether concerns about the validity of the results are a barrier for its use, only one of the eight interviewees believed so. She explained that some policymakers who demand decisions to be based on hard numbers would not accept results from choice experiments. When one could rely on the validity of the numbers, they could be used more as a hard fact (Interview 1).

I believe, you cannot use it like a statistic or traffic count data. It is not comparable to these hard facts. [ … ] And if it were clear, that everyone actually had the willingness to pay, then we could bring it as an argument like traffic count data. (Interview 1)

However, even the three interviewees most skeptical about the validity of the results did not see this as a significant barrier (Interview 2, 4, 7). Other barriers like resistance to change or fundamental skepticism towards putting a price on urban green were reported as more hindering for the use of choice experiment results (Interviews 7). They emphasized that the willingness to pay values can still serve in informative use as a vague measure of the citizen’s appreciation of urban green, even if the numbers are ‘fictive’ (Interview 2) or ‘hypothetical’ (Interview 4).

The willingness to pay exists in principle, so the topic is important for the people and they expressed this with a monetary value. I believe, one can look at this separately. Such a hypothetical willingness to pay expresses a certain importance for the individual, separated from the question whether one actually wants to pay this from the own wallet. (Interview 4)

Comparing the reported use and perceived validity across the different interviewees and cities does not indicate a major role of perceived validity either: The most extensive use of the results was reported by one of the three interviewees who was concerned about the validity (Interview 7).

5. Discussion and conclusion

We developed and conducted four choice experiments on urban green in a transdisciplinary process involving local policymakers from city administration departments responsible for planning and decisions regarding urban green in the four German cities Augsburg, Bremen, Karlsruhe and Leipzig. Subsequently, we conducted semi-structured interviews with eight policymakers involved in the choice experiment process. The interviews inquired about the use of the choice experiment results in the cities, perceptions of the validity of the results, and the role of perceived validity for the use of the results.

Policymakers from three of the four cities had already used the choice experiments results. A reason why the results had not been used in the fourth city, Bremen, may be that policymakers had received first results in written form only a few weeks before the interviews, compared to one year in the other cities. Predominantly informative use was reported, in particular awareness raising about the benefits of urban green towards city politics and the justification of positions and decisions. Given scarce financial resources, policymakers need to justify the high expenditures for climate change mitigation and adaptation measures. Stated preference approaches that directly ask people about their preferences and prove their appreciation of urban green may support this justification. No decisive use in any city was reported, only some interviewees saw the potential for future use in prioritization. These results are in line with previous studies by Waite et al. (2015), Forkink (2017) and Bergstrom and Loomis (2017), as they show predominant informative use of the estimates of economic valuation in general.

Notably, in our study, local policymakers report past and potential use of the results. This is in contrast to the findings of Primmer, Saarikoski, and Vatn (2018) that mainly national policyactors perceive economic valuation to be useful. A reason might be that the perceived usefulness and use of estimates differs less between local and national levels of decision-making, but more between the operational and strategic levels of the valuation. In our case study, the urban green measures are city-wide without a detailed spatial distribution and are, therefore, relevant to strategic planning. The lack of a spatial specification may also be a further reason why the results are not used decisively. Local operational decisions on urban green in municipal formal planning processes require a detailed specification of the measures. Valuation results may have to reflect the planning context precisely to be used decisively, regardless of the perceived validity. This is challenging in scenario-based choice experiments that are difficult to adapt to precise action plans and specific measures, as these often too complex to be reflected in a limited number of attributes.

The policymakers had mixed perceptions about the validity of the results. When asked generally, most interviewees perceived that the estimated willingness to pay equals the true willingness to pay of the city population. However, a few stated concerns about sample bias related to the socio-economic heterogeneity in their cities. This initial impression before the more specific interview questions is likely the most relevant for the policymakers’ decisions. It shows how they thought about the results before having a deeper discussion about potential threats to validity in the specific context of the interviews. Only when asked explicitly about hypothetical bias, warm glow and scope insensitivity, which are more prominent subjects of stated preference literature (e.g. Hausman 2012), some of the interviewees agreed that this might have caused a deviation between estimated and true willingness to pay. One explanation for this may be the conclusion by Rogers et al. (2013) that policymakers’ knowledge of economic valuation methods is too limited to have an independent opinion about its validity, even after being involved in the transdisciplinary choice experiment process of this study. However, this seems not to be the case for at least some interviewees who gave unprompted responses.

Although sample bias was the major unprompted concern of the interviewed policymakers, there is only little stated preference-specific literature on sample bias (e.g. Messonier et al. 2000). If these concerns of the interviewees extend more generally to policymakers considering the use of stated preference results, researchers might be able to improve the perceived validity by paying particular attention to sampling, but also by careful communication. The policymakers were mainly concerned that the estimated willingness to pay for urban green of the survey sample cannot capture the socio-demographic heterogeneity of the city population. This impression might arise when only average estimates are communicated. Assessing and communicating the heterogeneity of preferences between different socio-demographic groups within the sample might reduce this impression. Also, communicating the spatial heterogeneity of the respondents and their preferences might help, as is indicated by one interviewee whose concerns about sample bias faded after recalling a city map with the scattered places of residence of participants.

The interviews do not indicate that perceived invalidity was a major barrier for the use of the results. Most interviewees either believed the results were valid or stated that their concerns about validity did not discourage them from using the results as an informative tool. The most extensive use of the results was reported by one of the interviewees who were concerned about their validity. This differs from the finding by Marre et al. (2016) that two-thirds of surveyed decision-makers identify a lack of validity as a limit for the extended use of economic valuation results. However, it is in line with the different conclusion by Rogers et al. (2013) that researchers overestimate the role that validity plays for policymakers. Notably, several interviewees stated that the validity of results might be limited, but that this mattered less for the informative use of results than for decisive use. The interviewed policymakers used the results as one of several arguments in the informative uses of awareness-raising or justification and considered it sufficient for this purpose to know the approximate magnitude of people’s valuation or even only that a willingness to pay exists in principle.

In the literature on the use of economic valuation, methodological concerns are found to be a barrier, but perceived validity is only one part of this, among others (Dehnhardt 2013; Rogers et al. 2013). Also other barriers, especially for decisive use, might be important. The matching of valuation results with the data and information requirements of formal planning processes might be an important prerequisite for decisive use in policymaking at a local level. Rogers et al. (2013) and Marre et al. (2016) state that the policymaking framework and the nature of valuation information (monetary values for environmental goods and services are not yet part of formal planning instruments) are not conducive to the use of valuation results. Another major constraint not discussed here is a general rejection of putting a monetary value on nature (Dehnhardt 2013). Marre et al. (2016) found that conceptual concerns, such as the lack of reflecting on the complexity of ecosystems, further limit the use. Finally, the lack of familiarity with and knowledge about the method has been identified as a major barrier (Rogers et al. 2013). Our study addressed this by involving the interviewed policymakers in the process of designing and conducting choice experiments, which may be an explanation for the more frequent informative use than usually found in the literature.

We acknowledge some limitations of our analysis. We selected only policymakers as interviewees who were involved in the choice experiment processes, to ensure more knowledge about and familiarity with the method than is usually the case with less specialized local policymakers. While enabling more in-depth interviews about the validity of choice experiments and its role in the use of results than in previous studies, this approach also has potential downsides. First, involvement in the process as a selection criterion limited the number of potential interviewees. Future studies with larger sample sizes could inform on the robustness of our conclusions. Second, the approach creates the risk of selection bias. Policymakers who are a priori less sympathetic towards stated preference methods might be less likely to get involved in a choice experiment process and, therefore, to be selected for our interviews. Third, the use of the results and perceptions of their validity by the policymakers involved in the choice experiment processes might differ from policymakers who have not yet acquired similar familiarity with and knowledge about choice experiments. Thus, generalizing the findings to other groups of policymakers introduces new uncertainties. Further, the interviews were conducted by the researchers responsible for the choice experiments. This made a detailed discussion about perceptions and concerns regarding the choice experiments possible, but might also have caused some interviewees to underreport a critical perspective.

Given our findings that perceived invalidity seems not to be a major barrier to the use of stated preference results in spite of the prominent methodological debate, it would be interesting to see further studies on how this compares to the role of perceived validity for other non-market valuation methods. Our results also indicate that the role of perceived validity differs across various use types. Policymakers remain skeptical about using stated preference valuation results in a decisive way, while the perceived validity of the results seems less significant for an informative use. Further research may provide interesting insights by investigating differences in the usefulness, the use and barriers to the use of economic valuation results in different decision contexts and between different use types. The concerns we found about population heterogeneity and potential sample bias give rise to a further recommendation for future research. How to model heterogeneity among respondents is a frequently considered question in methodological stated preference research. To increase the potential for future use of stated preference results in policymaking, it may be helpful to assign a larger weight in this research to models of heterogeneity that can be adequately integrated in value assessments and can easily be communicated to and understood by policymakers to address existing concerns as those we found.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by Bundesministerium für Bildung und Forschung [grant number BREsilient (FKZ: 01LR1723B, 01 LR2013B), Stadtgrün Wertschätzen (FKZ 01UR1621A)].

Notes

1 All quotations have been translated from German by the authors.