If deliberation is the answer, what is the question? Objectives and evaluation of public participation and engagement in science and technology

ABSTRACT

Public participation and engagement in decision-making regarding science and technology (‘PP&E’) is an increasingly common practice. But what is known of whether PP&E achieves its goals? Surprisingly, little research evaluates PP&E. We put forth three reasons why PP&E advocates and practitioners should take evaluation seriously: the absence of evaluation causes PP&E's advocacy to fail a minimal burden-of-proof standard; PP&E's costs are greater than they appear; and these costs may be disproportionately borne by the already-disadvantaged. Evaluating PP&E would require identifying PP&E's objectives and assessing its success in meeting them. To this end we survey scholarship advocating PP&E and identify three sets of objectives: substantively improving decision-making, deontologically fulfilling widely-held norms, and politically redistributing power away from techno-scientific elites. While there is some ad hoc evidence of progress toward these goals, we find no robust evaluation of PP&E. We offer four recommendations that might assist in evaluating PP&E more thoroughly.

KEYWORDS:

• Public engagement
• public participation
• objectives
• Science and Technology Studies
• decision-making

Introduction

Calls for scientists and other researchers to engage with the public appear to be increasing. Three recent cases exemplify this trend. First, in response to actual and potential resistance to uptake, some academics requested deep public engagement in the development and distribution of COVID-19 vaccines (Working Group on Readying Populations for COVID-19 Vaccines 2020). To allay such concerns, the French government formed a participatory citizens’ collective to help guide the French vaccination program (Elzas 2021). Second, the team planning a low-impact, outdoor solar geoengineering experiment established an advisory committee that intends to ‘engag[e]and gather[] input from members of the global public who reside outside of the region of the experiment’ (SCoPEx Advisory Committee 2021). This is just one example of how solar geoengineering field research has been blocked – not by arguments that directly engage with the merits of the research – but by calls for wide, deep, upstream engagement. And in a recent Science Policy Forum, twenty-five scholars led by John Dryzek call for ‘global public deliberation to explore the science [of human genome editing] and its implications, beginning with a global citizens’ assembly’ (Dryzek et al. 2020, 1435), which would be a novel undertaking.

These are but a few examples of ‘public participation’, ‘deliberation’, and ‘engagement’ efforts, demands for which have grown in parallel with recognition of the importance of responsibility in science, technology, and innovation. While the Responsible (Research and) Innovation (RI or RRI) community has a wide range of descriptors for these efforts, from fostering ‘inclusive governance’ (Macnaghten and Guivant 2020) to ‘extracting public values’ (Capurro et al. 2015), the field places significant weight on the importance of increasing collective engagement with innovation. For example, Shannon Conley and Emily York write that ‘while approaches to public engagement and deliberative democracy as means for cultivating responsible innovation have faced legitimate scrutiny and critique, the need for collective responsibility in innovation and governance more broadly is dire’ (Conley and York 2020, S1). Indeed, some authors have even laid out such efforts as necessary for RI, arguing that ‘deliberation [is] a condition for RRI [and] a prerequisite not only in cases of conflict but also in the presence of uncertainties’ (Reber 2018, 58).

Growing recognition of, demands for, and practice of public participation and engagement in decision-making regarding science and technology (henceforth ‘PP&E’) may mark a substantial shift in the norms governing scientific research and technological development. By this abbreviated phrase, we mean a subset of what Rowe and Frewer call ‘public engagement’: ‘the practice of involving members of the public in the agenda setting, decision-making, and policy-forming activities of organizations / institutions responsible for policy development’ (Rowe and Frewer 2005, 253). Note that, in this definition, public engagement is instigated by decision-makers or researchers and does not include public-instigated ‘bottom-up’ activities, such as protests or citizen-science, which we briefly touch on separately in this paper. We also use ‘PP&E’ to indicate only those public engagement mechanisms that, again following Rowe and Frewer, have an ‘open response mode’ in which public participants are not restricted to a finite list of responses from which to choose (Rowe and Frewer 2005, 269), as opposed to traditional public surveying or opinion polling. As such, we take PP&E to include mechanisms such as focus groups, citizen panels, citizens’ juries, and deliberative opinion polling.

The expansion in the demands for PP&E target a wide range of public, quasi-public, and private institutions, to the point that ‘it is widely argued that citizen participation should occur at every stage of governance, from problem identification to resolution and review’ (Cook et al. 2013, 756). Moreover, its advocates claim that ‘participatory approaches enjoy remarkable support from governments, funding agencies, regulatory bodies and ethics commissions’ (Braun and Könninger 2018, 674–675). PP&E’s promise of improving decision-making and enhancing legitimacy by harnessing the problem-solving intelligence of collective agency (Landemore 2017, 288; Macdonald and Macdonald 2020) has seen it grow into an emerging industry (Lee 2014). Nevertheless, the proportion of scientific research that is scrutinised through PP&E exercises remains small.

This limited scrutiny is important as, despite its potential benefits, PP&E also comes with costs: financial investment, opportunity costs, delayed and foregone research, unacquired knowledge and – according to some critics of PP&E – the potential to legitimate unjust technological choices. Of course, these costs may be outweighed by PP&E’s benefits. But whether benefits or costs are greater remains unknown. In order to advance understanding of PP&E and help ensure its positive societal contribution, this paper focuses on the benefits side of this balance, exploring PPE’s objectives and the extent to which they have been met. Perhaps surprisingly, this is a largely novel set of questions as PP&E has rarely been subject to such critical scrutiny in the published literature (see Low and Buck 2020). For example, Paul Cairney points to ‘a tendency for studies to experiment with participatory processes, but struggle to measure, or find evidence of, their effectiveness’ (2016, 101).

The article develops in six successive parts. The first briefly traces the origins and history of PP&E. In order to facilitate evaluation of PP&E against its stated objectives, the next three sections characterise its three top-level sets of objectives, which we group as substantive, normative and political. The article then describes some deeper problems with the political objectives as well as opportunities and challenges to advancing the substantive and normative ones.

Public participation and engagement and Science and Technology Studies

As the foundational tenet of democracy, public participation in decision-making has a history at least as old as Athenian political philosophy. Deliberative democracy debates since the early twentieth century have returned the public’s precise role to the foreground (Lippmann 1922; Dewey 1927). The American federal government innovated with required public comment periods for administrative rules in the 1946 Administrative Procedure Act and specifically in environmental matters in the 1969 National Environmental Policy Act. Likewise, the Mackenzie Valley Pipeline Inquiry, launched in 1974 in Canada, was an important milestone in public engagement in contentious environmental issues. These developments were localised, prescribed processes of participation by those who might be materially affected by a specific decision.

Recent calls for PP&E have much broader scopes. Advocates seek PP&E efforts that are more frequent (applying to a greater range of issues), wider (including a greater range of stakeholders or the public), deeper (exploring in more open and/or extended manners), and earlier (relative to the practice or the technology’s expected use). The expectations of, demands for, and use of PP&E are especially evident in emerging technologies with possible impacts that are spatially and temporally expansive, and potentially or actually controversial. Consequently, proposals for PP&E typically extend beyond the local public that might be materially affected and include a wide range of stakeholders – potentially the entire world’s population in some cases – who might be affected only indirectly, socially, or ethically.

PP&E’s contemporary expansion can be traced to the academic field of science and technology studies (STS; here including the related domains of sociology of scientific knowledge and public understanding of science). This interdisciplinary approach, centred in sociology, arose in the 1960s and 1970s (Lengwiler 2008; Jasanoff 2017; Einsiedel 2014) with roots in neo-Marxist postmodernism, constructivism, and critical theory (Lynch 2016). It has evolved from descriptive, to normative, and then to prescriptive work (Delgado, Kjølberg, and Wickson 2011). In the mid-1990s, a near-consensus emerged within STS that the public should have greater roles in decisions regarding scientific research and technological development. The discipline soon thereafter took a ‘democratic’ or ‘participatory turn’, drawing more heavily from the norms and processes of deliberative democracy (Elam and Bertilsson 2003) as part of a reaction to what many STS scholars characterised as undue faith in experts (Collins and Evans 2002). Sheila Jasanoff’s 2003 essay ‘Technologies of Humility’ offers something of a manifesto for this more encompassing PP&E:

There is a growing need, I shall argue, for what we may call the ‘technologies of humility’ … They call for different expert capabilities and different forms of engagement between experts, decision-makers, and the public than were considered needful in the governance structures of high modernity … 

[P]articipation, I have argued, should be treated as a standard operating procedure of democracy (Jasanoff 2003, 227, 243; italics added).

Over the years, scholars have applied STS to policy- and decision-making settings. In many ways RI is the most recent major such application (Owen, Macnaghten, and Stilgoe 2012).

Three factors appear to have contributed to STS scholars’ increasing advocacy of PP&E. The first was growing criticism of the deficit model of public understanding of science (Irwin 1995; Bucchi and Neresini 2008; Einsiedel 2014). In this model, disagreements between scientists and the public were seen as resulting from the latter’s ignorance and unidirectional education was seen as the remedy. Some criticism of the deficit model was empirically grounded, based on the uncertified expertise that some lay persons might have (Wynne 1989; Epstein 1996; Kennedy 2019). Another line of critique of the deficit model (and of rational policy analysis more generally) was constructivist, describing relevant knowledge as socially constructed and valid epistemologies as plural (Amirrudin, Harrigan, and Naqvi 2021).

The second factor was a sense within STS and related disciplines that decisions at the intersection of the scientific and social were becoming increasingly complex and fraught, with growing divergence between physical risk assessments and public values (Bucchi and Neresini 2008; Felt et al. 2007). Some scholars asserted that the nature of (perceived) risk was changing from visible, natural, and external to society to invisible and unknown, arising as unintended consequences of the technologies that enable economic production (Beck 1992). STS scholars believed that such complexity and divergence contributed to a loss of trust in decision-making experts, institutions, and processes. Debates over genetically modified crops and foods in the 1990s and 2000s epitomise this factor.

Third, many STS scholars are sceptical of techno-scientific expertise on political grounds. More generally, some are sympathetic to political movements that seek to counter, diffuse, and redistribute concentrated power, wealth, and knowledge (Wynne 2002; Lengwiler 2008). As one STS scholar concludes,

STS as a project has been driven by doubts about the validity of the image of science (universalism, neutrality, impersonality, etc.) that underlay the liberal model. Following a sustained intellectual attack on the epistemological, sociological, and historical underpinnings of the liberal model of science, attention within STS is increasingly focused on the political implications of this critique and on what sort of political model is suggested by STS’s reformulations of the image of science (Thorpe 2008).

In this sense, STS’s critique is often a radical one that is suspicious of scientific practices and that calls to reduce techno-scientific elites’ epistemological power and decision-making authority.

PP&E has had substantial effects beyond the academy. It has been taken up by many public and private science and technology institutions, especially in Europe. There, several countries have established STS-inflected technology assessment offices and procedures that emphasise PP&E (Vig and Paschen 2000; Lengwiler 2008). An influential European Commission White Paper on Governance was informed, in part, by its Working Group on ‘Democratizing Expertise and Establishing Scientific Reference Systems’ (Gerold et al. 2001) Likewise, the European Union’s seventh cycle of scientific research funding was shaped by a report whose Working Group was dominated by prominent STS advocates of PP&E (Felt et al. 2007). Its eighth cycle, Horizon 2020, had RI as a key action and a cross-cutting issue (European Commission n.d.a). And the latest ninth cycle calls public engagement ‘a key element’ (European Commission n.d.b).

In our view, the push for increasing PP&E has been driven by important normative concerns and often helpful insights. However, institutional take-up of PP&E should be accompanied by evaluation. This is for multiple reasons: to ensure that the investment of resources is well targeted; to learn how to conduct better PP&E; to help fulfil deontological goals, such as those of democracy and justice; and to better understand people and society (Rowe and Frewer 2004, 516). Yet there has been notably little interest in evaluation. In 2004, Gene Rowe and Lynn Frewer asserted that PP&E should be evaluated and suggested how it could be (Rowe and Frewer 2004), but there has been little follow up in published literature. Several authors offer criteria for how PP&E exercises should be conducted, but these say little to nothing as to whether the exercises individually or collectively make progress toward any explicit or implicit objectives (Rowe and Frewer 2005; Chilvers 2008; Callon, Lascoumes, and Barthe 2009). For example, the authors of an article titled ‘Why should we promote public engagement with science?’ can say only that the question ‘“is public engagement worth it?” … is almost impossible to answer, especially as the most important benefits of engagement may be those that are challenging and inconvenient for the institutions that fund it’ (Stilgoe, Lock, and Wilsdon 2014, 7). Similarly, RI pioneer Phil Macnaghten states, ‘it is not self-evident how to appraise the quality, usability, relevance – and indeed legitimacy – of methods that have been crafted to cast light on how people are likely to respond to a technology where constituent social responses do not as yet exist’ (Macnaghten 2020, 18). To the extent that STS scholars have evaluated PP&E, the results are far from clear (Bucchi and Neresini 2008).

Evaluation requires objectives against which results can be measured. However, the proponents of PP&E in general and of individual activities are generally vague as to what it could and should accomplish. This imprecision has led Darrin Durant to assert that ‘STS participation theorists fail to clearly articulate why the public ought to be involved in science policy’ (Durant 2010, 190).

To advance our argument that PP&E deserves more robust evaluation, we describe what we see as three general sets of objectives of PP&E that we observe from its advocates. Building on and modifying a common tripartite division of PP&E’s objectives (Fiorino 1990; Stirling 2006), we propose three coherent sets of objectives (see Figure 1). First, PP&E could be a means to substantively improve decision-making. Second, it could be deontologically desirable in itself as it furthers widely-held norms. Third, some STS scholars value PP&E for its capacity to reveal the social construction of expertise, challenge the hegemony of technoscientific knowledge, and help redistribute political power away from elites. We describe these objectives as political ones.¹ The next three sections explore each of these sets of objectives.

Figure 1. Schematic of PP&E’s objectives.

Substantive objectives

In the first of our categories of objectives, substantive benefits exist when PP&E improves decisions related to science and technology, the scientific and technological outputs themselves, and/or the subsequent outcomes. As the editors of the 2008 edition of the Handbook of Science and Technology Studies wrote, ‘An assumption behind, and also a result of, research on Science, Technology, and Society is that more public participation in technical decision-making, or at least more than has been traditional, improves the public value and quality of science and technology’ (Hackett et al. 2008, 19). The improvements can be relatively direct – in which the public provides useful knowledge – or indirect – in which PP&E spurs learning processes among decision-making institutions.

When the organisers of a deliberative exercise on space policy (which we explore further below), for instance, underscore the importance of their undertaking by stating that PP&E ‘allowed citizens with various backgrounds, values, and knowledge to express important views on this topic that credentialed scientists and engineers, stakeholders, and policymakers might otherwise overlook or undervalue’ (Tomblin et al. 2015, 5), they are identifying substantive outcomes: experts will make better decisions for having access to the information provided by citizens.

Within the broad category of substantive objectives, there are several different approaches that might be called upon. ‘Crowdsourcing’ methods, for instance, lean on public input to increase the amount of data collected (e.g. augmenting fixed weather stations through additional reports by citizen weather watchers) and/or gain efficiencies in collecting that data (e.g. enlisting citizen scientists to conduct bird counts, thereby avoiding the costs involved in conducting the counts with one’s own staff). Efforts to unlock ‘hidden profiles,’ by comparison, focus on gaining access to information that might not otherwise be available (ranging from a criminal ‘tip line’ to collecting adverse event reports for a medicine or technology that would otherwise be unobservable). Different still, participants might be enlisted as part of creative or problem-solving processes (e.g. to brainstorm possible solutions or alternatives) to augment the ideas developed internally.

We also consider establishing sociological legitimacy as a substantive objective, as the goal of such increased legitimacy is to materially improve implementation and, therefore, outcomes. Aspects like ‘social licence’ or ‘community buy-in’ are forms of sociological legitimacy, wherein the public becomes more likely to take up, comply with, or otherwise support the decision thanks to their belief in the appropriateness of the decision-making process. As Macnaghten (2020, 9) points out, these sociological objectives are prevalent in PP&E activities, where such initiatives

are typically aimed at improving relations between science and society and restoring legitimacy. In practice, they have been developed for reasons that include the belief that they will help restore public trust in science, avoid future controversy, lead to socially robust innovation policy and render scientific culture and praxis more socially accountable and reflexive.

Indeed, this focus on results is what unifies the various substantive objectives. Substantive objectives aim for ex post, practical, consequentialist outputs: improving the quality of the decision being made or its implementation through buy-in and acceptance.²

A likely reason that substantive objectives are commonplace in PP&E efforts is that they appeal to all major constituencies, including the decision-makers, the participants, and the ‘decision-takers’. For those in positions of power – whether organising, funding, or benefiting from PP&E – they justify investment of resources into the process in hopes of finding cost-savings, discovering profitable ideas, or easing implementation. For those participating, substantive objectives offer hope and reason for investing one’s time. Substantive objectives offer the promise that your voice is valued by decision-makers, that your ideas might reshape your city, or that your perspectives might protect your neighbours. They also provide rhetorical promise – justified in some cases, and potentially manipulative in others – that decision-makers are committed to solving the problems in a way that benefits the community.

The extent to which PP&E efforts met their substantive objectives remains unclear. Although there are empirical evaluations of a few such undertakings (e.g. Futrell 2003), there is a notable gap of any using rigorous systematic review methodologies or high-quality, pre-registered evaluation frameworks. Moreover, PP&E practitioners may face perverse incentives that reward pre-event rhetoric that exaggerates potential outcomes, for intentionally leaving substantive goals vague in ways that enable a ‘success’ regardless of impact, or for using the messiness of policy-making processes as a shield against calls for specific demonstrations of impact. As Caroline Lee (2014, 199) summarised in her examination of deliberative processes more broadly, for many scholars, deliberative processes seem ‘insufficiently articulated with substantive policy outcomes to alter the larger political landscape.’ She concludes that research on deliberative processes finds that ‘“[t]he difference deliberation makes” on actual policy or power structures is, unfortunately, not very much’.³

Deontological objectives

As our second category of objectives, PP&E could also further widely-held norms that are desirable independent of their practical consequences. The deontological objectives that are most commonly discussed by PP&E’s advocates are drawn from deliberative democratic theory (Elam and Bertilsson 2003; Berg and Lidskog 2018; Jason Chilvers and Kearnes 2016; Durant 2010) where a parallel literature has discussed the benefits of public deliberation concerning science and technology (e.g. Dryzek et al. 2020). Although PP&E and deliberation are not synonymous, and PP&E exercises may have distinct objectives (Braun and Könninger 2018; see also ‘Political objectives,’ below), our account primarily draws on deliberative democratic theorists as their objectives are more developed.

Deliberative conceptions of democracy are commonly organised around ideals of political justification and of communication that draw on John Rawls’s and Jurgen Habermas’s works respectively. (We discuss a third ‘agonistic’ tradition in the next section.) The Rawlsian tradition emphasises ‘public reason’ among citizens who treat each other as free and equal if they justify their claims in terms of public reasons that others will have reason to accept. As Simone Chambers observes, Habermas’s phrase the ‘unforced force of the better argument’ captures an insight that is central to most theories of deliberative democracy: ‘reason-giving is both a means of arriving at better outcomes and a way of recognizing each participant as equal and free’ (Chambers 2018, 66). By contrast, the Habermasian tradition emphasises procedures that will create conditions of equal deliberation that rule out ‘coercion, deception, self-deceptions, strategizing, and manipulation’ (Dryzek 2014, 241). Both approaches view creation of political systems that promote freedom and equality as a deontological goal.

Deliberative theorists from both of these camps commonly advocate institutional reforms that will enhance the deliberative capacity of entire social systems (Chambers 2018; Dryzek 2009; D. Owen and Smith 2015). Advocates of this ‘systemic turn’ describe deliberative capacity as the extent to which a political system possesses structures that promote deliberation which is authentic (reciprocal, non-coercive and connected to more general principles), inclusive (includes the full range of interests and discourses in a political community), and consequential (deliberation influences social outcomes and decisions, however indirectly) (Dryzek 2009; see also Fiorino 1990). While such deliberation commonly involves the general public, theorists of deliberative democracy also point to the capacity for deliberative processes to integrate knowledge from across multiple, specialist domains. Thus, PP&E forms one element of a wider project of promoting deliberative capacity.

Evaluation of deliberative democrats’ claims, especially in respect of deontological objectives, is challenging. Evaluations of deliberative practice generally divide between two strategies (Bächtiger 2018). ‘Input-output’ approaches evaluate deliberative institutions against criteria that confirm both inputs (e.g. whether deliberation has occurred among a representative sample and deliberators had equal capacity to speak) and outputs (e.g. whether participants’ opinions changed) conform to deliberative standards. By contrast, process-based approaches seek to examine the quality of deliberation through empirical ‘speech act analysis’ and other measures of ‘deliberative quality’.

To what extent have PP&E exercises achieved the objectives of deliberative theory? To the extent that their deontological purpose is to create temporary conditions of relative equality for participants in deliberation, evidence suggests a measure of success (Macnaghten 2020). The evidence base concerning ‘deliberation’ exercises more generally (of which PP&E may be considered a subset) is stronger again; there is clear evidence that participants in many deliberative exercises perceive them as a positive experience, that citizens have the capacity to overcome polarised and populist framings, and that deliberative exercises commonly lead to opinion-change (Dryzek et al. 2019). However, there is little if any evidence that PP&E exercises have had wider, systemic impacts or opened political systems to greater equality. Advocates of PP&E admit that progress here may be necessarily limited. Reflecting on a decade-long series of PP&E efforts regarding diverse emerging technologies Macnaghten concludes

While, arguably, each of the projects were to a certain degree successful in crafting an (A)nticipatory methodology, in (I)ncluding new voices in the governance of science and innovation, in contributing to (some) additional (R)eflexivity in academic and policy practice, the extent to which they contributed to (R)esponsiveness in science policy institutions – and more widely in political decision-making bodies – is less clear (Macnaghten 2020, 46).

What’s more, scholars and practitioners sometimes describe how PP&E reproduces and reinforces existing framings, decisions, political inequalities, and exclusions (Macnaghten 2020). Eva Lövbrand and co-authors conclude that ‘many scholars of science and technology are uneasy with deliberative democrats’ efforts to prescribe objective principles by which deliberative encounters should be structured and evaluated.’ (Lövbrand, Pielke, and Beck 2011, 486) For example, in a survey of practitioners’ experiences, Jason Chilvers notes that ‘There is nothing unique about participation that makes it immune from framing effects, the exercise of power, interest-based manipulation, strategic behaviour, closing down debate, ignoring uncertainties, and unnecessarily excluding human/nonhuman actors’ (Chilvers 2008, 44). He also states that ‘public engagement experts’ are themselves assuming powerful positions in policy-making processes through their influence on deliberation. Likewise, Jan-Peter Voß and Nina Amelung have noted the irony that ‘anti-technocratic engagements with governance gave birth to efforts at establishing technoscientific control over questions of political procedure’ (Voß and Amelung 2016, 749). Others have identified such differentials among non-expert deliberators; between deliberators and the facilitators and experts that assist deliberation; and as an outcome of the entire PP&E process. Together these differentials mean that PP&E can reinscribe power inequalities through ‘expert reassurance’ (Barnett et al. 2012, 47). Moreover, arguments for ‘enclave deliberation’ among traditionally marginalised groups (Karpowitz, Raphael, and Hammond 2009) reflect concerns that deliberative processes have the capacity to marginalise diverse and minority perspectives. Another internal critique notes that the pursuit of equality can have perverse outcomes. For instance, Luigi Pellizzoni argues that since ‘the principle of equal stance is at odds with the acknowledgement of differences that cannot be settled,’ the ‘quest for more equality may lead to a disregard for diversity’ (Pellizzoni 2003, 209). These perspectives echo many theoretical critiques of the practical possibility of Habermasian ideal speech conditions and connect with a common critique of PP&E exercises as having the potential to reproduce structural inequalities.

Political objectives

In our final category of objectives, some proponents of PP&E hope that it will, by providing a democratic check on the scope of scientific judgement, achieve a deeper purpose of challenging epistemic authority and redistributing power away from technological and scientific elites (Jasanoff 2011, 232).⁴ These political objectives appear congruent with some criticism of PP&E, in the above paragraph, and in the previously cited claim that ‘the most important benefits of engagement may be those that are challenging and inconvenient for the institutions that fund it’ (Stilgoe, Lock, and Wilsdon 2014, 7). Some STS scholars propose PP&E methods that they believe will open up and recognise – rather than close down and obscure – difference and antagonism (Chilvers 2008; Van Bouwel and Van Oudheusden 2017). In their view, political participation should ‘increase the democratic supervision of science’ (Jasanoff 2003, 232), cultivate humility ‘about both the limits of scientific knowledge and about when to stop turning to science to solve problems’ (Jasanoff 2007, 33), and work to ‘expose … the unstated political and economic agenda … which is embedded within, and cultivated by contemporary technoscience’ (Wynne 2007, 109). Macnaghten reflects on contributing to the development of RI:

There is thus no alternative but to design participation in carefully orchestrated spaces, even if this points to the curious irony of having to engage reflexively in the development of anti-technocratic expertise … our internal criteria for success was not whether it contributes to predefined policy goals but rather the extent to which it challenges dominant policy, academic and industry norms and assumptions opening up new understandings, problem definitions and even policy narratives (Macnaghten 2020, 53).

More boldly, Jason Chilvers and Matthew Kearnes conclude that ‘much of the political and academic commentary concerning public participation tends to assume that the prima facie goal of public engagement initiatives is to speak social science and “public truths to power”‘ – a phrase that is commonly used by activist movements (Chilvers and Kearnes 2016, 265; see also Jasanoff 2003, 225). Likewise, Sarah Hartley and colleagues call for RI to (re)politicize scientific research and its governance (Hartley, Pearce, and Taylor 2017).

While we discuss political goals as a category of objectives for PP&E, it is important to note that while some of the STS scholars who most clearly identify PP&E’s political objectives believe these can be achieved through PP&E, others are wary of decision-maker instigated exercises, which they worry are intrinsically vulnerable to co-option. The concern here is that STS research can ‘inadvertently become an agent of tacit rationalisation of deep structures of power which science as institutionalised co-constructs and naturalises.’ (Wynne 2007, 106) For example, PP&E activities that involve randomly selected members of the public have been accused of creating contexts in which deliberation is ‘more easily aligned with official publics,’ as the lay public has been separated from the influence of ‘mobilized counterpublics’ (Hess 2011, 638). Rather than endorse the ‘invited’ or ‘hosted’ PP&E exercises whose objectives this article analyses, these scholars tend to advocate uninvited and disruptive forms of engagement, which they celebrate for their potential to insert the voice of politically mobilised constituencies into public debates and to disrupt scientific values (Wynne 2007; Wehling 2012; Chilvers and Kearnes 2016).

The radical STS scholars’ critiques of PP&E echo and sometimes cite (e.g. Jason Chilvers 2008, 445; Van Bouwel and Van Oudheusden 2017, 54; Macnaghten 2020) two critics of deliberative theory’s normative and procedural shortcomings. First, Iris Marion Young’s account ‘Activist Challenges to Deliberative Democracy’ postulates that ‘in the real world of politics, where structural inequalities influence both procedures and outcomes, democratic processes that appear to conform to norms of deliberation are usually biased toward more powerful agents.’ (Young 2001, 671) Since deliberative fora are created in the context of a ‘sedimentation of unjust structural inequality,’ these background inequalities shape the deliberative agenda, available discourses and the range of imaginable alternatives (Young 2001, 682). Young does not deny that deliberative fora such as PP&E exercises can bring certain benefits, but she argues that these must be balanced against the tendency for participation to legitimate the existing conditions of structural injustice. While she outlines an activist critique of deliberative democracy, Young’s own view is that responsible democratic communication includes both rowdy protest and respectful deliberation. As such, some STS scholars have drawn on Young’s work to advocate for ‘uninvited participation’ as opposed to formal PP&E (e.g. Wehling 2012, 54–55)

Second, Chantal Mouffe’s critique of the normative goals and conceptual foundation of deliberative democracy is more comprehensive. In her analysis, deliberative theorists misdiagnose the character of political consensus when they suggest that it can be grounded in reasoned agreement. Instead, Mouffe argues that consensus is always the ‘temporary result of a provisional hegemony [which] always entails some form of exclusion’ (Mouffe 1999, 756). She articulates an alternative account of a democratic public sphere in which it is ‘agonistic confrontation’ and pluralism which is central to democracy. Those STS scholars who embrace protest but reject organised PP&E commonly draw on Mouffe to argue for forms of ‘agonistic deliberation’ that ‘contrast with hegemonic framings and that open up novel spaces for political action.’ (Macnaghten 2020, 54) However, whereas Mouffe and Young both focus on the potential for political activism to destabilise all forms of power, STS scholars are more narrowly focused on disrupting ‘public quiescence for the powerful commercial interests which act as global techno-sciences’ patrons.’ (Wynne 2007, 108)

Has PP&E achieved the political objectives of reducing and redistributing techno-scientific elites’ hegemonic epistemic power? STS literature identifies more cases where it is protest movements, rather than organised PP&E, which have shifted research priorities (Kleinman 2000). We place such claims in three rough categories. First, some protests have reformed scientific and technological practices in ways that grant some power to marginal groups but accept the underlying goals of scientific research. As a leading example, ACT UP’s challenge to medical research practices increased the consideration given to the interests of drug-trial participants but supported the wider goal of finding a treatment for HIV. We note also that the transfer of power to marginal groups was imperfect; critics point to ACT UP’s domination by relatively powerful white men (Epstein 1995).

A second type of achievement is the protest movements that have disrupted the path of ‘favoured assumed monolithic innovation’ preferred by techno-scientific elites (Wynne 2007, 106). Activist challenges to genetically modified food and nuclear power are often cited as successful examples, although in both cases it has been a combination of invited ‘priority-setting’ and uninvited protest activity – not PP&E as it is meant here – that seems to have shifted scientific priorities. As with reformed practices, the extent to which participation has actually transferred power to marginalised groups in these examples is also debated. Specifically, critical scholars argue that the disruptions of biotechnology prioritised affluent consumers’ concerns over those of developing-world farmers (Herring and Paarlberg 2016). Some climate activists now critique movements opposing civilian nuclear power for placing relatively powerful communities’ abstract anxieties over more vulnerable groups’ concrete exposure to climate impacts (Symons 2019).

Finally, STS literature also canvases political challenges to scientific hegemony that align with populist or conservative movements, of which vaccine resistance, climate-change scepticism, and 5G and chemtrail conspiracies are key examples (Sismondo 2017). The extent to which PP&E has contributed to these ideologically diverse bottom-up contestations is unclear, and the relationship between STS, populist movements, corporate sponsored critiques of specific scientific claims, and the authority of science more generally remain contested (Kleinman 2020). Some scholars examining these trends emphasise that STS has ‘never been anti-science’ but rather raises wider questions of meaning and public concern, and that right-wing attacks typically focus on specific facts that ‘threaten entrenched economic interests’ rather than on science more generally (Lynch 2020, 55–56). And in some other cases, it seems that challenges arise primarily from corporate questioning of economically inconvenient scientific consensus (Lynch 2020), with populism serving merely as a useful front.

Ultimately, PP&E’s achievement of implicit political objectives appears limited. Some motivated political campaigns, such as those questioning the reality of climate change and opposing nuclear power, have made ‘progress’ toward their political goals, and scepticism of elite techno-scientism seems to be growing. But it is unclear whether these were the result of PP&E or of protests, generalised fear or ignorance of novel technologies, and/or corporate power’s interests. Beyond these, major scientific and technological domains – defence, health, social-medial and consumer technology – have not been significantly altered by either PP&E or uninvited participation activities.

Wider costs and deeper problems

To the limited extent that advocates of PP&E have explicated objectives, we find little evidence that it has met them – not because we believe there’s strong evidence of failure, but because there’s little rigorous evaluation undertaken. The combination of growing calls for and practice of PP&E with an absence of serious efforts at evaluation suggests that critical introspection is warranted.

But why should we be concerned about whether such activities have or achieve objectives? After all, PP&E has modest direct costs and seems to have beneficial effects. We offer three reasons.

First, at the most basic level is the burden of proof. Here, one group (advocates of PP&E) demands that others (scientists and other researchers) undertake burdensome actions that increase the cost of research. It seems elementary that the burden of demonstrating that the actions generally advance the objectives and that the objectives are beneficial or widely desired falls on those who make the demand. Although the substantive and deontological objectives appear beneficial and widely desired, PP&E advocates have not yet produced evidence that PP&E leads toward them. In the case of the political objectives, it is both unclear whether these objectives are beneficial and desired, and to the extent that they are desirable, whether they might not be better progressed through bottom-up protest than by organised PP&E.

Another reason for critical attention is that, as noted above, PP&E’s costs are wider than the direct ones of running PP&E exercises. These are not only the obvious financial costs, but if PP&E exercises have the influence that their proponents claim or desire, then informative and innovative projects will inevitably be delayed, scaled back, cancelled, and never planned. PP&E may even cause the public to be more sceptical of or hostile to scientific and technological developments, engendering a sort of self-fulfilling prophecy of deleterious social impact. For instance, PP&E processes have been used by activists to amplify anti-vaccine messages (Reiss and Romzek 2020). Ultimately, delayed, downscaled, cancelled, and unplanned scientific research and technology developments slow knowledge generation and innovation, and human welfare is concomitantly lesser than it otherwise would be. These costs may be worth paying, especially if the delayed, downsized, or non-occurring projects would have otherwise been relatively socially costly. Yet this possibility does not indicate that costs should be neglected.

A third reason to attend to PP&E’s impacts is that its costs may be disproportionately borne by already disempowered and vulnerable groups. Calls for PP&E have been loudest in cases of proposed scientific research and technologies with global political and ethical implications, but PP&E practice has been concentrated in the Global North. Although any discussion of impacts due to Global North–South differences is somewhat speculative, it seems clear both that PP&E exercises will prioritise the interests and concerns of the consulted communities (and perhaps of those who organise the consultation) and that some public attitudes differ significantly between the North and South. We may thus see a systematic prioritisation of rich-world worries of sometimes hypothetical risks that have been constructed in anticipatory terms (Herring and Paarlberg 2016). For example, safety concerns regarding vaccines are higher in Europe and the US, with favourable access to public health, than in most of the Global South, where infectious disease remains endemic (prior to the Covid-19 pandemic; Larson et al. 2016). Likewise, it seems probable that PP&E exercises in affluent communities concerning, say, development of gene-drive technologies that could be used to control malaria, may prioritise different issues than would an exercise conducted in Nigeria (Teem et al. 2019).

In addition to these costs, the contemporary STS-driven PP&E endeavour possesses deeper problems linked to its political objectives. Specifically, these objectives are rarely explicitly stated and appear not to be widely shared even among many of PP&E’s advocates, indicating that these objectives are in tension with the deontological ones regarding maintaining democratic connection between the public and science and technology. This is akin to what Lövbrand and co-authors call ‘A democracy paradox in Studies of Science and Technology’ (Lövbrand, Pielke, and Beck 2011). To be clear, the bottom-up protest that radical STS scholars advocate can have a beneficial role in democratic societies. For example, it advanced the development and distribution of AIDS drugs, described above, and shut down profit-driven attempts at environmentally risky ocean fertilisation to remove atmospheric carbon dioxide (Strong et al. 2009). But such protests are not the product of deliberate PP&E as we define the concept here and as the concept is generally understood. Conflating PP&E and protest serves only to obfuscate the roles, objectives, and achievements of both.

Despite being increasingly normative, many RI and STS scholars struggle with whether their work should also be prescriptive and, if so, how. Some of them have tried to bridge the theory-decision-making gap; a central piece of this bridge has been PP&E. However, what its advocates hope to accomplish and – whatever it is – whether they have done so still remains unclear.

STS scholar Brian Wynne argues that

Academic and practical public engagement should centralise this as a normative issue – no major innovation of these kinds [i.e. favoured assumed monolithic innovation with hegemonistic tendencies] should be entertained without a full and serious open-minded process of appraisal of not just risks, but of benefits-claims and promises, and of alternatives (Wynne 2007, 106).

He made this assertion in the context of nuclear power, genetically modified crops, and unnamed ‘biomedical trajectories’; however, his argument for appraisal of risks, benefits-claims, and promises should likewise apply to innovative techniques of engaging public attitudes and influencing decisions, such as PP&E. While the primary evident criticism of PP&E among STS scholars is that it suppresses dissent, ‘closes down’ options, and fails to challenge and usurp extant powerful interests, including the decision-makers who conduct the PP&E (Stirling 2006; Wynne 2007) – some basic scrutiny of benefits-claims and promises also seems appropriate.

Challenges and opportunities

In order to support this kind of scrutiny of costs versus benefits – both of PP&E broadly and of any given local implementation – a practice of methodologically rigorous evaluation is needed. Advocates of PP&E articulate an ambitious, diverse, and challenging set of objectives around improving decision-making and fulfilling democratic potential. Yet, to our knowledge, there are no meta-analyses or comprehensive systematic reviews of PP&E impacts, nor is there even a sufficient evidence pool of transparent, arms-length evaluation of specific activities that could enable these broader evaluations. In short, to help advance PP&E’s funding, reputability, and consistent utilisation, commensurate investment is needed into developing clear and specific objectives, conducting independent evaluation, and offering transparent reporting of successes and failures alike. We thus call for ex ante, clear, specific, consistent, and measurable objectives for PP&E efforts and for their robust ex post evaluation.

The deployment of rigorous evaluation methods into PP&E settings will, by definition, be challenging. Substantive objectives can be measured in principle, for instance, but methodologies will need to be developed for rigorously capturing impact in muddled policy-making settings without falling victim to confirmation bias or moving goalposts. Arguably, deontological objectives are aspirational and – while measurable progress may be made – they are never strictly met. And for both sets, any sort of benefit–cost analysis would often require comparison of incommensurate kinds of investments and outcomes. It can also take time for decision-making processes to play out, making it all the more challenging to feed the results of timely evaluation into nuanced improvements to ongoing activities.

Yet these complexities urge more sophisticated approaches to evaluation led by those with expertise in program monitoring, rather than simply writing rigorous evaluation off as undesirable or unattainable. As PP&E practice matures, the precision with which it is planned, monitored, and evaluated should also increase, allowing for even greater understanding of what methods achieve which objectives. In other words, PP&E should become a more serious undertaking by critically assessing its own efforts in order to cast light upon those that do not suffice, rather than simply allowing nebulous definitions of success to impede improvement, as the above quotations of Stilgoe, Lock, and Wilsdon (2014, 7) and of Macnaghten (Macnaghten 2020, 18) imply.

Along these lines, we suggest that PP&E advocates address four shortcomings in current efforts that could help them toward at least the substantive and deontological objectives. The first recommendation is to clearly define observable criteria of ‘success’ prior to PP&E. It is critical that those facilitating PP&E are explicit about which objectives among the multiple categories (or others) are being pursued in a given activity. Once the general type of objective has been identified (e.g. to make a substantive contribution to the quality of governance), more reflection is necessary on the specific expected contribution (e.g. to ensure that specific hidden profiles are made visible in the decision-making processes). In particular, objectives should be ex ante, clear, specific, consistent, and measurable. Regardless of whether these are (ideally) formally pre-registered or, at least, internally identified, this set of characteristics is essential. Using vague words and adopting unmeasurable criteria causes assessing the extent to which objectives have been achieved to be difficult if not impossible. Worse, this creates the risk that any outcome can be – and too often is – interpreted as successful.

To draw on an example of a deliberative activity that has made commendable efforts at evaluation – yet simultaneously shows just how tricky rigorous evaluation is in this context – consider the public deliberation hosted by the Expert and Citizen Assessment of Science and Technology (ECAST) network to inform NASA’s Asteroid Initiative. It identified ‘two main goals’: to ‘elicit[] nuanced information from a diverse group of citizens whose insights would not otherwise be available to decision makers’ and to ‘provide public views … as input into NASA’s decision-making process’ (Tomblin et al. 2015, 4). These admirably capture substantive objectives, but they are not operationalized in a fashion that makes them measurable or falsifiable. In this case, the goal of eliciting ‘nuanced information’ lacks an ex ante, clear, and specific definition that is – to reference an imperfect yet useful concept – falsifiable. In other words, it is important that this criterion could realistically be failed (i.e. what would it look like for the information elicited to be non-nuanced?) for it to be meaningful. The criteria of ‘providing public views … as input to NASA’s decision-making process’ is, similarly, sufficiently non-operationalized as to be unfalsifiable. Indeed, one could imagine a terribly run engagement effort that, in the style of ‘loading dock science’ simply provided a non-actionable report to decision-makers, as still ‘passing’ this criteria (even if the report was never read, it indeed provided input!).

This leads to the second, related problem: the importance of independent and arms-length evaluation as a marker of credibility and to counter unintentional bias. When the organisers themselves evaluate their own undertakings, it creates strong incentives for even the most honest facilitator to emphasise the success of an initiative in order to maintain current access and/or gain future funding. For example, the organisers of the ECAST Asteroid PP&E wrote the post-event report discussed earlier (Tomblin et al. 2015; Tomblin et al. 2017; Farooque, Tomblin, and Sittenfeld 2017). This is not uncommon. Such practitioners are, generally speaking, ongoing recipients of substantial funding for running such projects, invested in the development and deployment of the methodology, and/or closely professionally connected with colleagues and institutions for whom this is a primary activity.⁵ (In Kaplan et al, for example, the authors highlight having completed 40 PP&E activities and having another 35 scheduled.) This isn’t to diminish the important perspectives that those involved in facilitating PP&E can offer in describing the events, and it is critical that these voices be involved in developing evaluation frameworks and that their reflections of the experiences are included. But, as the practice of PP&E grows in funding and influence, arms’ length scrutiny becomes all the more important in helping to refine methodologies and identify the most promising practices of PP&E.

Third, reporting on success or failure against specific criteria should be transparent and timely but is often largely absent. This is an incredibly difficult challenge, as it can be tempting to set criteria that require significant access to decision-makers’ processes of using, taking up, or rejecting this advice. Indeed, as the NASA Asteroid Initiative deliberation organisers themselves identify, ‘assessment of the importance of the [deliberative process] to internal NASA decision-making processes is a challenge, as pre-decisional information surrounding government procurement decisions cannot be shared publicly’ (Tomblin et al. 2017, 158). As a result, the question of rigorously evaluating impact with transparent methods is a complex one.

As mentioned above, we chose the example of the Asteroid Initiative precisely because it is an example of PP&E where the organisers have put significant effort into reporting on their impact. For example, the organisers point out three pieces of evidence suggesting impact, namely (1) that NASA reported ‘internal briefing discussions … focused on the diversity of the values participants discussed’ (Tomblin et al. 2017, 165); (2) that ‘citizens’ preferences were consistent with NASA’s final choice’ (Kaplan et al p. 7); and (3) that ‘participants’ strong emphasis on planetary defense during the deliberations influenced the creation of NASA’s Office of Planetary Defense’ (Kaplan et al p. 7). Even acknowledging that we, as outside observers, are limited by the impact that has been documented and published (as opposed to evaluation still ongoing), this kind of disclosure is commendable.

There are still, however, opportunities for improvement. Per the first two recommendations, such evaluation would be stronger if clearer objectives were specified ahead of time (i.e. how is the ‘nuance’ of views elicited or their ‘impact’ being measured?), as well as if the evaluations were conducted by arms-length program evaluation specialists. Per this third recommendation, a way to increase transparency of outcome reporting might be through comparison with an evaluation protocol pre-registered prior to the event. Likewise, credibility would be increased if there was more robust disclosure of the impact criteria upon which the event was unsuccessful. For instance, subsequent interviews with NASA personnel as part of a doctoral dissertation highlighted that NASA chose to share the Asteroid Initiative report only in limited online venues to ‘to avoid as much scrutiny as possible’ of the use of PP&E (Torres 2021, 102), as well as the way that agency administrators were concerned about the potential of the activity returning input other than that in alignment with what was already ‘politically blessed’ (Torres 2021, 37–38), both indicating the possibility of a more complex evaluative story than otherwise suggested. This disclosure of lack of success or negative impacts is doubly important in cases where evidence of impact rests upon testimony of the very group which commissioned the PP&E event, and which thereby has incentive to report positive impact from their investment of time, money, or sponsorship.

Finally, as we’ve articulated above, many of the substantive and deontological justifications for PP&E highlight the importance of attending to expertise: generally, expertise within the publics that have often been overlooked. But, conducting methodologically rigorous program evaluation is also a specialism that requires extensive expertise – and distinctively different expertises than those required to run such programs. Because evaluations are, at present, largely conducted by specialists in planning PP&E efforts rather than those trained in evaluation methodologies, basic methodological problems can be unfortunately common, such as using skewed scales that pre-load success; fostering bias through framing, researcher demand, acquiescence, and social desirability; and failure to benchmark against controls. Just as PP&E recognizes that various publics bring important expertise to decision-making tables, avoiding epistemic trespassing into program evaluation is also important (Ballantyne 2019) and can be achieved by commissioning evaluation specialists rather than risking accusations of self-promotion and moving goalposts.

Lest we sound negative about the potential impact of PP&E, we wish to reiterate that we endorse, in some cases strongly, many of the goals to which it aspires and admire its proponents’ efforts in advancing these. Indeed, it is precisely because of the early successes of PP&E – and the enthusiasm with which calls for and funding of PP&E are expanding – that the opportunity presents itself for increased professionalisation and rigour in evaluative processes. Developing clearer objectives pre-event, reducing conflicts of interest, transparently disclosing successes and failures alike, and leveraging evaluation expertise will help allow for more informed conversations about the tradeoffs between costs and benefits of PP&E.

Acknowledgments

The authors thank the editor and the two anonymous reviewers for their helpful comments.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by Open Philanthropy and the Social Sciences and Humanities Research Council [Grant Number: 1006-2019-0001].

Notes on contributors

Jesse L. Reynolds

Jesse L. Reynolds researches how society can develop norms, procedures, and institutions to manage opportunities and challenges at the intersection of sustainability and new technologies. In this, his approach draws from law, political sciences, and economics. Reynolds has extensively researched the governance of geoengineering and is also exploring the roles of new biotechnologies, such as gene drives, and artificial intelligence in the conservation of biodiversity and facilitating sustainability. His book The Governance of Solar Geoengineering: Managing Climate Change in the Anthropocene (2019) was published on Cambridge University Press, and Reynolds’s research has been in WIREs Climate Change, Current Opinion in Environmental Sustainability, Climatic Change, Environmental Politics, the Journal of Environmental Law, and elsewhere. He currently is Executive Secretary of the Global Commission on Governing Risks from Climate Overshoot and Senior Policy Officer at the Paris Peace Forum. Prior to these, Dr. Reynolds was an Emmett / Frankel Fellow in Environmental Law and Policy at the Emmett Institute on Climate Change and the Environment of the University of California, Los Angeles School of Law.

Eric B. Kennedy

Eric B. Kennedy works at the intersection of expertise, decision-making, and institutional epistemologies. Trained in Science and Technology Studies, his work focuses on knowledge production and decision-making in the context of disaster and emergency management. Kennedy was co-editor of the Rightful Place of Science volume on Citizen Science, and has authored a variety of manuscripts on topics ranging from use of predictive models in wildfire management to forms of expertise used by public constituencies when challenging policy. He is an Assistant Professor of Disaster and Emergency Management at York University, as well as a member of the graduate program in Science and Technology Studies. He is also the founder and director of the Forum on Science, Policy, and Society, a Canadian not-for-profit organization dedicated to training the next generation of science policy leaders.

Jonathan Symons

Jonathan Symons Jon's current research primarily focuses on global climate politics, including debates over solar geoengineering and carbon dioxide removal. His most recent book Ecomodernism: Technology, Politics and Climate Crisis (Polity Press, 2019) explores the argument for increased state investment in low-carbon innovation. Jon also works on processes of international norm change. His previous book Queer Wars: The New Global Polarization over Gay Rights (co-authored with Prof. Dennis Altman, Polity, 2016) explored the debate between those arguing that ‘LGBTQI rights are human rights’ and those who believe rights protections should not extend beyond the ‘traditional values of mankind’. Jon is a Senior Lecturer in Politics and International Relations in the Macquarie School of Social Sciences.

Notes

1 The ‘classic’ tripartite division included the instrumental goal of making resulting decisions more legitimate. But legitimacy has multiple definitions. In some contexts, it means that those likely affected – and often the public more broadly – appear to support, accept, or comply with the decision, institution, or process, which can in turn increase effectiveness. We thus include efforts toward this practical, consequentialist ‘sociological legitimacy’ within our substantive objectives. Other times, legitimacy means that the decision, institution, or process satisfies independent norms of justice or democracy. We include efforts toward such ‘normative legitimacy’ within deontological objectives.

Some STS scholars criticise decision-makers utilising PP&E instrumentally with the goal of convincing the public of what they were going to do in any case (Stirling 2006; Stilgoe, Lock, and Wilsdon 2014). Here, we do not assume such Machiavellian motives to rubber stamp pre-existing decisions.

2 These pragmatic ambitions can, of course, be presented in slightly different language in different contexts. As Gudowsky and Bechtold (2013, 1) summarise from the literature, decision-makers’ objectives for public engagement include achieving ‘socially more robust decisions’ that bring ‘citizens and institutions closer together’ (the latter quoting Monaghan 2007), the need to broaden information underpinning decisions, to increase decisions’ perceived legitimacy, and a desire to shape collective identity.

3 Lee goes on to make several important and valuable points developing this general position. For example, deliberative processes can be effective in minimising opposition or extreme views, galvanising support for selected paths, or even just for creating energising processes. As such, we emphasise that this quotation is her characterization of the literature, rather than Lee’s personal analysis. But, even in Lee’s work, lasting evidence of impact can be difficult to find. For example, in the case of the $14.5 billion Unified New Orleans Plan that she examines, ‘little evidence of the congresses appeared online’ even four years later (Lee 2014, 19).

4 Importantly, a dissenting minority of STS scholars views scientific expertise and culture as a democratic check on populism and authoritarianism. Some of these defenders of scientific expertise describe ‘social constructivist’ STS scholars’ position as resembling, at least superficially, populist critiques of science as reflecting the interests of elite social groups (Collins and Evans 2019, 210).

5 The one example we were able to find of arms-length examination, a reference to the initiative in a Government Accountability Office report, doesn’t actually conduct any evaluation, but simply reports testimony from NASA that the effort ‘According to NASA officials, the results of these forums provided NASA with insights into public understanding and views on NASA’s asteroid work’ (p. 44).