Participation in monitoring and evaluation for RRI: a review of procedural approaches developing monitoring and evaluation mechanisms

ABSTRACT

Three arguments support the involvement of actors in the design and implementation of the Monitoring and Evaluation (M&E) for Responsible Research and Innovation (RRI). First, it strengthens the evaluation; second, it allows taking advantage of the performative function; third, it is aligned with the concept of RRI. In Europe, the trend to institutionalize RRI triggered an interest in developing specific M&E methods and tools, but how actors participate in these processes is still being determined. This paper investigates the extent to which the participation of actors occurs or is expected to occur by using the three stages of translation proposed by Callon and colleagues – problematization, development of the research, and transfer to a real setting. Through a systematic review of 25 approaches developing M&E mechanisms, our findings show that participation occurs or is foreseen mainly in the later stages and is specially linked with learning and trust-related purposes.

KEYWORDS:

• Responsible research and innovation
• evaluation
• monitoring
• participation
• inclusiveness

Introduction

Responsible Research and Innovation (RRI), or responsible innovation, has gained momentum in Europe, especially since 2009 (Timmermans 2017). This is due to its adoption by funding agencies, including the European Commission (Zwart, Landeweerd, and van Rooij 2014), and recognition of the need to review the relations between science, innovation and society (Flink and Kaldewey 2018). In turn, this sparked interest in monitoring and evaluation methods and tools. Such mechanisms aimed to support the implementation of more responsible governance of science and innovation, or to assess the level of maturity of the concept of RRI in a certain project, institution or policy. We will refer to the amalgam of outputs resulting from this interest as monitoring and evaluation (M&E) mechanisms for RRI.

A common feature in the RRI literature refers to the inclusion of new types of knowledge facilitated by the involvement of societal actors in the research and innovation process (Timmermans and Blok 2021). The challenges of societal engagement in public research, scientific governance and industry have been widely documented (Bauer, Bogner, and Fuchs 2021; Brand and Blok 2019).

Despite the general acknowledgement of the need for more inclusive governance of science and innovation in the RRI literature, it is still being determined if diverse actors participate in the processes to develop and implement M&E mechanisms for RRI. As mentioned, by M&E mechanism, we refer to methods and tools designed for monitoring or evaluation purposes, such as procedures, evaluation grids and quantitative or qualitative indicators. This paper investigates when, how much and in what ways the participation of different actors is considered in the approaches followed in the process towards M&E for RRI and their implementation in real settings. We will focus on contextualization through participation to refer to strategies and processes that allow or call for actors’ participation in designing of these mechanisms to adapt them to a specific context.

We systematically reviewed the literature on the processes designing M&E mechanisms for RRI. The literature review was designed to identify processes where M&E mechanisms were developed to analyse whether and how participation was embedded in such processes or expected to occur in subsequent phases of implementation. Our research question is: why and how do different actors participate or are expected to participate in the design and implementation of M&E of RRI?

To address our research question, we propose to analyse these processes in three stages: problematization, development of research, and implementation in real settings, based on the theory of translation of Callon, Lascoumes, and Barthe (2009). We will use the term actors to refer to the broader range of possible publics and concerned groups interacting and cooperating with the research team in knowledge creation and decision-making.

Our work makes two main contributions. First, we provide the first systematic review of research on the participatory nature of M&E of RRI. Second, we discuss the results to allow reflection on decisions about the involvement of actors in the M&E process for RRI. Our findings have relevance for policy-makers, practitioners, researchers and other actors interested in operationalizing RRI at different levels.

Three arguments in favour of participation in the design of M&E for RRI

The reasons for fostering the participation of stakeholders and the public in the governance of science have been widely discussed (Fiorino 1990; Funtowicz and Ravetz 1993; Stirling 2007). Fiorino (1990) classified arguments for citizen participation into substantive, instrumental and normative arguments. Following this author and Stirling (2007), substantive arguments relate to assuring a better quality of the outputs of the research process, instrumental arguments refer to creating more legitimate and trustful outputs, while normative arguments refer to the right thing to do regarding the procedural approach. We identified three relevant specific arguments for each category to justify participation in the design and implementation of M&E mechanisms for RRI.

First, a substantive argument refers to the fact that including different actors in designing M&E mechanisms and contextualized strategies for their implementation is likely to increase evaluation effectiveness and reduce the risks of negative impacts deriving from their implementation. For example, indicators are a standard instrument for M&E and were proposed in one of the early attempts to develop M&E mechanisms for RRI (Strand et al. 2015). However, using indicators and quantitative metrics for M&E purposes required special attention, as the research policy and evaluation literature advise. These instruments require caution regarding the nature and purposes of indicators (Heink and Kowarik 2010) and the relation between purposes and specific methods and techniques (Molas-Gallart 2012; 2015). The misuse of quantitative indicators and guidelines might result in unintended consequences (Wilsdon et al. 2017), such as goal displacement, biases against interdisciplinarity, reduction of task complexity and changes in institutional arrangements (De Rijcke et al. 2016; Hicks et al. 2015). This is especially relevant to our study as strategies of participation (and contextualization) have been proposed to mitigate potential unexpected and unwanted impacts of indicators (Barré 2010; Ràfols 2019).

Second, the instrumental argument of reinforcing the participation of actors in the development of processes towards M&E for RRI might result in greater ownership of the process. Implementing M&E could lead to the adoption of strategic behaviours by researchers and changes to institutional arrangements (De Rijcke et al. 2016). If these institutional changes work against the policy’s objectives, they produce unwanted effects of implementing M&E mechanisms. To avoid these undesirable effects, greater participation might increase the actors’ sense of ownership and commitment to the policy objective.

Third, a normative argument would refer to the expectation that attempts to design M&E mechanisms for RRI consider the views of different stakeholders and actors and incorporate their values in the different phases of the design process. RRI involves developing research and innovation processes governed by anticipation, reflexiveness, inclusiveness and responsiveness (Stilgoe, Owen, and Macnaghten 2013). The involvement of different actors in the research and innovation processes, especially in decision-making, is common to most RRI definitions and accounts (Burget, Bardone, and Pedaste 2017; Timmermans and Blok 2021; Wickson and Carew 2014). Although some critical views have emerged against the practical consequences of putting participation and inclusiveness at the centre of the RRI perspective (Brand and Blok 2019; van Mierlo, Beers, and Hoes 2020), social engagement and inclusiveness are recognized as a key principle in RRI discourses.

This raises a question about the reasons to have actors participating in the design of M&E mechanisms found in our literature review. In the following section, we present how participation was presented in the initial proposals of M&E for RRI.

Participation in early attempts M&E for RRI

The development of M&E mechanisms for RRI began in 2013. An early paper by Wickson and Carew (2014, 270) proposed a set of quality criteria for RRI. It recommended the strategy of contextualization through the participation of actors as a way to adapt the outputs of the research to the specific context where the criteria would be used:

Of course, both this rubric and the approach we have outlined should remain open to evolve and be adapted, critiqued and amended, as appropriate to different contexts. We specifically see scope for different research groups, innovation organizations, funding bodies and interested stakeholders to engage in analytic-deliberative processes to create their own criteria, and/or indicators for the quality criteria we present, and to articulate these statements across an evaluative scale.

Further development of the M&E for RRI was boosted by two calls issued by the European Commission (EC). In 2014, the EC appointed an expert group, which produced a report proposing a tentative set of indicators and asked users to ‘use this framework to pick and choose those indicators that fit their activities and those of their R & I network the best’ (Strand et al. 2015, 16). Strand and colleagues recommended the participation of actors to create specific sets of contextualized indicators:

Our ambition has been to present the European Commission as well as other actors within the European Research Area with a toolbox from which they may choose and tailor sets of indicators for the monitoring, promotion and development of RRI. It is obvious that one cannot create a prioritised list of indicators without – explicitly or implicitly – prioritising the objectives to be achieved within a particular policy context.

For this reason, we cannot offer a general prioritised list of indicators for actors in the European Research Area. National and regional actors, universities and research institutes, civil society organizations, funding agencies and others should devise their own process of deliberation in order to choose and tailor the indicators proposed in Chapter 2, and add their own indicators according to their own needs, goals and concerns. (Strand et al. 2015, 41)

Also in 2014, the EC published a call for proposals for a four-year study on the ‘Monitoring Responsible Research and Innovation’ (European Commission 2013), the so-called MoRRI project. The MoRRI project resulted in indicators to monitor key areas of policy related to RRI (Peter et al. 2018) to identify differences among European Union (EU) member states.

It is striking that the EC did not request involving other actors to provide input to the process, especially considering that public engagement was one of the policy key areas proposed by its approach to RRI (European Commission 2012). While the call for tenders did not specify specific inclusion requirements, academics involved in the MoRRI consortium stressed the ‘need for caution in the construction and application of indicators in general and for RRI specifically’ (Mejlgaard, Bloch, and Madsen 2019, 198). Mejlgaard, Bloch, and Madsen (2019) highlighted the need to consider the potential systemic effects of indicators (see Hicks et al. 2015) and responsible use of metrics (Wilsdon et al. 2017).

We have seen now that there are three arguments to promote the participation of diverse actors in developing M&E mechanisms for RRI. We also have presented that in early attempts to develop them, there were two approaches, one requesting such actor’s inclusion and another focusing on purely technical development. This leads us to consider to what extent participation has been embedded or foreseen in the rest of the processes towards the development and implementation of M&E for RRI.

Analytical framework and research questions

As previously mentioned, our analysis is structured using a framework based on Callon et al.’s (2009) theory of translation. This framework involves three stages of research, the translation 1, 2 and 3 in the nomenclature used by Callon et al. The early and first stage of the research involves the problematization phase when the research approach is designed, and key decisions such as monitoring and evaluation purposes are taken. The second stage refers to the development of the research itself. The third stage refers to the implementation into real settings of the research outputs.

Regarding translation 3, our analysis will focus on the expected or foreseen participation of actors by the team carrying out the process. A stage-based framework allows us to address the question of why and how the participation of actors occurs in existing studies developing M&E for RRI and identify the patterns of levels of cooperation between researchers and other actors in this literature (see Figure 1). This framework has been used successfully for similar analyses, such as Ràfols (2019), who uses it to identify how developing research evaluation metrics would benefit each research stage.

Figure 1. Analytical framework for the literature review based on Callon, Lascoumes, and Barthe (2009).

Materials and methods

Our systematic literature review identifies the relevant literature and includes a directed content analysis of the text. Tranfield, Denyer, and Smart (2003) point out that systematic review provides methodological rigour and a base of reliable knowledge derived from a range of studies, allowing researchers ‘to map and to assess the existing intellectual territory’ (Tranfield, Denyer, and Smart 2003, 208). Directed content analysis is a deductive method for analysing textual data in which ‘theory or prior research about a phenomenon exists that is incomplete or could benefit from further description’ (Hsieh and Shannon 2005, 1281). We followed the content analysis steps proposed by Kaid (1989).

First, we conducted an iterative discussion of study objectives and theoretical approaches. This was crucial for formulating the conceptual structuring model.

In this stage, data selection was based on a review protocol, which included a search strategy and snowballing related to references and citation tracking in different issues of the Journal of Responsible Innovation, a scientific journal publishing work on RRI. Annex 2 and Figure 2 provide detailed information on the data search and protocol used to select the data. Our final sample includes 37 articles, book chapters and reports. Some of this material referred to the same mechanisms, so we created clusters of documents representing the 25 procedural approaches.

Figure 2. Flow chart of sources selection.

In the initial phases of our study, we constructed an analytical framework based on theory and prior research. Therefore, we opted for a directed content analysis method whose main strength is that ‘existing theory can be supported and extended’ (Hsieh and Shannon 2005, 1283). Our coding strategy included the following steps:

• Reading the articles and highlighting all text relevant to the elements in our analytical model (Figure 1).

• Applying deductive coding to the highlighted citations and constructing a codebook (Annex 2) that included all the elements in our analytical framework, their operational definitions and subcategories based on the existing theory.

• Inductive coding of additional citation information and its inclusion in the codebook.

• Reviewing the inductive codes and deciding whether a new category or a subcategory of the analytical framework was needed.

Findings

We reviewed 37 documents (19 journal articles, 10 book chapters and 8 project reports). Some clusters of documents referred to the development of different mechanisms (methods or tools) developed under the umbrella of the same project and with common members of the team. When this occurred, we clustered them as they referred to the same procedural approach. So, from the 37 documents reviewed, we identified 25 M&E procedural approaches developing mechanisms. A complete list of the reviewed documents and clusters is provided in Annex 1 and a brief description of the diversity of the mechanisms will be presented firstly. Then, the findings regarding the features of participation of actors are presented in the succeeding sections. First, we analyse the arguments presented in the documents to justify the need for actors’ participation in the processes. Then we present a general overview of whether participation has been embedded in the described process (translations 1 and 2), foreseen for future stages (translation 3) or not considered or mentioned in the processes reviewed. Afterwards, we analyse the relationship between the M&E purposes of the mechanisms and the existence or not of participation in the process. Finally, we present in more detail the patterns of participation in translations 2 and 3, which are the phases where more embedded and foreseen participation. We include quotes relevant to each of the findings and mechanisms analysed, which are also provided in Annex 3.

Finding 1: a diverse pool of 25 mechanisms of M&E for RRI

The 25 mechanisms identified are diverse regarding different elements. In the first instance, we found mechanisms consisting of developing M&E tools, including quantitative and qualitative instruments such as qualitative questions and indicators. In contrast, others propose methodologies based on a series of procedural steps. Also, the unit of analysis or evaluand differs among the mechanisms. Some are designed to assess units clearly defined as research and innovation projects, megaprojects or institutions, while others are more ambiguous, referring to research and innovation activities, policies or strategies. Limited cases consider people or complete systems of innovation and countries as the unit of analysis. A basic description of the mechanisms and their unit of analysis are summarized in Table 1.

Table 1. Basic description of the mechanisms and of the participating features.

Code and name	Evaluand	Description of the mechanism	Participation time qualifiers	Participating actors^2	Role of participating actors	Method of participation	Duration of participation
1. Five-stage societal process model	Innovation project	Five-stage societal process, qualitative threshold	T2	Local communities Innovators	Criteria providers	Observation Interviews	Not specified
2. Quality criteria and indicators for RRI	Project	Rubric of qualitative performance indicators, seven quality criteria	T2, T3	Business and Industry Scientist and researchers Experts Various actors foreseen for T3	Criteria providers	Workshop including World-café, small groups and plenary discussion, outcome space posters	Several months and two days’ workshop (17 people) in T2
3. Guide to entrepreneurs […] on RI criteria	Project	Grid with 24 criteria for 4 dimensions, to be assessed in 5 levels. Plus qualitative questions and indicators to support the assessment	–	–	–	–	–
4. RRI Tools – Self-reflection Tool	R&I strategies and activities	Self-reflection tool with six policy agendas, four process requirements and indicators (qualitative questions, including the possibility of create new questions)	T2	Business and Industry CSO Policy Representatives and decision-makers Experts Stakeholders	Criteria providers Design reviewers	Consultation workshops Meetings and one to one online conversations (Focus Group and world-Cafe) Questionnaires and data from users	Several workshops during two months for criteria providers Several events during the project execution for design reviewers. Total 130 people
5. Responsible Port Innovation	(Mega) Project	Methodology (nine steps) and methods (qualitative questions)	–	–	–	–	–
6. EC Expert Group Indicators	RRI initiative (activities, policies)	Six dimensions, with performance (process and product) and perception indicators (plus two dimensions with no developed indicators)	T1, T3	RFO (T1 and 3) RPO (T3) CSO (T3) Policy representatives and decision-makers (T3) Research and project managers and administrators (T3) Scientists and researchers (T3) Publics (T3) Stakeholders (T3)	Commissioning client (T1) Criteria providers (T1, T3) No commissioning client/end-user (T3)	Framing (T1) Deliberation (T3)	Through the process (T1) Not specified (T3)
7. MORRI Indicators	Research System (strong aggregation of institutions)	Monitoring system of indicators with 6 dimensions and 36 indicators	T1, T2	RFO (T1, T2) Scientists and Researchers (T2) Experts (T2) Publics (T2) Stakeholders (T2) Innovators (T2)	Commissioning client (T1) Criteria providers (T1, T2)	Workshop and meetings (T1) Video conference (T2) Visioning workshop (T2)	Through the process (T1, T2)
8. KPIs for the industry	Innovation projects Group of projects	RP1^3: Tool based on quality assessment method providing eight key performance indicators (KPIs), quality scores and scenarios. RP2: Methodological framework with 2 categories, 8 components and 92 key performance indicators	T2, T3	P1: Innovators P2: Business and Industry (T2 and 3) and Companies and SMEs (T2 and 3)	RP1: Criteria provider, No commissioning client/end-user RP2: Criteria providers (T2 and T3); Respondents (T3)	Scoring success-related items	RP1: Not specified RP2: Several activities through the project involving more than 100 stakeholders
9. Res-AGORA Tools	R&I strategies and activities	Monitoring tool (RRI Trends) Stakeholder workshop method (Co-Construction Method) Self-assessment tool (Responsibility Navigator)	T1, T2, T3	RFO (T1)RPO (T2) Business and Industry (T2) CSO (T2) Policy representatives and decision-makers (T2) Stakeholders (T2, T3)	Commissioning client (T1) Criteria providers (T2, 3)	Stakeholder workshops	Various workshops during the project
10. Responsible Project Management	Megaproject	Integrative framework including 6 principles of sustainability, four dimensions of RI and instruments of accountability with customized variables	T3	Stakeholders	Criteria provider No commissioning client/end-user	Deliberation	Not specified
11. PERFORM analytical framework for science education	Project (aggregation in some cases of people performance)	Analytical framework composed by 4 key learning dimensions, 32 assessment criteria and learning outputs and 86 indicators	T2, T3	Students (T2) Stakeholder (T3) Scientists and researchers (T3)	Criteria providers (T2, T3)	Participatory action research approach and workshops (T2)	Eleven workshops in three countries (T2)
12. Framework aligning activities, aspirations and stakeholders	Innovation governance	Framework assessing alignment of two aspirations with three dimensions, five types of activities and stakeholders	–	–	–	–	–
13. RRI maturity models	Institution (Industry)	RP1: 3 Key Performance Indicators (KPIs), 12 sub-indicators and 5 levels of for each sub-indicator RP2: RRI Maturity model and self-assessment tool, 3 categories, 14 components, 5 levels	T2	Business and Industry Stakeholders	Criteria provider Design reviewers Respondents	RP1: Interviews and Case study (1) RP2: Interviews, Bottom-up case study, Stakeholder dialogue, Multi-stakeholder workshops, Large-scale Delphi Study, Focus groups and Case studies	RP1: 30 interviews in 11 countries + 5 interviews for a case study RP2: interviews (30 people); Bottom-up case study (5); Large-scale Delphi Study (150 people); Focus groups (15); Case studies (4)
14. INPERRI AHP participatory approach	RRI initiative (activities, policies)	Methodology based on the use of the Analytical Hierarchy Process technique and a participatory approach	T2, T3	RFO (T2) RPO (T2) CSO (T2) Companies and SMEs (T2) Policy representatives and decision-makers (T2) Research and project managers and administrators (T2) Scientists and researchers (T2) Experts (T3) Stakeholders (T3)	Criteria providers (T2, T3)	RP1: Face-to-face and online interviews RP2: Participatory workshop	RP1: 12 interviews (one per expert) RP2: 1 day participatory workshop
15. Analytical framework of RRI in Smart Farming	Project	Analytical framework composed by four RI dimensions and nine indicators	T3	RFO RPO CSO Business and Industry Companies and SMEs Policy representatives and decision-makers	Criteria providers	Deliberation	Not specified
16. ENRRICH Peer evaluation approach	Project	Peer evaluation approach	T2	Students	Criteria providers	Participatory techniques that are built on a bottom-up approach (discussion)	Not specified
17. Responsible Innovation in Health Tool	Health Innovation	Screening (four criteria), assessment (nine attributes, five value domains) and rating (scoring system with two components)	T2	Innovators (T2) Experts (T2, T3)	Criteria providers (T2) Evaluator coordinator (T3)	Delphi study Interviews	Two-round Delphi study with 19 experts in the second round Interviews (23)
18. RRI index	Company	RRI Index with 6 dimensions and 11 components	–	–	–	–	–
19. COMPASS self-check tool	Company	Self-assessment tool, 4 sections, 43 questions, 249 answer options	T2	RFO RPOCSO Companies and SMEs Experts	Design reviewers	Consultation to experts Interviews Group discussions	One consultation to experts 84 participants in interviews and group discussions and 30 individuals in a second-round of feedback
20. Future-oriented RRI evaluation	R&I strategies and activities (platforms)	Methodology based on four steps	T2, T3	Experts (T2) Stakeholders (T2, T3)	Design reviewers (T2) Criteria providers (T3)	Workshop	One workshop with 20 people (T2)
21. RRI intensity level	(ICT) Project	Method of three steps to ex-ante assessment of technology readiness level and innovation potential	–	–	–	–	–
22. Responsible creativity and innovation scale	People (in business context)	Scale with seven items	–	–	–	–	–
23. Reflexive Monitoring in Action for RRI	Research Project	Methodology (4 criteria, 17 sub-criteria and inviting questions)	T2	Experts (T2) Stakeholders (T2) Evaluator specialist (T3)	Criteria provider (T2) Evaluator coordinator (T3)	Experts interviews Stakeholder consultation workshop	Through the process (T2)
24. Qualitative Multi-criteria Self-Questionnaire	Project	Methodology; self-assessment; 6 questions and 31 sub-questions (criteria)	T3	Business and Industry	No commissioning client/end-user Respondents	Questionnaire structure through a MCDA technique	Not specified
25. Societal Readiness Thinking Tool	Research Project	Stage gate model for projects with reflective questions in social dimension	T2	RPO Scientists and Researchers Policy representatives and decision-makers	Criteria providers Designer reviewers	Design Sprint Focus groups Thinking Aloud interviews	Two-days design sprint 6 Focus groups with 38 participating actors 6 Thinking Aloud interviews

Finding 2: the primary motivation to promote contextualization through participation is to increase the evaluation effectiveness

The three arguments discussed in favour of participation have been identified in the studies analysed (Table 2). The most common argument for participation is the substantive one, which is mentioned in the documents of nine of the mechanisms (see Annex 3 for proof quotes). References to the substantive argument relate to increments in the evaluation effectiveness of the mechanisms and identification and reduction of risks, negative impacts and trade-offs of their implementation.

Table 2. Motivation to promote participation, references to participation in the process and primary and secondary purposes.

	Motivation to promote participation			References to stages for participation			Primary and secondary purposes
	Substantive argument (9)	Instrumental argument (6)	Normative argument (6)	T1 (3)	T2 (15)	T 3 (10)	Knowledge creation (6 + 7)	Decision making and accountability (8 + 9)	Learning and reflexivity (10 + 5)	Trust and cooperation (1 + 3)
1. Five-stage societal process model	X	X			X		X
2. Quality criteria and indicators for RRI	X				X	X	X	X	X
3. Guide to entrepreneurs […] on RI criteria								X
4. RRI Tools – Self-reflection Tool			X		X		X	X	X
5. Responsible Port Innovation							X
6. EC Expert Group Indicators		X		X		X		X	X	X
7. MORRI Indicators				X	X		X	X	X
8. KPIs for the Industry	X				X	X	X	X
9. Res-AGORA Tools			X	X	X	X	X	X	X	X
10. Responsible Project Management	X					X		X	X
11. PERFORM analytical framework for science education	X	X	X		X	X	X		X
12. Framework aligning activities, aspirations and stakeholders							X	X
13. RRI Maturity Models	X				X		X	X	X
14. INPERRI AHP participatory approach	X	X	X		X	X		X	X
15. Analytical framework of RRI in Smart Farming		X				X	X
16. ENRRICH Peer evaluation approach	X		X		X				X
17. Responsible Innovation in Health Tool					X		X	X
18. RRI index								X
19. COMPASS self-check tool					X			X	X
20. Future-oriented RRI evaluation		X	X		X	X			X	X
21. RRI intensity level							X	X
22. Responsible creativity and innovation scale								X
23. Reflexive Monitoring in Action for RRI	X				X			X	X
24. Qualitative Multicriteria Self-Questionnaire						X			X	X
25. Societal Readiness Thinking Tool					X				X

Notes: In purposes bold refers to primary purpose.

In some cases, the substantive argument refers to participatory strategies, including the innovators or end-users of the mechanisms, as exemplified in this quote from mechanism 8 ‘KPIs for the industry’.

The KPI we use below are based on earlier studies, in which these KPIs are identified, analysed and validated (Flipse et al., 2013a,b). Their relevance to RRI needed to be discussed in collaboration with the organization in which the KPIs are identified; […]. Namely, the KPIs only become relevant when people talk about these in relation to their work, thereby actively considering also the socio – ethical and socio-economic aspects of their work (reflexivity, anticipation) and translating these considerations into concrete actions (inclusion and responsiveness). (Flipse et al. 2015, 138)

In other cases, integrating views from actors beyond the end-users to operationalize the M&E criteria was identified as vital for ensuring effectiveness. This applies to mechanism 1, ‘Five-stage societal process model’ proposed by Voeten et al. (2014), which considers it necessary to integrate the innovators and the local community in the definition of threshold values to avoid the imposition of Western normative frameworks.

In regard to the first issue, from our assessment as researchers, we were inclined to asses that Bat Trang village could be labelled as experiencing responsible innovation. During our discussions in later rounds of validating our tentative field assessments, we were confronted with the views of innovators and villagers in the other villages who had a different judgement than us about the whether the outcomes were negative or positive. […] Any attempt that we – as western researchers, not living in the village – might make to define threshold values for these criteria, would involve imposing our normative framework about what is acceptable and what is not. (Voeten et al. 2014, 165)

The instrumental argument was identified in six mechanisms and refers to promoting actor participation to increase the ownership and support the policy objectives underlying the M&E exercise by exploiting the actors’ performative function. Among the six mechanisms referring to this argument, we found mechanism 6 ‘EC Expert Group indicators’, a finding that was expected as this strategy was identified prior to the review during the research design of this paper.

An additional value of involving stakeholders in indicator development will be the fact that if the stakeholders become the ‘owner’ of the monitoring they will be more ready to accept this as a valuable instrument to improve their performance. (Strand et al. 2015, 5)

The third set of justifications, the normative arguments, refers to the alignment with RRI theory and the demand for inclusiveness and public engagement. Six mechanisms refer to this type of argument. The following quote exemplifies this type of argument:

In the development of the tool we wanted to apply the RRI approach itself, thus involving members from different stakeholder groups from the very first draft to the final prototype. (Schrammel et al. 2016, 5)

In two cases (Mechanism 1 ‘Five-stage social process model’ and 4 ‘Self-reflection Tool – RRI Tools’), the team’s reflections about the relevance of promoting participation in the process have led to changes in their initial plans and their methodological design.

This deliverable is a follow-up of the working definition that can be found in D1.1. Although this deliverable was originally scheduled for month six of the RRI Tools project (close after submission of D1.1), we decided to re-conceptualize its role and meaning to some extent, postponing it to after the Stakeholder Consultation Workshops that were held throughout Europe in months nine to eleven (i.e. September to November 2014). We considered it crucial that the reflections of the participants of the stakeholder consultation workshops informed the criteria displayed here, so as to give the criteria a firmer ground in RRI practices throughout Europe. (Kupper et al. 2015, 8)

Finding 3: participation is more common in the latter than early phases of the process

Analysing the identified mechanisms shows limited participation in the early phases of research problematization (translation 1). In contrast, participation tends to be more commonly embedded during the research development stage (translation 2) and expected in the application of the implementation in real settings (translation 3) (Table 2).

We observed only three mechanisms referring to inputs during translation 1 from actors other than the research team. These three cases were the only explicit references identified to the role played by external actors in this research phase. The participation in these three cases consisted of funding organizations acting as commissioners of the research (see Annex 2 for the definition of actors’ roles). We identified explicit references to these funding organizations providing input in translation 1 in mechanisms 6 (EC Expert Group indicators), 7 (MoRRI indicators) and 9 (Res-Agora Tools), both of which were funded by the EC. In all these cases, participation in the problematization phase consisted of commissioning the design of the M&E mechanisms. It was limited to setting research evaluation and monitoring priorities, purposes or criteria.

Although we did not identify any other direct references to the role of funding commissioners, we can assume that all of the processes that received funding as a result of calls for research that steered the research questions and purposes of the mechanisms included at least limited participation of the funding agency in setting those research priorities. This applies, at least, to the eight other cases that received funding under the EC 7th Framework and the Horizon 2020 programmes (mechanisms 2, 4, 8, 11, 13, 16, 19 and 25). However, the level of participation by the funding agency in the problematization phase would be limited to the terms of the call for proposals in which research priorities and research questions are established. We also found that participation in translation 1 did not include other actors’ participation apart from the funding agency in the problematization phase.

In the case of references to embedded or planned participation in translation 2 (development of the research) and translation 3 (implementation in real settings), this involved 19 of the 25 mechanisms analysed (Table 2). Fifteen mechanisms involved the participation of actors in translation 2, and other ten mechanisms foresee participation in the application of the research outputs in real-world contexts (translation 3). We identified six cases of participation in both phases 2 and 3 (mechanisms 2, 8, 9, 11, 14 and 20).

Finally, six mechanisms identified neither embedded nor planned participation in research stages (mechanisms 3, 5, 12, 18, 21 and 22). As we aimed to explore the features of participation in the existing M&E methods and tools, we have not included such mechanisms in all the analysis of findings regarding participation (finding d). However, they will be mentioned in the following finding about the relationship between participation and the purposes of the M&E mechanism.

Finding 4: participation is most commonly linked to ‘Learning and reflexivity’ and ‘Trust and cooperation’ purposes

In our analysis of the problematization phase, we coded the purposes of the M&E mechanisms – several of the sample documents referred to more than one purpose for a particular mechanism. We, therefore, included subcategories for a primary and secondary purposes, identified by comprehensive analysis of the original codification. For example, in the case of mechanism 9 (Res-Agora Tools), we found references to all the categories of the purposes and three mechanism components with complementary functions: RRI Trends (monitoring tool), Co-Construction Method (stakeholder workshop method) and Responsibility Navigator (self-assessment tool). To identify the primary purpose, we analysed the general objective of the project and the role and relation between these components. We identified that the RRI Trends and Co-Construction Method supported the design of the Responsibility Navigator, whose primary purpose was included in the ‘Trust and Cooperation’ category.

The most frequent purposes (see Book of Codes in Annex 2 for the definition of the categories) are ‘Learning and Reflexivity’ (10 as primary purpose and 5 as secondary), ‘Decision Making and Accountability’ (8 as primary purpose and 9 as secondary), ‘Knowledge Creation’ (6 as primary purpose and 7 as secondary) and ‘Trust and Cooperation’ (1 as primary purpose and 3 as secondary) (Table 2).

A special mention should be made regarding mechanisms 6 (EC Expert Group indicators) and 7 (MoRRI indicators) and their primary purposes. These mechanisms are among those that explicitly include the participation of the funding actor in the problematization phase, as mentioned in Finding 3: Participation is more common in the latter than early phases of the process. In these two cases, the funder’s (EC’s) primary purposes are included in the category ‘Decision making and accountability’, but the authors of the documents analysed refer to the importance of ‘Learning and reflexivity’ as the primary purpose of the M&E exercise.

Some specific patterns emerge from our analysis of the relation between the primary purposes and the existence of embedded or planned participation. Participation seems to be linked to ‘Learning and Reflexivity’ and ‘Trust and Cooperation’. The ten mechanisms included in the ‘Learning and Reflexivity’ category involved embedded or planned participation in translations 2 or 3, and 3 mechanisms (9, 11 and 20) included participation in both research phases. Mechanism 9 (Res-Agora Tools), the only mechanism in the ‘Trust and Cooperation’ category, embeds participation in translation 2 and plans participation in translation 3. None of the six mechanisms where participation was not considered in any of the stages (3, 5, 12, 18, 21 and 22) was aimed primarily or secondary at ‘Learning and Reflexivity’ or ‘Trust and Cooperation’. Instead, they were aimed at ‘Decision Making and Accountability’ (3) and ‘Knowledge Creation’ (3).

Finding 5: in translations 2 and 3, the participation of actors as criteria providers is aligned with conceptualization through participation

In the analysis of participation in translation 2, we found three roles of actors participating in translation 2 (see annex 2 for further details on roles): criteria providers – those that participate in the process to define the operationalization and evaluative criteria-, design reviewers – those that participate in the process to provide feedback to the M&E design (i.e. usability tests), and respondents or data providers – those that provide evaluation information.

The participation of actors as criteria providers during the research development is aligned with the principles of engagement in RRI. It implies consideration of actors as providers of knowledge and value to the design of M&E mechanisms in different types (different intensity and time commitment) of participation. Participation as design reviewers is focused mainly on providing inputs into the usability of the research outputs and less on involvement in the design and decision processes. In these cases, participation might respond more to the correct research method application than to alignment with RRI principles. Similarly, when the actor participating in translation 2 acts as a respondent, their role is limited to providing the necessary information to perform the assessment or evaluation, so it has methodological importance. However, it does not reflect, per se, an alignment with the principles of RRI. We will therefore analyse in more detail the features of participation in translation 2 of actors as criteria providers. The existence of actors participating in this role responds to what we call contextualization through participation: strategies and processes that allow or call for actors’ participation in designing these mechanisms to adapt them to a specific context.

In 13 mechanisms, actors were involved as criteria providers of input and knowledge to develop the evaluative and monitoring criteria (1, 2, 4, 7, 8, 9, 11, 13, 14, 16, 17, 23 and 25). In these cases, actors were involved through a consultive or deliberative approach in the criteria design. Proof quotes can be consulted in Annex 3, including:

To arrive at a comprehensive model of RRI and its criteria, we engaged in a process of iterative conceptual modelling (Figure 9.1, and see Klaassen et al. 2017 for a more extensive description). Central to this methodology for concept development are different and disparate forms of expertise, confronted in a series of iterative steps which, in this case, sought to answer our question ‘What is RRI?’. (Klaassen et al. 2020, 225)

In some cases, the participation of actors in translation 2 affects the dimensions of responsiveness and reflexiveness in the development of the process. For instance, as an example of responsiveness, including actors as criteria providers in translation 2 implied a change to the initial research plan described in finding a (page 13) on mechanisms 1 and 4. In other cases, researchers reflect on the risk of imposing normative frameworks derived from the non-participation of other actors in mechanism 1 ‘RI Conceptualization’ or regarding the degree of participation in mechanism 11 ‘PERFORM project’.

At this stage of the project (Month 7), participant students have been already included in the assessment design, through the explorative workshops and the identification and validation of criteria and indicators relevant to them. This is a rather basic level of participation, represented by the implementation of methods to gather participants’ opinions and insights about topics of their own interest to be included in the assessment design; such as exploratory workshops or focused discussions. (Heras et al. 2016, 59)

The intensity of actor participation as criteria providers in terms of time and number of actors involved varied across mechanisms (see Table 1). Sometimes, it implied multiple meetings over several months, many participants or using techniques such as Delphi studies. In other cases, they involved explorative sessions with limited participants and short duration.

In translation 3 there are also different roles foreseen for actors beyond the research teams in the mechanisms analysed. Similarly, to translation 2, there are several cases in which actors are expected to participate in adapting the mechanisms to the context of use by adopting the role of criteria providers through the process that we refer to as contextualization through participation. Additionally, references to the participation of actors acting as evaluator coordinators, final end-users, respondents or data providers are also foreseen in this phase.

Regarding actors participating as criteria providers, we found nine references to planned contextualization through participation in the implementation phase in real settings (2, 6, 8, 9, 10, 11, 14, 15 and 20). It is exemplified in the following quote (see other proof quotes in Annex 3):

Thus, there are critical questions that need to be kept in mind and solved when the model is applied. […] Fourthly, the evaluation should pay careful attention to the identification and engagement of stakeholders to ensure, not only fair and wide, but also effective participation. And finally, the far from simple challenge of functional indicators and their measurement needs to be addressed to provide appropriate follow-up indicators and incentives for RRI. (Nieminen and Ikonen, 2020: 265)

The commitment to the participation of actors in this phase is especially relevant in mechanism 9, as a specific method (the Res-Agora Co-Construction Method) is provided for facilitating cooperation with actors during the implementation of the tool in real settings.

Among the other roles identified in translation 3, reflecting on the strategy proposed to involve experts and independent assessors in the implementation phase. We have coded this role as evaluator coordinator, and it is referred to in mechanisms 8 and 17. In the case of ‘KPIs for the Industry’ it is pointed out the need to reflect on the role of external assessors to provide an independent assessment that complements self-assessments. The ‘Responsible Innovation in Health Tool’ provides recommendations on the skills that the person carrying out the assessment should fulfil to use the mechanism adequately.

We also found contextualization to adapt the M&E mechanism to the context of the unit of analysis but not linked strictly to participation. The expectation to contextualize the research output was limited only to the adaptation of the tool to the characteristics of the unit of analysis in the evaluation and monitoring exercise.

Regarding the patterns of contextualization and participation identified in translation 3, contextualization through the participation of actors as criteria providers is well aligned with RRI. This strategy allows different actors to participate actively in adapting the mechanisms to the application context. The second and third patterns identified, the participation of experts in research in implementation and the contextualization of the unit of analysis, are relevant from an evaluation perspective. However, these types of contextualization and participation do not imply, per se, integration of the RRI principles regarding actors’ involvement.

Discussion

In this section, we will discuss our findings in light of existing literature. We want first to contextualize our research to two seemingly opposed directions regarding the expected contextualization of M&E mechanisms for RRI. On the one hand, recent literature points out the need for monitoring and evaluating RRI at the territorial level (Völker et al. 2023) and through engaging stakeholders within processes that facilitate contextualized monitoring and evaluation for RRI (Holtrop et al. 2022). In the opposite direction, there are also calls for developing global indicators that overcome the contextualized approaches of M&E (Jensen 2022). Hence, our work contributes to the first approach, by exploring the role of actors’ participation in processes towards M&E mechanisms for RRI and discussing its implications. To our knowledge, there is just one previous study reviewing M&E methods and tools for RRI by van de Poel (2020), and its focus is on identifying pitfalls in M&E for RRI. So, we discuss our findings compared with previous literature on features and challenges of participation in RRI. The RRI scholarship has been defined as ‘highly reflexive’ (Völker et al. 2023, 05). We want to contribute with our analysis to open a window for further reflexivity in M&E research and development for RRI.

The discussion is structured into three subsections. First, we explore the implications of having found 25 processes towards developing M&E mechanisms for RRI. Then we discuss the findings from our analysis on the arguments used justify participation, the observed features of participation and the relation of those features with the monitoring and evaluative purpose of the M&E mechanisms. Finally, we present a set of recommendations to integrate some learnings from our findings in future research and development of M&E mechanisms for RRI.

Twenty-five processes towards M&E mechanisms for RRI

We found processes developing 25 M&E mechanisms, which confirm the growing interest in developing these methods and tools in parallel to the growing interest in the RRI notion. This fact aligns with Jensen’s observation (2022) of the uneven amount of measurement initiatives for data collection and analysis of RRI.

The unit of analysis of the mechanisms is diverse. In some cases, these units are well-defined (such as projects and companies), but in others, they ambiguously refer to strategies or activities. In its analysis of M&E mechanisms for RRI, van de Poel (2020) defines the innovator as ‘the actor that is the object of the RRI assessment’ and assumes that it is ‘a specific organization rather than to the entire knowledge or innovation system’ (2020, 341). In our analysis, most mechanisms focus on project or company levels. However, we also found some examples of mechanisms addressed to evaluate people or innovation systems, innovation governance or countries. In this regard, Jensen (2022) advocates for ‘establishing globally relevant and usable indicators is challenging but essential given the global nature of science’. So, we identify divergent positions and expectations on who might be the object of assessment in the existing literature. This might be due to the co-existence of diverse approaches to RRI and the difficulty of assigning responsibility roles in the R&I system.

The growing existence of M&E mechanisms can be considered a result of efforts to institutionalize the concept of RRI.¹ A relevant amount of the processes analysed received funding from calls for proposals steering specific research priorities (such as, at least, the 7th Framework Programme and Horizon 2020 of the EC). We coded the role of funding agencies commissioning the development of M&E mechanisms for RRI as commissioning clients, capturing a similar function to the one proposed by van de Poel when defining the regulator or standard setter as the actor setting standards that ‘can also concern how RRI assessment is to be carried out and by whom’ (2020, 342). Therefore, we could infer that growing funding has increased the number of teams interested in the topic. These two elements combined have, in turn, increased the number of developed mechanisms.

Our paper examined the actors’ participation in the processes towards M&E for RRI. We hypothesize that these processes can be considered an object of analysis from an RRI perspective, providing new inputs on challenges for the participation of actors. Therefore, we wanted to explore how participation as a proxy of inclusiveness in terms of Stilgoe, Owen, and Macnaghten (2013), or public engagement in terms of the European Commission (2012), was embedded or foreseen in those processes. This strategy aligns with Smith et al.’s (2021) suggestion that RRI should be considered a form of knowledge production. It also relates with Rip’s (2014, 2) notion of RRI as social innovation due to ‘the roles and responsibilities of actors and stakeholders in research and innovation’. So, we examined the extent of contextualization of M&E mechanisms based on actors’ participation in these processes. RRI narratives emphasize public and stakeholder engagement and inclusion (Burget, Bardone, and Pedaste 2017; Fraaije and Flipse 2020; Timmermans and Blok 2021), but our findings show that contextualization through inclusive participation of actors in the design and expected implementation of M&E processes for RRI is not equally distributed in terms of the phases where it occurs and with relation to the evaluative purposes of the M&E mechanisms.

Arguments to justify participation

The first feature analysed on the participation of actors in M&E mechanisms for RRI refer to the various arguments proposed to justify the participation of different actors in the design of M&E mechanisms for RRI. Our results show that the arguments in the documents analysed include substantive, instrumental and normative arguments. The substantive argument was the most mentioned, with nine processes referring to it, and the instrumental and normative were mentioned in six cases each. We could have expected higher levels of consideration of the normative argument in the documents reviewed since this argument refers to a core aspect in the different accounts of RRI. Following some of the more common operationalizations of RRI (European Commission 2012; Stilgoe, Owen, and Macnaghten 2013), both public engagement and inclusion are considered key elements or dimensions of RRI. Even though there are some cases where the normative argument is mentioned, it is found in 6 out of 25 cases.

On the other hand, most mechanisms directly referring to arguments (any of them) for the participation of actors also embedded or foreseen participation at some stage of the process. Participation was found embedded or planned in 19 of the 25 mechanisms analysed.

Features of participation in M&E mechanisms for RRI: when and how

Regarding the moment participation occurs, we have observed a trend towards a higher consideration of participation in the latter than early phases of the design and implementation process. Translation 1 refers to the early stage when the monitoring and evaluation questions are defined, and critical theoretical and methodological decisions are taken, such as the determination of the M&E purposes and the unit of analysis of the M&E exercise.

Our findings suggest that participation in the early stages is limited to the commissioners establishing the M&E priorities, purposes or criteria. The lack of diversity of actors’ participation in this early phase (or of documentation found referring to them) reduces opportunities for discussions (and contestation) in defining the aim and purposes of M&E. This has implications on two levels. On the one hand, participation of actors in RRI is expected to occur since the early stages of the process (van den Hoven et al. 2013; von Schomberg 2013) and to be, among others, inclusive, continuous and open to set framing issues (Bauer, Bogner, and Fuchs 2021). This would imply allowing space for contestation and dissent about the monitoring and evaluative purposes of the mechanisms, opening up alternatives of appraisal (Stirling 2007) and promoting societal alignment to better management of uncertainty at early phases (Ribeiro et al. 2018). Taking this argument to the extreme, contestation in translation 1 might even imply deciding not to develop or implement an M&E mechanism in a particular context if the possible negative effects did not counterbalance the positive ones. This possibility would fit into the concept of responsible stagnation used by de Saille and Medvecky (2016), which is explained in these terms ‘It should be noted that a better “output” does not necessarily mean a better “product” – the better outcome may be no product’ (Ten Holter 2022, 282).

On the other hand, societal engagement is expected to improve R&I decisions, allowing ‘participants to contribute their knowledge, experiences and perspectives and raise questions and concern about the direction of R&I’ (Bauer, Bogner, and Fuchs 2021, 352). The decisions taken in the early phases of the process around the definition of purposes of M&E are of key importance for the configuration of roles and context-specificity in the latter phases (van de Poel 2020). Adapting M&E mechanisms to their objective, based on contextualization and participation strategies, aligns with Hicks et al.’s (2015) recommendation to contextualize indicators with their evaluative purposes and to consider the socio-economic and cultural contexts of use and potential variations according to the research field or epistemic context. This type of adaptation is considered vital to contextualizing them to ‘geographic, social and epistemic conditions’ and to the ‘value preferences of the stakeholders involved’ (Ràfols 2019, 15).

In the latter stages of the process (translations 2 and 3), we found that participation refers to actors acting as criteria providers that through strategies of consultation and deliberation contribute to the conceptualization and definition of the M&E criteria. The strategies undertaken in translation 2 to embed actor’s participation in the process to operationalize the M&E criteria vary in terms of duration, number of actors participating and techniques used. We found some exemplary cases of high levels of commitment and involvement of stakeholders in terms of the range of actors involved and time dedicated to defining M&E criteria. The more or less participation of actors in this phase might be due, among other issues, to the resources available by the research teams that could limit or facilitate the strong involvement of actors.

In the case of translation 3, we analysed whether contextualization through participation was expected to occur if the research outputs developed in translation 2 were applied in practice. Our objective was to identify if the actors’ involvement before using the outputs of translation 2 was expected to adapt the mechanisms to the particular context. We also identified some exemplary cases of contextualization of the mechanisms through the planned participation of actors. In this case, we analysed how the research and developing teams envisioned the participation of actors to contextualize the research outputs (the tools or methodologies developed) to be used in real settings. In translation 3, the scarcity of resources could not justify the lack of consideration of actors’ participation in translation 3 as it is a foreseeable exercise.

Relation between the evaluative purposes and participation

Another interesting finding is the relation between the evaluative purposes and the presence or foreseen participation in the process. The most frequent purposes in our analysis are ‘Learning and Reflexivity’ (10 as primary purpose and 5 as secondary), ‘Decision Making and Accountability’ (8 as primary purpose and 9 as secondary), ‘Knowledge Creation’ (6 as primary purpose and 7 as secondary) and ‘Trust and Cooperation’ (1 as primary purpose and 3 as secondary) (Table 2). A common feature regarding the purposes of the analysis of van de Poel (2020) is that some research approaches try to respond to multiple purposes. Van de Poel argues that mechanisms should clearly respond to one rationale and clearly state them to address possible pitfalls.

We could expect that the features of participation were related to the purposes and evaluative aspects of the M&E exercise. Greene (2007, 18) refers to ‘evaluation purposes [that] can be roughly aligned with different philosophical paradigms, but more importantly, are aligned with different audiences for evaluation studies’. Therefore, higher levels of participation by a broader range of actors could be expected, as shown in our findings, in mechanisms that respond to ‘Learning and Reflexivity’ and ‘Trust and Cooperation’ purposes (Ligero Lasa 2015). However, an RRI approach should involve a certain level of participation or, at least, critical reflection about lack of participation, regardless of the research objective, since ‘engaging a range of stakeholders for the purpose of substantively better decision making and mutual learning’ characterizes all of the proposed definitions of RRI (Wickson and Carew 2014, 255). Future research on the M&E for RRI would benefit from higher reflection in this regard to work towards a ‘more ambitious vision for RRI’ (Owen, von Schomberg, and Macnaghten 2021, 223) and engage, as far as possible, with RRI as the site for ongoing debate, the site of praxis and the site for politics, as proposed by these authors.

From another point of view, when mechanisms have an accountability or decision-making purpose, the need to develop instruments that allow comparison might imply less space for participation. Making global and representative actors participate might be challenging when comparisons are necessary. In this line, Jensen (2022) calls for ‘high quality indicators’ that ‘given the global nature of science, […] need to be relevant to countries across all world regions’. So, when the purpose is comparing, and the levels are large units of analysis such as countries, there might be less scope for participation to happen.

Recommendation for future research and development of M&E for RRI

From our analysis, we conclude that the level of participation of different actors in developing M&E mechanisms for RRI could be strengthened, especially in the early phases of the research. Increasing participation of actors, especially those directly interested and affected by the M&E exercise, would better align the principles of the RRI notion with the M&E theoretical and practical exercises.

As one key funding agency for this type of processes, the European Commission is the only identified actor participating in decisions about the purposes and units of analysis for the mechanism, apart from the team of researchers and authors developing these mechanisms. The research and development of M&E mechanisms for RRI would benefit from discussing in detail the role of embedded or foreseen participation through the entire process, facilitating new opportunities for creating higher reflexivity in the field. We see scope for integrating (and reporting) increased participation in decision-making and co-producing framings with different actors, especially in translation 1. Considering that M&E mechanisms for RRI are tools ‘entwined with the political and organizations context’ in which they operate (Völker et al. 2023, 05), we suggest opening up space for contestation and increasing social appraisal (Stirling 2007). To this concern, implementing evaluative conversations since the early phases of the process, as proposed by Holtrop et al. (2022) and identifying structured approaches for the management of stakeholders’ involvement and decision-making (Ten Holter 2022) could reinforce the reflexivity and responsiveness of the research in this field. By using these methods and tools, teams developing M&E mechanisms could better explain the levels of stakeholders’ participation and the decision-making process since the early phases of the research, providing interesting thoughts for the RRI community.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the Ministry of Economy Industry and Competitiveness (Spain) under Grants CSO2016-76828-R and BES-2017-081141.

Notes on contributors

Irene Monsonís-Payá

Irene Monsonís-Payá is an early career researcher at the Institute of Innovation and Knowledge Management, Ingenio (CSIC-UPV), Universitat Politécnica de València. In her PhD dissertation, she focuses on processes of production of monitoring and evaluation instruments in the field of Responsible Research and Innovation. Her academic and professional interests include issues in the intersections of sustainability, science, management and education.

Edurne A. Iñigo

Dr Edurne A. Iñigo is an Assistant Professor at Deusto Business School, San Sebastian, Spain. She was a postdoctoral researcher at Wageningen University, where she conducted research leading to this manuscript focusing on responsible innovation in business. Her research focuses on the relationship between responsibility, ethics and sustainability in innovation, with a particular emphasis on the design of transitions towards sustainable systems. She has published in Technological Forecasting and Social Change, Industrial Marketing Management and Journal of Cleaner Production, among others. Before joining academia, she worked as a consultant for governance, business and legal aspects related to sustainability for the public and private sectors.

Vincent Blok

Dr Vincent Blok is Associate Professor in Philosophy and Ethics of Technology and Innovation at the Philosophy Group, Wageningen University (The Netherlands). In 2005 he received his PhD degree in philosophy at Leiden University with a specialization in philosophy of technology. Blok's research group is specialized in Business Ethics, Philosophy of Technology and Responsible Innovation. Together with six PhD candidates and four Post-docs, he is involved in several (European) research projects. His books include Ernst Jünger's Philosophy of Technology. Heidegger and the Poetics of the Anthropocene (Routledge, 2017) and Heidegger's Concept of philosophical Method (Routledge, 2020). Blok published over hundred articles in high ranked philosophy journals like Environmental Values, Business Ethics Quarterly, Synthese and Philosophy & Technology, and in multi-disciplinary journals like Journal of Cleaner Production, Public understanding of Science and Journal of Responsible Innovation. See www.vincentblok.nl for more information about his current research.

Notes

1 We thank one of the reviewers for drawing our attention to this point.

2 Acronyms: CSO (Civil Society Organizations), RFO (Research Funding Organizations), RPO (Research Performing Organizations).

3 P: Process.