Multidimensional and multiscale assessment of agroecological transitions. A review

ABSTRACT

Assessing benefits and limits of agroecological transitions in different contexts is of foremost importance to steer and manage agroecological transitions and to feed evidence-based advocacy. However, assessing agroecological transitions remains a methodological challenge. The objective of this research was to investigate to what extent existing multiscale and multidimensional assessment methods were suitable to assess agroecological transitions. We used a literature review to identify and select 14 existing multiscale and multidimensional assessment methods related to sustainable or resilient agriculture. We then analyzed these 14 methods according to five evaluation criteria that reflected key requirements for assessing agroecological transitions: 1) be adaptable to local conditions, 2) consider social interactions among stakeholders involved in the transitions, 3) clarify the concept of agroecology, 4) consider the temporal dynamics of the transitions to better understand barriers and levers in their development and 5) use a participatory bottom-up approach. The methods adopted different approaches to consider each evaluation criterion, but none of them covered all five. The two evaluation criteria most often employed were the adaptability to local conditions (used by 13 of the methods) and the consideration of social interactions (used by all 14 of the analyzed methods). To be adaptable, methods mobilized generic guidelines with flexible content and/or included a contextualization phase. For social interactions, most methods mobilized social-related indicators, and two included stakeholder mapping. Two methods clarified the agroecological concept by mobilizing different sets of principles. Two other methods considered temporal dynamics of the transitions, mobilizing a trajectory of change to understand barriers and levers in their development. Finally, seven methods adopted a bottom-up participatory approach, involving stakeholders in both their development and use. To balance the existing trade-offs between the evaluation purpose, the time requirement and the level of participation in the different approaches adopted by the 14 methods studied, we suggest combining some of the approaches in a complementary mode to cover all 5 criteria and therefore improve the assessment of agroecological transitions.

KEYWORDS:

• Agroecology
• Evaluation
• Tool
• Method
• Framework

1. Introduction

Facing daunting economic, environmental, demographical and political challenges, agriculture and rural areas are at a crossroads and need in-depth transformation (HLPE, 2019; IAASTD, 2009). Agroecology is often viewed as one of the most promising approaches to overcome these manifold challenges and achieve resilience and sustainable development in agricultural systems (FAO, 2018; IAASTD, 2009; Leippert et al., 2020). Building on ecological concepts and principles that reduce the dependency towards external resources and, in particular, the use of chemical inputs (Altieri, 1995), the focus of agroecology has evolved in recent years (Gliessman, 2016; Wezel & Soldat, 2009). Beyond agricultural practice at field, farm and landscape scales, agroecology now encompasses the ecological, economic, social, political and cultural aspects of food systems (Barrios et al., 2020; Francis et al., 2003; Wezel et al., 2020). It considers the relationship between food production and society at large (Francis et al., 2003) and is at the heart of territorial development challenges (Wezel et al., 2016).

Agroecology is a way of redesigning food systems to achieve ecological, economic and social sustainability (Gliessman, 2016). Agroecological transitions correspond to a progressive but systemic transformation through the ecologization of agriculture and food systems (Magrini et al., 2019) which implies changes in both practices and organizational aspects (Bergez et al., 2019; Tittonell, 2020), and concerns multiple stakeholders (Magrini et al., 2019). These changes involve gradual and often discontinuous trajectories of change (Tittonell, 2020), resulting in different levels of transition in the process of converting from simplified industrial agroecosystems to complex and diversified agroecological systems (Duru et al., 2015; Gliessman, 2016). Changes in agroecological transitions are both multidimensional and multiscale (Côte et al., 2019; Duru et al., 2015; HLPE, 2019).

Assessing agroecological transitions requires considering the multidimensionality of the transition towards sustainability (Trabelsi et al., 2019; Wiget et al., 2020) and the multiple scales of the changes (Magrini et al., 2019). A variety of agroecological transitions have emerged and been described worldwide (Altieri & Nicholls, 2012; De Schutter & Vanloqueren, 2011; Pretty, 2008). Despite high expectations regarding their potential and performances, there is currently a lack of analysis regarding both the barriers and levers in their development, and their multidimensional impacts (GTAE, 2018; Wiget et al., 2020). The High-Level Panel of Experts (HLPE, 2019) highlighted the need to develop novel interdisciplinary assessment methods that can tackle the full complexity of agroecological transitions and enable the evaluation of their performance in a holistic way. Assessing benefits and limits of agroecological transitions in different contexts is of great importance to gather evidence about stories of success and failure, and increase knowledge on how to accelerate these transitions (Dendoncker et al., 2018). Such assessment methods are needed by development actors, agroecological transition leaders, decision makers and researchers (De Schutter & Vanloqueren, 2011; GTAE, 2018; IAASTD, 2009). The evaluation purposes of such assessment methods are twofold: 1) accompany and support stakeholders in steering and managing agroecological transitions by providing them knowledge on the barriers and levers of the process and on the benefits and limits of the transition, and 2) support evidence-based advocacy and policymaking, by providing knowledge for decision makers (GTAE, 2018; Wiget et al., 2020). However, assessing agroecological transitions remains a methodological challenge. It is complicated by the development of these transitions over time, the diversity of their modalities and starting points (Wezel et al., 2020), and the broadening scope of agroecology, which embraces multiple geographical scales and multiple dimensions embedded with environmental, economic and social interactions (Wezel & David, 2012). These challenging aspects have consequences on the ways of analyzing the performances of agroecological systems and the ways to consider the barriers to and levers for transition.

Among the requirements for assessing agroecological transitions, five are largely acknowledged in the literature. The first one is to be adaptable to local conditions (Hatt et al., 2016; Martin et al., 2018; Trabelsi, 2017; Wiget et al., 2020). This is particularly important for agroecology because it is about adapting practices that are relevant to the local environment and context rather than implementing practices in a systematic way (Hatt et al., 2016; Martin et al., 2018; Trabelsi, 2017). Assessment methods therefore need to adapt to the local farm management systems (Trabelsi, 2017; Wiget et al., 2020). The second requirement is to consider social interactions among stakeholders, which many authors point to as a key element in agroecological transitions (Dendoncker et al., 2018; Magrini et al., 2019; Martin et al., 2018; Wiget et al., 2020). Agroecology represents a transformative vision that puts governance, power and democracy at the centre (de Molina, 2013). Conditions for the development of agroecological transitions include in particular political and societal pressure, broad coalitions among a diversity of stakeholders and building institutions (Runhaar, 2021). The third is to clarify the concept of agroecology. Given the multiplicity of definitions of agroecology (Wezel & Soldat, 2009), it is essential to mobilize common agroecological assessment guidelines in order to specify which approach is being used for the assessment (Wiget et al., 2020). The fourth is to consider the temporal dynamics of the transitions to take into account the gradual and often discontinuous trajectories of change (Tittonell, 2020) and to better understand barriers and levers in their development (Magrini et al., 2019; Martin et al., 2018). The fifth requirement is to use a participatory bottom-up approach which is often put forward as a necessary condition for the development of agroecology (Martin et al., 2018; Méndez et al., 2013; Wiget et al., 2020).

Although there is a multiplicity of assessment tools in the field of sustainable agriculture (Sadok et al., 2009) and a growing interest in developing agroecology-specific tools (Levard & Bertrand, 2019; Mottet et al., 2020), no systematic review has been conducted to analyze how these methods addressed the five requirements for the assessment of agroecological transitions. The objective of this study is thus to use a systematic literature review and analysis of relevant papers on these five requirements in order to investigate the suitability of existing multiscale and multidimensional assessment methods for assessing agroecological transitions.

2. Materials and methods

2.1. Identification and selection of assessment methods

To identify and select existing assessment methods, we went through a systematic literature review. We focused on identifying publications that provided multiscale and multidimensional methods assessing sustainability or resilience of current agroecosystems. The review adopted the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyzes) approach to literature search and selection (Moher et al., 2010).

We considered publications related to sustainability or resilience because they are the main stated goals of the transition to agroecological systems (Altieri & Nicholls, 2012; Barrios et al., 2020; Dendoncker et al., 2018; Gliessman, 2016; Miles et al., 2017; Mottet et al., 2020; Wezel et al., 2020). A sustainable agroecosystem was defined as one that fulfilled a balance of several goals over time: the maintenance or enhancement of the natural environment provision of human food needs, economic viability and social welfare (Hansen, 1996). Resilience was defined as the ability of a social or ecological system to absorb disturbances while retaining its organizational structure and productivity, the capacity for self-organization, and the ability to adapt to stress and transform following a perturbation (Cabell & Oelofse, 2012). The core principles on which agroecological systems build (i.e. diversity, efficient use of natural resources, nutrient recycling, natural regulation and synergies) characterize their inherent resilience potential (Altieri et al., 2015).

We defined assessment methods as frameworks and as methodologies. Frameworks are integrated and structured procedures, akin to protocols, which contain several prescribed stages that ought to be followed to meet a pre-determined objective (Gasparatos, 2010). They have a set of predefined rules and provide a list of indicators and criteria (Lairez et al., 2015) but do not specify the analytical tools that must be used for the measurement of the indicators and their analysis. Methodologies are more prescriptive than frameworks. Methodologies propose a set of organized principles, together with specific modes of inquiry and tools (Pahl-Wostl et al., 2013).

The literature search was conducted in January 2020. Keywords were entered in the Clarivate Analytics’ Web of Science without timeframe limitation. The search was conducted both in ‘English’ and ‘French’ languages, with the following search query for ‘Topics’: (agroecolog* OR agro-ecolog* OR (agro NEAR ONE ecol*) OR sustain* OR resilien*) AND (assess* OR evalua*) AND (tool* OR assessment method* OR framework*) and (farm* NEAR (system* or scal*)) AND (transition* OR develop*). This initial search yielded 900 hits (Figure 1), spanning from 1992 to 2020. Once the duplicates were removed, we proceeded with the screening phase, where we selected publications which 1) were in French or English, 2) focused on agroecosystems, defined as a cultivated ecosystems that can exist in different spatial scales from the field to the farm and beyond (Neyton et al., 2022), 3) were not specific to a unique crop, 4) studied sustainability or resilience and 5) provided an assessment framework or a methodology. The screening yielded 407 records from which we identified 350 original assessment methods (Figure 1). Note that one record could include several assessment methods, and more than one publication used a unique method.

Figure 1. Systematic review flowchart based on a PRISMA flowchart (Moher et al., 2010) illustrating the selection of records and assessment methods.

To determine eligibility among these 350 assessment methods, we selected those that met certain conditions: they were already implemented in at least one case study. They were real-world situations rather than strictly focusing on simulation or adopting a scenario-based approach. They analyzed either the barriers and levers in the development of the agricultural systems assessed or their multidimensional impacts (excluding risk assessments). And they aligned with our definition of an assessment method (frameworks and methodologies). Finally, we selected the assessment methods that covered all three dimensions of sustainability (environmental, economic and social) and addressed multiple scales (i.e. considered more than two of the following scales of the agroecosystem and the food system: field, household, farm, value chain, landscape, regional and national). This eligibility phase yielded 11 assessment methods.

We complemented these 11 assessment methods with three non-academic assessment methods that were shared by development organizations and often mentioned in the grey literature: 1) Self-diagnosis of agroecological practices in family farming (subsequently named Autodiag), developed by a Belgian non-governmental organization (NGO) in collaboration with its partners (Arango et al., 2019); 2) Assessment method for the economic-ecological analysis of agroecosystems (Lume), developed through a collaboration between the Centre for Agroecology, Water and Resilience (CAWR) of Coventry University and the Brazilian NGO, Agricultura Familiar e Agroecologia (AS-PTA) (Petersen et al., 2020); and 3) the Memento for the evaluation of agroecology, assessment method for evaluating its effects and the conditions for its development (Memento GTAE), developed through a collaboration between French NGOs (Agrisud International, AVSF, CARI, GRET) and research institutions (AgroParisTech, CIRAD, IRD) (Levard & Bertrand, 2019).

2.2. Overview of the 14 selected assessment methods

We described each of the 14 selected assessment methods according to their content, the system they assessed, their geographical application, their designers, their targeted end-users, their specificity to agroecology, their stated evaluation purpose, their scale of assessment, and the different stages of their development and use in which stakeholders were involved. Indeed, stakeholder involvement can occur at various stages in both the development and use of the assessment methods (De Olde et al., 2017; Triste et al., 2014). As suggested by De Olde et al. (2017), we distinguished three different stages of involvement: 1) in the development of the method, 2) in the use (implementation) of the method and 3) in an ex-post evaluation of the method. We also specified the type of the assessment method, framework or methodology, according to the definitions of Gasparatos (2010) and Pahl-Wostl et al. (2013).

2.3. Analysis of the 14 selected assessment methods according to the five evaluation criteria

Five evaluation criteria were used to qualify each of the 14 assessment methods: 1) be adaptable to local conditions, 2) consider social interactions among stakeholders involved in the transitions, 3) clarify the concept of agroecology, 4) consider the temporal dynamics of the transitions to better understand barriers and levers in their development and 5) use a participatory bottom-up approach. We analyzed the 14 methods to determine if, and how, they considered each of these criteria.

The adaptability of a method relates to its ability to account for local conditions (sociocultural, environmental and economic). Being adaptable to local conditions requires flexibility. Flexibility can be achieved by mobilizing locally-relevant criteria in assessment methods (Trabelsi, 2017) and by adapting measurement units and assessment methods to the specificities of the agroecological system assessed (Wiget et al., 2020). To understand if a method was adaptable to local conditions, we looked at the way the method reflected local specificities: 1) through the level of flexibility of the method, and 2) through the inclusion of a contextualization phase, reflecting the local specificities of the system being assessed.

To evaluate a method’s consideration of social interactions among stakeholders involved in the transitions it assessed, we determined if, and if so, how, the method analyzed three aspects: 1) the social network in which the system is embedded, 2) the diversity of stakeholders connected to the system and 3) the influential power of each stakeholder in the system being assessed. We further considered the different ways the methods considered elements of governance, equity and justice, which are core elements of agroecology (Anderson et al., 2021; de Molina, 2013; FAO, 2018).

The clarification of the concept of agroecology refers to different principles that can be mobilized for this purpose (Wezel et al., 2020) and cover social, ecological, economic and political issues. We analyzed whether or not the methods included such a clarification step. If so, we determined when this step took place in the method, which set of principles was mobilized, and how and with whom the clarification step was executed. Note that an assessment method that was not specific to agroecology did not include, by definition, such a clarification step.

We analyzed whether or not the methods considered the temporal dynamics through their ability to describe the trajectory of change and analyze past events (e.g. organizational, production-related) to provide information on the barriers and levers of the development of the system being assessed.

Different levels of participation exist depending on the objective of a method, and different typologies have been developed to qualify the types of participation (Reed, 2008). According to Pretty (1995), participation ranges from passive participation, where stakeholders are told what is to happen and act out predetermined roles, to interactive participation or self-mobilization, where stakeholders participate in joint analysis, shape the process, or take initiatives largely independent of external institutions. According to Pretty (1995), participation can also be ‘consultative’, in-between passive and self-mobilization, where stakeholders answer a set of pre-defined questions without being involved in decision-making, or ‘functional’, in which more space is given to stakeholders to share their views. However, functional participation does not allow stakeholders to shape the process according to their objectives. Binder et al. (2010) distinguish a gradient of participation approaches: top-down, top-down with some stakeholder participation, and bottom-up with participation throughout the assessment process.

The time of involvement throughout the different stages of the development and use of an assessment method can vary (De Olde et al., 2017; Triste et al., 2014). Reed (2008) highlights that when participation is relevant to the objective of the method, it should be considered as early as possible and throughout the process. We distinguish three different stages of involvement brought forward by De Olde et al. (2017): 1) in the development of the method, 2) in the use of the method and 3) in an ex-post evaluation of the method. We analyzed the 14 methods according to the time of involvement of stakeholders in the assessment process, the way they were involved, and the objective of their involvement. For this analysis we mobilized the typologies of Binder et al. (2010) and Pretty (1995), and we considered the stages in which stakeholders were involved (De Olde et al., 2017). This allowed us to use a binary variable to distinguish assessments: either bottom-up assessments, in which a broad engagement with various knowledge systems takes place with an interactive participation involving stakeholders in both the development and use of the method, or top-down assessments with either consultative or functional participation, involving stakeholders only in the use of the method.

3. Results

3.1. Overview of the 14 selected assessment methods

The 14 selected assessment methods varied in terms of assessment scales, geographical application, type of assessment method (framework or methodology), year of design, specificity to agroecology, category of designers, targeted end-users, evaluation purpose and stage of involvement of stakeholders (Table 1). The number and specific scales of the agroecosystem varied among the different methods. They all considered the farm level and at least one other level. Other scales considered were the field (in 9 methods), the household (in 1 method), the value chain (in 4 methods), the landscape seen as the surrounding environment of the farm (in 13 methods), the regional (in 1 method) and the national level (in 8 methods). Half of the methods considered three scales of assessment, six methods considered four scales, and one considered five scales (Autodiag). The most frequent scale combinations were the following: field, farm, landscape (in 4 methods); farm, landscape, value chain and national level (in 2 methods); and field, farm, landscape and national level (in two methods).

Table 1. Overview of the 14 selected assessment methods (11 selected from the systematic review, three shared by development organizations and often mentioned in the grey literature).

Name (abbreviated)	Agroecosystem assessed	Geographical application	Designers^a	Targeted end-users^b	Specific to agroecology	Evaluation purpose	Scale^c	Stakeholder involvement^d	Type of assessment method^e	Reference
Self-diagnosis of agroecological practices in family farming (Autodiag)	Any type	Any	N	F; N	Yes	Localized steering	P; F; VC; L; Na	Use	M	Arango et al. (2019)
Integrated valuation of ecosystem services to understand and steer agroecological transitions (Dendoncker)	Any type	Any	R	N; R; T	Yes	Design; Localized steering	P; F; L	Development; Use	F	Dendoncker et al. (2018)
Farm Sustainability Indicators (IDEA)	Any type	France and Europe	R	N. S	No	Benchmarking	P; F; L	Use; Post use phase	M	Zahm et al. (2019)
Assessment method for the economic-ecological analysis of agroecosystems (Lume)	Any type	Any	R	N; R	No	Localized steering	F; L; Na	Use	M	Petersen et al. (2020)
Assessment method for evaluating its effects and the conditions for its development (Memento GTAE)	Any type	Any	N; R	N; R; T	Yes	Advocacy; Benchmarking	P; F; L; Na	Use	M	Levard and Bertrand (2019)
Framework for Assessing the Sustainability of Natural Resource Management Systems (MESMIS)	Smallholder	South America	R	N. S	No	Localized steering	P; F; L	Development; Use	F	López-Ridaura et al. (2002)
Framework to assess the resilience of farming systems (Meuwissen)	Any type	Europe	R	R	No	Localized steering	F; VC; L	Use	F	Meuwissen et al. (2019)
Multiscale Assessment methodological Framework (MMF)	Smallholder	South America	R	R	No	Localized steering	P; F; L	Development; Use	F	López-Ridaura et al. (2005)
Qualitative expert Assessment Tool for CA adoption in Africa (QAToCA)	Any type	Any	R	R; T	No	Benchmarking	F; L; VC; Na	Use	M	Ndah et al. (2015)
Sustainability Assessment of Farming and the Environment (SAFE)	Any type	Europe	R	R	No	Localized steering	P; F; L; R	Use; Post use phase	F	Van Cauwenbergh et al. (2007)
Sustainability assessment Adaptive and Low-input Tool (SALT)	Any type	Any	R	R	No	Localized steering	P; F; Na	Development; Use	F	Calleros-Islas (2019)
Sustainable Livelihood Framework (SLF)	Any type	Any	T	U	No	Localized steering	H; F; L; Na	Development; Use	F	DFID (1999)
Tool for Agroecology Performance Evaluation (TAPE)	Any type	Any	F; N; R; T; UN	U	Yes	Advocacy; Benchmarking	P; F; L, Na	Development; Use; Post use phase	M	FAO (2018); Mottet et al. (2020)
Participatory design assessment methodology of ‘territorial biodiversity-based agriculture’ (Tata Box)	Any type	France	R	R	Yes	Design; Localized steering	F; L; VC; Na	Development; Use; Post use phase	F	Duru et al. (2015)

^a Designers: F: Farmers; N: NGO; R: Research; T: Technical institutions; UN: Multiple UN collaborating nations and institutes; N.S: not specified.

^b Targeted end-users: F: Farmers; N: NGO; R: Research; T: Technical institutions; U: Universal; N.S: not specified.

^c Scale: P: field; H: household; F: farm; VC: value chain; L: landscape; R; regional; Na: national

^d Stakeholder involvement: As suggested by De Olde et al. (2017), we distinguish three different stages of involvement: (1) in the development of the method, (2) in the use (implementation) of the method and (3) in a post use phase (reflections on the method).

^e Type of assessment method: F: Framework according to the definition of Gasparatos (2010); M: Methodology according to the definition of Pahl-Wostl et al. (2013).

Eight out of 14 assessment methods were developed for application in any geographical context. Four were especially developed for the northern hemisphere, and two for the southern hemisphere. Eight out of the 14 assessment methods were designated as frameworks, according to the definition of Gasparatos (2010). Six were designated as methodologies, as defined by Pahl-Wostl et al. (2013). Half of the assessment methods (7 out of 14) were designed recently, five of them in 2019 and two in 2020. The other assessment methods were designed between 1999 and 2018. Five assessment methods focused specifically on agroecology, and two (Dendoncker, Tata Box) focused specifically on agroecological transitions. These five assessment methods were all designed between 2017 and 2020 which illustrates the recent growing interest in developing agroecology-specific methods.

Most of the assessment methods were designed by researchers (10). One assessment method was designed by a non-governmental organization (Autodiag) and one by a technical institution (SLF). Only two methods were designed through a multistakeholder collaboration. Those occurred between researchers and non-governmental organizations (Memento GTAE) and between researchers, non-governmental organizations, technical institutions, farmers (through producers’ organizations) and the United Nations Organization for Food and Agriculture (TAPE). Those two methods were also among the ones designed most recently (in 2019 and 2020). TAPE was refined through feedbacks received by end-users between 2019 and 2022. Five of the assessment methods specified targeting several end-user groups. These end-users’ groups were technical institutions, non-governmental organizations, and research institutions. Other methods did not specify the targeted end-users or mentioned being universal. Only one assessment method (Autodiag) specifically referred to farmers as targeted end-users.

The evaluation purposes of the assessment methods were manifold. The most common stated purposes were localized steering assessment (10 methods), benchmarking, or comparing different systems (4 methods), design of agroecological transitions (2 methods), and advocacy (2 methods). Four assessment methods were designed and used for multiple purposes. Two of these were for design of agroecological transitions and localized steering assessment (Dendoncker, Tata Box) and two were for advocacy and benchmarking (Memento GTAE, TAPE). These evaluation purposes echo the ones we have aimed for in this paper. Localized steering assessment and agroecological design provide guidance and support for stakeholders in the steering and management of agroecological transitions, while benchmarking and advocacy provide support to evidence-based advocacy and policymaking, by providing knowledge for decision makers.

Half of the assessment methods (7) involved stakeholders only in their use. One method (SALT) involved stakeholders for the use of the method and for one stage of its development (i.e. the selection of indicators). In the other assessment methods (7), stakeholders were involved in both the development and use of the method. These seven assessment methods presented a strong level of stakeholders’ involvement. Most of them corresponded to frameworks (6 out of 7). Given their higher degree of flexibility, frameworks allowed for more extensive involvement of stakeholders throughout their development and their use. One method (TAPE) corresponded to a methodology. TAPE involved a broad range of stakeholders in its development stage, in particular through a participatory and inclusive multi-stakeholder consultation phase with more than 450 participants over 4 months and an international in person workshop with 70 participants (Mottet et al., 2020). Finally, only four methods (IDEA, SAFE, TAPE and Tata Box) involved stakeholders in an ex-post evaluation (i.e. following the method’s use). This ex-post evaluation aimed at improving the methods by building on feedback from stakeholders regarding their use of the method and the results of the assessments. Interestingly, the designers of these three methods were researchers, except for TAPE for which the United Nations Organization for Food and Agriculture had a coordinating role in the multistakeholder collaboration. Feedback was gathered differently in these methods. In IDEA, SAFE and TAPE, individual users’ feedback was collected. For SAFE, the views expressed by farmers after the presentation of the assessment results were also considered. Tata Box proposed a more in-depth reflective work involving all stakeholders that took part in the development and use of the method, through interviews and questionnaire.

3.2. Analysis of the 14 selected assessment methods according to the five evaluation criteria

The methods adopted different approaches to consider each evaluation criterion, but none of them covered all the five criteria (Table 2). The number of criteria covered by the methods ranged from one to three.

3.2.1. Be adaptable to local conditions

The analysis of the 14 assessment methods showed that being adaptable to local conditions was achieved by two approaches (Table 3): 1) by providing generic guidelines with flexible content, and 2) by including a contextualization phase (Table 4). Two methods used the first approach, five used the second approach, and six methods used both approaches. Both approaches share two advantages: 1) stakeholders define the evaluation objectives; 2) indicators are aligned with local conditions.

Table 2. Coverage of the five evaluation criteria by the 14 multidimensional and multi-scale assessment methods selected in the review (✓: coverage of the criteria by the assessment method).

5 evaluation criteria	Autodiag	Dendoncker	IDEA	Lume	Memento GTAE	MESMIS	Meuwissen	MMF	QAToCA	SAFE	SALT	SLF	TAPE	Tata Box	Number of methods
1) Be adaptable to local conditions	✓	✓		✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	13
2) Consider social interactions	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	14
3) Clarify the agroecological concept	✓												✓		2
4) Consider the temporal dynamics to better understand barriers and levers				✓	✓										2
5) Use a participatory bottom-up approach		✓				✓		✓			✓	✓	✓	✓	7
Number of evaluation criteria covered by the assessment methods	3	3	1	3	3	3	2	3	2	2	3	3	4	3

Table 3. Approaches used by the 14 assessment methods to consider the five evaluation criteria (✓: approach considered in the assessment method).

5 evaluation criteria		Approaches	Autodiag	Dendoncker	IDEA	Lume	Memento GTAE	MESMIS	Meuwissen	MMF	QAToCA	SAFE	SALT	SLF	TAPE	Tata Box
1) Be adaptable to local conditions	Flexible content with a non-fixed set of indicators	Series of steps with specific questions, objectives, outcomes		✓				✓	✓	✓			✓			✓
		List of system components												✓
		Hierarchical structure of principles, criteria, indicators										✓
	Contextualization phase	Flowchart				✓		✓
		Series of guiding questions	✓	✓						✓	✓					✓
		Agrarian diagnosis					✓
		Description vulnerability context												✓
		Template							✓						✓
2) Consider social interactions	Social-related indicators		✓	✓	✓	✓	✓	✓		✓	✓	✓	✓	✓	✓
	Stakeholder mapping								✓							✓
3) Clarify agroecological concept	Principles		✓												✓
4) Consider the temporal dynamics	Trajectory of change					✓	✓
5) Use a participatory bottom-up approach	Bottom-up, interactive			✓				✓		✓			✓	✓	✓	✓

Table 4. Specific advantages and limits of the two approaches that the methods mobilized to be adaptable to local conditions.

Approaches		Advantages	Limits
Providing generic guidelines with flexible content		No requirement for specific methods or tools No fixed indicator set Indicators can be chosen in a participatory selection procedure	Operationalization can be challenging if methodological support is needed Comparability of the results
	Series of steps with specific questions, objectives and outcomes	Guide the assessment process
	List of system components	Facilitates operationality
	Hierarchical structure of principles, criteria, indicator	Facilitates operationality
		A clear operationalization procedure through specific tools Characterization of the system being assessed
Including a contextualization phase	Flowchart	Stakeholder involvement; interactions analysis Identification of locally relevant indicators Not time-consuming	No benchmarking
	Guiding questions or shared diagnosis	Stakeholder involvement Identification of locally relevant indicators	No interactions analysis Time-consuming No benchmarking
	Agrarian diagnosis	Identification of locally relevant indicators Benchmarking	No stakeholder involvement No interactions analysis Time and resource consuming
	Vulnerability analysis	Stakeholder involvement Identification of locally relevant indicators Comprehensive characterization	No interactions analysis Time and resource consuming No benchmarking
	Template	Rapid assessment	No stakeholder involvement No interactions analysis Insufficient detail to identify relevant indicators

Generic guidelines with flexible content provided an operational strategy for the evaluation without prescribing specific methods or tools for its application. This was a particular characteristic of frameworks. Specific methods or tools were left to the user’s choice, which favours the users’ creativity and freedom to mobilize methods and tools they deemed relevant, but it raised the issue of operationalization for users seeking for methodological support and comparability of results between systems assessed. In these frameworks, generic guidelines with flexible content suggested a series of steps with specific questions, objectives and outcomes that guided the assessment process (Dendoncker, MESMIS, Meuwissen, MMF, Tata Box). To facilitate the operationality, these guidelines proposed lists of system components to be considered in the evaluation (SLF), or a hierarchical structure of principles, criteria and indicators (SAFE). Generic guidelines with flexible content did not fix a set of indicators. In four methods, indicators were chosen in a participatory selection procedure, through participatory workshops (MESMIS, MMF, SALT, SLF). In one method (SAFE) indicators were selected throughout an expert-led procedure. Two methods (Dendoncker, Meuwissen) did not specify the way indicators were selected, and one method did not mobilize any indicators (Tata Box).

A contextualization phase allowed to reveal the system’s specificities and provided an analysis of the context in which the system is embedded. It allowed the stakeholders to define their evaluation objectives and to align the indicators with local conditions (i.e. environmental and socio-economic indicators specifically relevant to the system assessed). In the selected assessment methods, five different approaches were used to contextualize the system (Table 4). The first approach mobilized stakeholders to build a flowchart describing the functioning and the key features of the system being assessed (Lume and MESMIS). It identified the limits of the system assessed and highlighted the interactions of the system with internal and external components, such as the community, the markets, or the state. This first approach did not provide information on other systems for benchmarking but made it possible to determine local relevant indicators for the assessment. It also required little time and resources. The second approach mobilized stakeholders to answer a series of guiding questions to define the scope and scales of the assessment (MMF), or to achieve a shared system diagnosis by confronting different points of view (Autodiag, Dendoncker, QAToCA, Tata Box). Like in the first approach, this approach determined the limits of the system being assessed, provided a detailed analysis of the system's context, and made it possible to determine locally relevant indicators for the assessment. Like in the first approach, it did not allow benchmarking. In contrast to the first approach, it identified existing challenges and involved a broader range of stakeholders with different viewpoints but did not highlight the interactions of the system with internal and external components. The time required for the organization of multistakeholder workshops was pointed out as a limit in this approach. The third approach (Memento GTAE) mobilized an expert-led agrarian diagnosis, which involved determining a typology of farms and a description of the socio-economic and environmental context. It also made it possible to determine locally relevant indicators for the assessment. Contrary to the first two approaches, Memento GTAE allowed benchmarking, although it was limited to the surrounding farms embedded in the same context. Like the second approach, the agrarian diagnosis required resources and time. The fourth approach (SLF) proposed a description of the vulnerability context (i.e. the external factors that influenced access to assets). This approach mobilized stakeholders to analyze the types of livelihood assets, considering five capitals namely human, social, natural, physical and financial considered to be key for sustainability. This fourth approach provided a comprehensive characterization of the system being assessed and made it possible to determine local relevant indicators for the assessment but did not allow any benchmarking. It also required time, financial and human resources, mentioned by the method as a counterpart of its holistic potential. Finally, the fifth approach consisted in providing a brief description of the system being assessed, mobilizing a template with a list of criteria to consider. This approach made it possible to specify the context in which the system was embedded and the existing challenges (Meuwissen, TAPE). This approach was more succinct in time and in the information gathered than the first four. It did not provide sufficient details on the system’s specificities to identify relevant local indicators.

Among the 14 assessment methods analyzed, one only (IDEA) did not provide either a flexible content with generic guidelines or include a contextualization phase. This related to the method’s evaluation purpose, which was to ensure benchmarking across various systems by using a fixed set of indicators with pre-defined calculation and scoring. This benchmarking implemented in IDEA remained limited to French (including overseas’ islands) and European contexts.

On the contrary, six methods mobilized both generic guidelines with flexible content and a contextualization phase to adapt the assessment to local conditions (Table 3). This exposed the possible advantages of the complementary use of both approaches to ensure a high level of adaptation to local conditions: 1) providing enough flexibility to allow stakeholders to shape the assessment according to their views, leading to a more meaningful assessment for them, and 2) including a contextualization phase to understand and share a meaningful view of the contours, the functioning, and the specificities of the system being assessed with all stakeholders involved. All six methods corresponded to frameworks, and their common evaluation purpose was to provide a localized steering assessment.

3.2.2. Consider social interactions among stakeholders involved in the transitions

The 14 assessment methods used two different approaches to consider social interactions (Table 3). Social-related indicators were used to analyze the social network in which the system was embedded. A mapping of stakeholder social interactions was used to analyze the diversity of stakeholders connected to the system and the influential power of each stakeholder in the system being assessed.

Mobilizing social-related indicators was adopted by all assessment methods except Meuwissen and Tata Box. But the issues addressed by this approach varied among the assessment methods; we identified four different social-related objectives: 1) determining the social links between consumers and producers or between other producers (Autodiag, IDEA, Lume, MESMIS, QAToCA, SAFE, SLF, TAPE); 2) qualifying the degree of involvement in social networks in terms of the existence and intensity of local networks and participation in the social networks or events (Lume, SLF, TAPE); 3) estimating the territorial anchorage of the system, such as the opening of the farm to the public (IDEA, Dendoncker); and 4) qualifying the gender balance, such as women and youth empowerment (TAPE, Memento GTAE). Although mobilizing social-related indicators allowed an analysis of the system’s embeddedness in a social network, it did not include an analysis of the diversity of stakeholders interconnected with the assessed system, nor the influential power of each stakeholder in that system.

The second approach of mapping stakeholder social interactions analyzed how the assessed system was interconnected to social networks by 1) considering the influential power of farmers with external stakeholders (Meuwissen) or by 2) positioning various stakeholders with respect to their power of influence on agroecological transition and their opinion (supporters or opponents) (Tata Box). Combined, the two approaches (mobilizing social-related indicators and mapping stakeholder social interactions) could lead to a comprehensive image of the social context of the system assessed and help to expose the existing stakeholder networks as well as the multilateral and power-driven interplay between stakeholders.

Governance, equity and justice were considered in the 14 methods in two different ways. The first way was the scope of the method, i.e. the issues addressed by the methods (all methods). The second was the implementation process proposed by the method (Tata Box). Half of the methods addressed issues related to institutions, public policies (Autodiag, Lume, Meuwissen, QAToCA, SAFE, SALT, SLF) and equity, in particular regarding gender divisions and power-relations (Dendoncker, Lume, Memento GTAE, MESMIS, MMF, SAFE, SALT, SLF, Tata Box). Other issues addressed were related to the organisation of producers (Lume, QAToCA, SALT, SAFE, SLF), producers’ participation in the governance of natural resources (Memento GTAE, QAToCA, TAPE, SLF) and the empowerment of producers, in particular their rights and ability to improve their knowledge (IDEA, QAToCA, SAFE, Memento GTAE, SLF). Only one method addressed the issue of human rights (SLF). In Tata Box, equity and transparency were brought forward as essential elements to guarantee the good implementation of the method. The method evoked the fact that the process needed to ‘establish a relationship of equivalence with all participants’ and that ‘the fairness of the process is embodied in the concepts of transparency, empowerment of participants and neutrality’ (Audouin et al., 2018, p. 17).

3.2.3. Clarify the concept of agroecology

Only two assessment methods (TAPE, Autodiag) clarified the concept of agroecology (Table 3). They did this at distinct stages, using different sets of principles and mobilizing different approaches. The clarification step was the first step (out of 4) of TAPE. It was called Characterization of Agroecological Transitions (CAET) (Lucantoni et al., 2023; Mottet et al., 2020), and it enabled an understanding of how advanced the system was in the agroecological transition, before its multidimensional performance was assessed. Clarifying the concept of agroecology was the key focus of Autodiag. Autodiag aimed at ‘making a diagnosis of the situation within a given community with regard to the various principles that define agroecology, to debate them in order to potentially lay the foundations for a thoughtful agroecological transition’ (Arango et al., 2019). TAPE used the 10 Elements of Agroecology (Barrios et al., 2020; FAO, 2018). These Elements built on the five principles of agroecology defined by Altieri (1995) and the five levels of agroecological transitions defined by Gliessman (2016). They stemmed from a multistakeholder international negotiation process and were endorsed by member organizations of the United Nations Organization for Food and Agriculture (FAO), which made them recognized internationally (Barrios et al., 2020; FAO, 2018). Both methods mobilized a qualitative assessment built on a scoring system. In TAPE, the 10 Elements were disaggregated into 36 indices with five levels of transition that took the form of descriptive scales (Lucantoni et al., 2023) that followed a scoring system ranging from zero to four. Autodiag used a set of 13 original principles which focused on environmental, socioeconomic, political and organizational issues. Each of these principles was correlated to practices carried out by farmers. For each practice there was a set of descriptive scales related to a scoring system ranging from one to four. Both the approach and the time needed for the qualitative assessment differed between the two methods. Clarifying the concept of agroecology was the core aim of Autodiag, which included no other additional steps. Autodiag was built on a participatory process that focused on dialogue and interaction between members of farmers’ organizations or representatives of a population and took place on a 2.5 day workshop (Arango et al., 2019). The clarification step in TAPE was estimated to last one hour and could either be conducted as a self-assessment by producers or community leaders or as a guided exercise led by technicians, CSO workers, extensionists, scientists or government agents (Mottet et al., 2020).

The other three assessment methods, although designed to be specific to agroecology, did not attempt to clarify the agroecological concept. One was built on the assumption that the system assessed was agroecological (Memento GTAE). Two methods referred to literature to provide a definition of the concept of agroecology but did not mobilize any principles (Dendoncker, Tata Box).

3.2.4. Consider the temporal dynamics of the transitions to better understand barriers and levers in their development

Two methods (Lume, Memento GTAE) mobilized a trajectory of change in order to analyze key events or factors that explained barriers and levers in the development of the system assessed (Table 3). The analysis of the trajectory of change provided a coherent understanding of the evolution of the system and its development conditions by chronologically recording the main internal and external events. Memento GTAE mobilized an agrarian diagnosis that also included a regionally and nationally scaled historical analysis of the environmental and socio-economical context of the system assessed and a typology of other neighboring systems in the region (Jouve & Tallec, 1994). The agrarian diagnosis therefore permitted a local benchmarking and helped to clarify whether the transition of the system was part of a broader transition or if it was marginal. This additional information allowed a better understanding of the enabling environment that supported or hindered the agroecological transition of the system being assessed (i.e. the barriers and levers).

3.2.5. Use a participatory bottom-up approach

A participatory bottom-up approach was used in seven of the methods (Table 3). Six of these methods provided general guidance without any pre-determined procedure, and corresponded to frameworks. One method corresponded to a methodology (TAPE). Stakeholders were involved both in the development and use of the methods (Table 1). We qualified participation of stakeholders as interactive, because it is aimed at shaping the overall assessment according to their views and objectives and ultimately developing a joint analysis. Two methods also involved stakeholders to build an action plan (Dendoncker, Tata Box). The flexibility of the frameworks and of the methodology allowed the assessments to evolve according to stakeholder objectives and views. In TAPE, end-users were able to select or add indicators they considered essential for the assessment. All seven methods were characterized by the involvement of diverse categories of stakeholders, such as farmers, technicians, community representatives and researchers. Stakeholder involvement was guaranteed through participatory workshops (Dendoncker, MESMIS, MMF, SALT, SLF, TAPE, Tata Box) and could also take the form of an evaluation team (Dendoncker, MESMIS). Tata Box was the only assessment method clarifying the in-depth selection procedure of the stakeholders. Those stakeholders were involved throughout the development and use of the method and characterized through a stakeholder mapping. However, this type of participatory approach also presented the disadvantage of being time and resource consuming (Binder et al., 2010). SLF highlighted the ‘need of time, financial and human resources’, considered by the authors as a counterpoise to its holistic potential (DFID, 1999). MESMIS required the establishment of an evaluation team of 9–20 individuals and over two years to be implemented (López-Ridaura et al., 2002). Dendoncker and Tata Box both relied on collaborative learning processes and were built on series of workshops that required time. TAPE mobilized a four-year long participatory process (2018-2022) that included the development of the method and the feedback from end-users following its use (FAO, 2018). They also involved many stakeholders (i.e. anyone affected directly or indirectly by the agroecological transition). Only MMF and SALT did not specify the time needed for their development and use. Finally, most of these methods resulted in localized assessments, which raised the issue of genericity and comparability of the results. TAPE however resulted in benchmarking and advocacy. To achieve this, TAPE as a methodology was more prescriptive than the frameworks and focused on large samples of farms, allowing for extrapolation of results.

The other seven assessment methods adopted top-down approaches. Stakeholders in these methods were only mobilized during the methods’ use and to answer a pre-established set of questions as an individual (IDEA, Memento GTAE) or through dedicated working group sessions (Autodiag, Lume, Meuwissen, QAToCA, SAFE). In these methods, stakeholders did not have the possibility to contribute to the process with their own knowledge, nor modify or adapt the method according to their particular context or objectives, nor to select their own set of indicators.

4. Discussion

Our review resulted in the selection of 14 multidimensional and multi-scale assessment methods. The analysis showed that although they used different approaches, each of the assessment methods covered one or more of the five criteria, but none of them covered all five criteria. This suggests that no single method seems to be appropriate for assessing agroecological transitions. The reason may be that developing a single method that covers all five criteria may present numerous constraints and be too cumbersome to implement by end-users and therefore ineffective. The more comprehensive a method is, the more difficult it is to use (Marchand et al., 2014) and a critical success factor is the ease of use of a method by its end-users (Van Meensel et al., 2012). Difficulties relate to the technical and financial resources, the time and the expertise it requires (Van Meensel et al., 2012). An evaluation is a matter of compromise between a desire for comprehensiveness and a need for operationality, and it should be adapted to its purposes and end-users. Involving end-users in the methodological choices made during the development phase of the method would be a way to ensure its operationality (Cerf et al., 2012).

In this discussion, we consider the two main evaluation purposes announced in the introduction: 1) accompany and support stakeholders in steering and managing agroecological transitions, and 2) support evidence-based advocacy and policymaking. Binder et al. (2010) and Reed et al. (2006) highlight the necessary trade-offs to be considered in assessment methods. These trade-offs relate to the evaluation purpose (localized steering assessments versus global assessments and benchmarking), the time requirement (time-consuming versus ‘fast’) and the level of participation (bottom-up versus top-down). They need to be reasoned for each of the five evaluation criteria that are key to assess agroecological transitions. Analyzing the specific feasibility of each method, by identifying and quantifying all the constraints is beyond the scope of this study. However, this analysis has to be done in the future to improve the efficiency of assessment methods and then enhance quantity and quality of data acquired by assessments.

Adapting to local conditions allows a method to be tailored to the needs of end-users (De Olde et al., 2018) and to produce context-specific knowledge (Martin et al., 2018). This makes assessments more legitimate and useful (De Olde et al., 2018; Van Meensel et al., 2012). Producing context-specific knowledge that meets end-users’ needs is key for steering and managing agroecological transitions. Our findings show that the more a method is adaptable to local conditions, the more the method is specific to the context of the assessment and the less it is possible to compare results with other contexts. Indeed, the six most adaptable methods (Dendoncker, MESMIS, Meuwissen, MMF, SLF, Tata Box), which are all frameworks, mobilize both generic guidelines with flexible content and a contextualization phase with the aim of providing a localized steering assessment. The least adaptable method (IDEA) provides a standardized procedure for the assessment that allows comparison across different contexts and aims at benchmarking. These results suggest a trade-off that depends on the time and resource requirement and the evaluation purpose. Adapting to local conditions requires time and resources with the involvement of end-users to shape the assessment according to their views and needs. It also requires some animation expertise to consider different views and objectives and agree on the way to adapt the method. The trade-off regarding the evaluation purpose raises the issue of the feasibility of aiming at both supporting stakeholders in steering and managing agroecological transitions and supporting evidence-based advocacy and policymaking. Wiget et al. (2020) propose a way to address this trade-off. In order to achieve both evaluation purposes, they suggest harmonizing locally adapted assessments (i.e. exploiting commonalities) to allow for a certain level of comparability. Similarly, we suggest considering the approaches mobilized by the methods most adaptable to local conditions: include a contextualization phase and have a flexible content with a non-fixed set of indicators. However, this flexible content must be accompanied by a real effort to identify commonalities (e.g. similarities of indicators changed) between the different locally adapted assessments. This would allow aiming for both evaluation purposes (i.e. supporting stakeholders in steering and managing agroecological transitions and supporting evidence-based advocacy and policymaking).

The assessment of social interactions among stakeholders involved in the transitions is an important gap in the knowledge about agroecological transitions (D’Annolfo et al., 2017). Agroecology is not only a science and a set of practices, it is also a social movement (Wezel et al., 2009). Considering social interactions is therefore of key importance when analyzing agroecological transitions. It makes it possible to understand the power relations among stakeholders as well as their role and position in relation to the agroecological transition (Magrini et al., 2019). This information is of strategic use in the case of steering and management of agroecological transitions because it enables stakeholders who are driving the transition to identify other stakeholders with whom they can build relationships that enhance the transition. Considering social interactions makes it possible to understand the key role of some categories of stakeholders in enhancing agroecological transitions and provide information that can support recommendations for advocacy and policymaking. Our results show different strategies to consider social interactions depending on the methods. Mobilizing social-related indicators is the most common approach and allows an assessment of the intensity of social interactions. Selecting relevant social-related indicators to be included in the assessment might be the easiest and fastest way to consider social interactions. Yet, social-related indicators do not allow to pinpoint or categorize each of the stakeholders influencing the agroecological transition. Social-related indicators synthesize information regarding the level of social interactions but do not provide a detailed mapping of the diversity of stakeholders involved, nor do they provide an analysis of stakeholders’ role and position in relation to the agroecological transition. Stakeholder mapping is a way to achieve this. None of the methods selected include both approaches because none of them were aimed at considering social interactions in such a comprehensive manner. Although it may represent various constraints, regarding time, resources and the level of expertise needed, we suggest combining both approaches to achieve a comprehensive analysis of social interactions among stakeholders involved in the transitions.

Clarifying the concept of agroecology in the assessment allows it to propose a precise image of agroecology and avoid confusion and diverse interpretations of the term (Barrios et al., 2020; Wezel et al., 2009). It allows a better understanding of the object of the assessment (i.e. how and why it is agroecological). Clarifying the concept of agroecology provides a diagnosis on the agroecological strengths and weaknesses of the assessed system (Mottet et al., 2020), which is important for steering and managing agroecological transitions. By mobilizing a set of principles, it is possible to compare the wide variety of agroecological transitions that exist and to draw a parallel between their advancements in the transition process and their multidimensional impacts. The comparison between different agroecological transitions is key to the support of evidence-based advocacy and policymaking. Among the methods analyzed, only two integrate this criterion (Autodiag, TAPE), while mobilizing two different approaches. The faster and standardized approach mobilized in TAPE is interesting for benchmarking and provides information to support evidence-based advocacy and policymaking. Moreover, it used the 10 Elements of Agroecology (Barrios et al., 2020; FAO, 2018), which is a set of principles that has an international recognition, ensuring comparability between different agroecological transitions (Wiget et al., 2020). The approach mobilized in Autodiag is more suitable to accompany and support stakeholders in steering and managing agroecological transitions. It is a more participatory approach that requires more time but allows an exchange of views on the concept of agroecology. This approach increases the collective awareness about the position of agroecology the system being assessed. The faster and standardized approach used in TAPE revealed the need to contextualize the meaning of the principles through more exchanges with local stakeholders (Lucantoni et al., 2023; Mottet et al., 2020). In Autodiag, the approach relying more on participation can provide relevant elements that answer this need. The two approaches can thus be complementary. To clarify the concept of agroecology, we suggest mobilizing the 10 Elements of Agroecology (Barrios et al., 2020; FAO, 2018) and combining both approaches through a guided exercise organized in the form of a focus group to allow an exchange of views on the concept of agroecology. This combined approach would allow a level of comparability (mobilizing a set of internationally recognized principles) in a limited time and include the involvement of stakeholders in the clarification of the concept of agroecology.

Considering the temporal dynamics of agroecological transitions helps us to understand changes over time and highlight the conditions for change (the barriers and levers). These are keys to the steering and management of agroecological transitions and the support of evidence-based advocacy and policymaking. Most of the assessment methods analyzed are static. To assess a changing system, one strategy is to use a diachronic analysis to see how selected key indicators change over time. This is an approach suggested in MESMIS and TAPE, by mobilizing historical data. This strategy presents two disadvantages: it is costly because the assessment needs to be done several times, and it does not necessarily provide information on the reasons for change or the barriers and levers in the transition. Only two of the methods proposed to develop this dynamic approach (Lume, Memento GTAE). Both methods mobilize a trajectory of change, although with different approaches. The approach mobilized in Memento GTAE allows for local benchmarking (contrary to Lume) but requires more resources and time than the approach mobilized in Lume. We suggest considering temporal dynamics of agroecological transitions to better understand barriers and levers in their development by mobilizing a trajectory of change as in Lume.

In terms of participation, a bottom-up approach has advantages. It strengthens the legitimacy and ownership of the assessment and its results (De Olde et al., 2018), and it adds a perception of usefulness through the alignment of end-users’ objectives with the objectives of the method (Van Meensel et al., 2012). Using a participatory bottom-up approach supposes identifying end-users from the start and involving them both in the development and use of the method. It also allows the involvement of stakeholders with different views, obligations, skills, types of knowledge and resources (Binder et al., 2010; Méndez et al., 2013), which is the essence of agroecology (Méndez et al., 2013). A participatory bottom-up approach presents the disadvantage of being time and resource consuming (Binder et al., 2010). Top-down assessment methods have the advantage of a relatively fast procedure that is highly standardized and reproducible with pre-selected indicators, system definition and scale of analysis (Binder et al., 2010). These aspects facilitate benchmarking and comparison among different systems (Binder et al., 2010; Lairez et al., 2015; Reed et al., 2006). However, top-down assessment methods often fail to engage local communities (Reed et al., 2006), which may impede the recognition of the results emerging from the evaluation (Binder et al., 2010). However, even in bottom-up approaches, none of the analyzed assessment methods specify that the stakeholders involved in the assessments are the actual end-users of the method and its results. Results also show that half of the assessment methods involve stakeholders only in the use of the method and in the valorization of its results. This echoes the views of Binder et al. (2010) and De Olde et al. (2016) suggesting that existing assessment frameworks are developed by experts without actively involving stakeholders and considering their needs.

To ensure a satisfactory level of participation but also limit the time needed and achieve a certain level of comparability, a combination of the advantages of both bottom-up and top-down approaches could be considered. Reed et al. (2006) and Binder et al. (2010) propose adopting a hybrid approach. In such an approach, the objectives are first defined with the end-users of the method and its results and then critically examined by researchers and complemented by theoretical considerations (Bossel, 1999; Robèrt et al., 2002; Wiek & Binder, 2005). This approach allows for back and forth interaction between a co-design with end-users and a formalization period through research (David, 2012; Hatchuel, 2000). The benefits of adopting such an approach have been widely documented (Calvo & De Rosa, 2017; Mackenzie & Davies, 2019; Moser, 2016; Steen et al., 2011) and include, in particular, the generation of original responses to challenges, the improvement in understanding end-user's needs, and a greater efficiency in decision-making. Experiments of this hybrid approach in practice relate its potential to create a knowledge partnership, bringing together end-users and designers in the co-construction of a common objective (Garcia Parrilla et al., 2016). This partnership brings together different representations, valuations and expectations through an iterative and dialogical process (Cerf et al., 2012; Wiek & Binder, 2005). This iteration creates a learning environment between designers and users (Cerf et al., 2012) that enables individual and collective learning and broadens the type of factors that can be taken into account (Garcia Parrilla et al., 2016). However, authors also highlighted the difficulty of assuring a common vision of sustainability within this diversity (Garcia Parrilla et al., 2016; Mackenzie & Davies, 2019; Wiek & Binder, 2005). A hybrid approach also presents the advantage of integrating an operational issue, thus focusing on the use situation (Cerf et al., 2012; Mackenzie & Davies, 2019).

Two of the five evaluation criteria are interlinked, i.e. being adaptable to local conditions and use a bottom-up participatory approach. Our findings indicate that the level of flexibility of methods align with their level of participation. The most adaptable assessment methods are also the most participatory. They are all frameworks with a loose and flexible structure that allows the assessment to evolve according to stakeholder’s objectives and views. It is also evident that the more participatory and flexible a method is, the more time and resources are needed, and the less benchmarking is possible.

The governance, e.g. the access, sovereignty, storage, of the data produced by the methods is a tricky and important question very little addressed in the methods. Data governance should be discussed at the beginning of the assessment process to increase involvement of all stakeholders, including farmers, to acquire, store and disseminate results. Placing farmers at the centre of data-gathering processes for example, grants farmers an agency over both the inputs and outputs of their data and advances their meaningful participation in data activities (Van Geuns et al., 2023). Farmer participation is essential to build a knowledge base by and for farmers (Richardson et al., 2022) and then to achieve effective transformative changes. All the methods analyzed mentioned that results were to be shared with farmers, but they did not specify how farmers, or other stakeholders could access the data produced. Access to data is essential for farmers to make use of it and steer their activities. Only one method (TAPE) referred to a storage system, a ‘global database’, for future use of the data (FAO, 2018; Mottet et al., 2020). The issues of sovereignty and access to data in a usable form to support the agroecological transition steering in the medium and long term need to be addressed in the future development of assessment methods.

Complementarity between decision-support methods originally designed to be used alone may exist (Cerf & Meynard, 2006). A smart combination of existing methods can make sustainability assessment more profound and can broaden the insights of different end-user groups (Van Passel & Meul, 2012). Considering the existing trade-offs between the evaluation purpose, the time requirement and the level of participation, we have highlighted above some suggestions on the approaches that could be built upon to consider each of the five evaluation criteria. Combined in a complementary mode these approaches could cover all five evaluation criteria.

5. Conclusion

Assessing the benefits and limits of agroecological transitions in different contexts is of foremost importance to gather evidence about stories of success and failure and increase knowledge on how to accelerate these transitions. This knowledge is key to the steering and management of agroecological transitions but also to the strengthening of evidence-based advocacy. However, assessing agroecological transitions remains a methodological challenge. Despite high expectations regarding their potential and their performances, there is currently a lack of combined analysis regarding the barriers and levers in the development of these transitions and their multidimensional impacts. In this paper, we analyzed the extent to which existing assessment methods addressed five of the key requirements for assessing agroecological transitions brought forward by the literature. Our systematic review led to the selection of 14 assessment methods, none of which considered all five of those evaluation criteria. These assessment methods adopted different approaches to consider one or more of the five evaluation criteria based on their evaluation purpose, their structure, and their participatory approach. Considering the existing trade-offs between the evaluation purpose, the time requirement and the level of participation in the different approaches adopted by the 14 methods studied, we suggest combining some of the approaches in a complementary mode to cover all five criteria and therefore improve the assessment of agroecological transitions.

Acknowledgements

We thank Ben Boswell for English reviewing.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

We would like to acknowledge the support of the AVACLIM project (‘Agroecology, Ensuring Food Security and Sustainable Livelihoods while Mitigating Climate Change and Restoring Land in Dryland Regions’), funded by the Green Environmental Fund (Grant ID CZZ2009) and the French Global Environmental Fund (Grant ID ID-9993).