One alternative health device! A methodological proposal to analyze research projects’ orientation towards national health problems

ABSTRACT

In this paper, we present a device to generate assessment tools for research projects oriented to collaborate with national health problems. We made use of the Stokes’ model (1997) for knowledge production as an analytical framework that shows the interaction between two dimensions: the consideration of use and the search of fundamental knowledge. We made an explicit effort to incorporate the social participation in science as a complementary dimension to scientific knowledge production, to generate a device based on the Stokes’ model. When operationalizing it, we presented a set of orienting scales that are useful when dealing with the complex task of generating research projects’ assessment tools. We conclude that methodological proposals for research evaluation are much needed and that devices like this should be taken as a part of human decision-making processes, not as a substitute for them.

RESUMO

Neste artigo pretendemos apresentar um dispositivo para gerar ferramentas de avaliação para projetos de pesquisa orientados a colaborar com os problemas de saúde nacionais. Utilizamos o modelo de produção de conhecimento de Stokes (1997) como um quadro analítico que mostra a interação entre duas dimensões: a consideração do uso e a busca do conhecimento fundamental. Fizemos um esforço explícito para incorporar a participação social na ciência como uma dimensão complementar à produção do conhecimento científico para gerar um dispositivo baseado no modelo de Stokes. Ao operacionalizá-lo, apresentamos um conjunto de escalas orientadoras que são úteis para lidar com a complexa tarefa de gerar ferramentas de avaliação de projetos de pesquisa. Concluímos que propostas metodológicas para avaliação de pesquisas são muito necessárias e que dispositivos como este devem ser tomados como parte do processo decisório humano, e não como seu substituto.

RESUMEN

En este trabajo presentamos un dispositivo para generar herramientas de evaluación de proyectos de investigación orientados a colaborar con problemas nacionales de salud. Hicimos uso del modelo de producción de conocimiento de Stokes (1997) como un marco analítico que muestra la interacción entre dos dimensiones: la consideración del uso y la búsqueda del conocimiento fundamental. Hicimos un esfuerzo explícito por incorporar la participación social en la ciencia como una dimensión complementaria al conocimiento científico para generar un dispositivo basado en el modelo de Stokes. Al operacionalizarlo, presentamos un conjunto de escalas orientadoras que resultan útiles a la hora de afrontar la compleja tarea de generar herramientas de evaluación de proyectos de investigación. Concluimos que las propuestas metodológicas para la evaluación de la investigación son muy necesarias y que dispositivos como este deben ser tomados como parte del proceso de toma de decisiones humanas, no como un sustituto de eso.

KEYWORDS:

• Research evaluation
• health research
• STI for national problems

PALAVRAS-CHAVE:

• Avaliação da pesquisa
• pesquisa em saúde
• CTI para problemas nacionais

PALABRAS CLAVE:

• Evaluación de la investigación
• investigación en salud
• CTI para problemas nacionales

1. Introduction

If our aim is to analyze the potential of knowledge to collaborate in the solution of a national problem, we need to understand how this knowledge is produced. When dealing with health problems, this statement gains in size: knowledge from research activities has always been fundamental to generate new solutions (WHO et al. 2013). However, the focus on proposing specific tools to produce useful knowledge in this area is relatively recent. Therefore, often these processes are not accompanied by frameworks that could guide them in achieving STI policy goals: new approaches to research evaluation are much needed (Bozeman and Sarewitz 2011), particularly if we consider that STI policies are part of the myriad of social policies useful to promote development processes (Bianco, Gras, and Sutz 2016). As noted by Yegros-Yegros et al. (2020), health research tends to present an unbalanced agenda that privileges the needs of high-income regions over those coming from developing countries; this situation represents a huge potential for tools to foster knowledge production that could help to close this gap.

One of the key scientific knowledge production activities is the development of research projects. In this document, we propose a methodological framework to generate an assessment tool for research projects evaluation for STI policies oriented to attend knowledge demands from societal problems in the health field. Our tool is based on a operationalization of Stokes’ “Pasteur’s Quadrant” (Stokes 1997), that could serve as an analytical device to understand how research projects are oriented towards the solution of national problems. Configured in two axes, Stokes’ framework breaks the linear idea of a one-way road from basic to applied research: it is a two-dimensional view of knowledge generation and its possible application.

Our proposal is based on the experience of using this analytical framework in the case of diabetes research projects in Mexico (Natera et al. 2019), we will make use of this problem to illustrate how to operationalize it by defining three categories for each of the axes. These categories are conceived in terms of input (definition and initial requirements of projects), process (characteristics and development of research), and output (potential research results). The paper is structured as follows: Section 2 presents the literature review, in which we include a discussion about research evaluation for national problems in health and present the Stokes’ model. Section 3 makes a critical revision of the Pasteur Quadrant; it emphasizes the incorporation of social participation in science as a complementary dimension, and defines the variables of the conceptual model. Section 4 contains the operationalization of our health device, a set of orientating indicators that could serve as a guide to construct specific assessment tools for research projects oriented toward health issues. Finally, Section 5 contains the conclusions and final remarks.

2. Literature review

2.1. Research evaluation for national health problems

Research evaluation is a difficult task, especially when dealing with research projects. To begin with, evaluators have the responsibility of assessing a project that will generate new knowledge in unavoidably uncertain environments, where their criteria should leverage their research experience to foresee the potential of knowledge production from a set of activities. They are normally assisted by questionnaires, templates, and discussion processes among peers, and yet they all have to face the challenge of giving an expert opinion on an unknown object (Piovani 2014). Furthermore, if those tools fail to provide the adequate guidance, the evaluation process could be misled and policy objectives will probably be unmet. This issue grows in size when the research objects are centered on the attention of national problems, since they now also incorporate the enormous complexity of intricated social interactions that could have an effect on peoples’ living conditions.

The Leiden Manifesto (Hicks et al. 2015) is a sound statement against the indiscriminate use of blind tools, particularly quantitative ones, for research evaluation. They state that the abuse of bibliometric indicators is producing a perverse effect on the development of the scientific community, the pressure to produce papers included in high citation databases is creating a perverse behavior: the “publish or perish” working condition in science is preventing scientists from undertaking other activities than writing papers, forcing them to devote most of their time in paper-related tasks, while leaving little space for other scientific activities; particularly those that could link them to societal problems.

The effect of research evaluation systems on the possibility of profiting from the investment in science, technology and innovation (STI) is huge. Evidence suggests that evaluation systems are hampering the potential of applying knowledge to solve development issues (Arocena and Sutz 2001), particularly when STI capabilities are required to unravel the needs of marginalized groups, seeing as the time needed to create participatory spaces, trust, and to fully understand the nature of a problem competes with the necessity of producing outcomes that are valued within the evaluation systems. Additionally, the publication structure comes with a hegemonic burden, in which advanced countries possess the majority of the high-impact scientific journals, therefore documents should be written in English, and the research agenda is aligned with their interests: for researchers living in developing countries, this is a barrier to orientate their scientific production to local and contextualized agendas (Bianco, Gras, and Sutz 2016), or to produce transdisciplinary research that could combine different strands of knowledge (Rafols et al. 2012).

In Latin America, Gras (2022) surveyed 12 national STI agencies to analyze the evaluation practices for oriented research. In her analysis, we observe that most of the available evaluation tools are path-dependent: they have been adapted from previous assessment exercises, particularly those linked to basic research, and have been complemented with requirements of linkages with non-academic actors (for instance, asking for letters of interests from civil society organizations). The problem with this approach is that it does not fully embrace the complexity of dealing with national problems: the research logic should be transformed to incorporate social actors, giving them a new space in the decision-making process to approve projects related with their living standards. If the research object is related with health, the implications of having inadequate assessment tools might be more direct and of greater responsibility.

Rafols and Yegros (2017) investigated whether the research agenda on health is responding to social needs. The short and painful answer is: No. There is an unbalanced relationship between the investment level on specific diseases that is not proportional to the burden that they represent for health, not to mention the scandalous fact of the 90/10 gap – in health, 90% of the investments on research and development is oriented to 10% of the global population (Philip 2017). The generation of data about health problems and the integration of social actors are presented as the main recommendation to overcome this disequilibrium. Therefore, new ways of approaching and generating assessment tools are required; in the next section, we propose to draw from the Pasteur Quadrant to have an alternative device.

2.2. A model to frame scientific research: the Pasteur Quadrant

This part of the literature review is related to the knowledge production process. If we look at the possibilities of applying knowledge in order to collaborate in the solution of a national problem, such as diabetes, we need to understand how this knowledge is produced. An alternative is looking at Gibbons et al. (Nowotny, Scott, and Gibbons 2003) Mode 1 and Mode 2 descriptions. He organizes knowledge production in terms of the quest of new basic scientific knowledge (Mode 1), versus a problem-solving oriented strategy (Mode 2). However, he heuristically configured some specificities of each mode: multidisciplinary, variety of actors involved, and characteristics of knowledge’s main objective. Even though we find this very interesting and provoking, we decided to pursue another avenue related to the Stokes (1997)’ Pasteur Quadrant. We think that due to health research complexity, this latter approach provides a more interesting tool to analyze the structuring of a research project: it provides a stylized framework that organizes in a clear tool the characteristics of the knowledge production process.

The Stokes’ Pasteur Quadrant offers a two-dimensional perspective on knowledge creation and potential applications (see Figure 1). It is set up along two axes: (1) research driven by a search for fundamental comprehension, and (2) research driven by consideration of use. According to Stokes (1997), works in any region of Stokes’ two-dimensional space are required in any research system, directly challenging the linear notion of a one-way road from basic to applied research. In this framework, research is subdivided into four quadrants. The first, which he calls Bohr’s Quadrant for Niels Bohr’s work on the structure of the atom, comprises basic research. The second is called Pasteur’s Quadrant and encompasses what he calls “basic research inspired by the application.” The third is named the Edison’s Quadrant; it is the traditional applied research. Finally, there is an unnamed quadrant, corresponding to the absence of the two major research locomotives in the Stokes’ scheme, investigations that are not motivated by the consideration of their use nor by the search of fundamental understanding are located in this quadrant.

Figure 1. Stokes’ Pasteur Quadrant. Source: Stokes (1997).

According to Stokes (1997), the biologist Louis Pasteur, whose bacteriological studies were conducted at the request of the French wine industry, characterizes the work of basic scientists who pursue fundamental knowledge while choosing questions and methods constructed around potential real-world applications. Pasteur always undertook an applied study, however, he made fundamental contributions to science that spawned the entire field of microbiology and changed the way we view the cause and prevention of disease. Pasteur’s Quadrant illuminates a path where applied research is not opposed to scientific creativity or rigor, hence, he contributed to move away from the basic-versus-applied research logic, and his work suggests the idea of use-inspired basic research. Therefore, the Pasteur Quadrant has in mind Louis Pasteur’s work on immunology and vaccination, which both advanced our fundamental understanding of biology and, at the same time, saved countless lives. In this quadrant, scientists work to advance scientific theories and methods while addressing practical problems. In Pasteur’s Quadrant, (the upper right), we find research that both seeks to expand the frontiers of understanding, and draws inspiration from practical considerations. In addition to Pasteur and others, Stokes cites here research by John Maynard Keynes and by the Manhattan Project.

The upper-left quadrant is exemplified by the theoretical physicist Niels Bohr’s work; we could argue that the research on the atomic structure was conducted without any consideration of applications, despite the fact that many were possible. We take the chance to make here an important clarification: we do not believe that the progress of scientific knowledge, even without pre-identified application, comes without social value: on the one hand, because scientific activity is a social process that creates capabilities for current and future studies, and on the other, because we need to consider the dynamics of knowledge application, wherein we cannot predict the effects of new knowledge in the future. However, we do expect that by introducing the characteristics of recognized problems, the possibilities of solving today’s issues by incorporating knowledge will increase.

The work of inventor Thomas Edison, who did not focus on exploring the more profound scientific implications of his discoveries, and instead pursued commercially successful electrical light, represents the lower right quadrant of pure applied research. In this sense, he was more concerned with practical scientific questions than with the underlying theoretical implications of his discoveries and inventions.

According to Stokes, the lower-left quadrant is not unoccupied but rather filled with

research that systematically explores particular phenomena without having in mind either general explanatory objectives or any applied use to which the results will be put, a conception more at home with the broader German idea of Wissenschaft than it is with French or Anglo-American ideas of science. (Stokes 1997, 74)

Disciplines like art history may fit in this quadrant, as they are focused on specific phenomena, are not primarily searching for the fundamental understandings referred to here, nor are they seeking any kind of practical application.

3. The Pasteur model and the assessment of research projects in the health sector

Stokes (1997) uses this approach to study the National Institutes of Health in the United States between 1960 and 1970, and finds that they were very successful in supporting Pasteur-type research. We chose this approximation to knowledge application in the healthcare sector for this reason. Several works have been inspired by the idea of the Pasteur’s Quadrant to describe different research systems, research programs, setting a research agenda or classification of journals, such as Ahumada Barona and Miranda Miranda (2003), Simmons et al. (2005), Tsao et al. (2008), Balaram (2008), de Sousa, Zamudio lgami and de Souza Bido (2009), Tijssen (2010).

Our proposal draws particularly from previous research (Natera et al. 2019), in which we used a specific configuration of the Pasteur Quadrant to analyze the knowledge application potential of financed projects by the Mexican agency CONACYT (the Spanish acronym for the “Mexican National Council for Science and Technology”) for diabetes research. In that case, we analysed 303 projects, financed between 2002 and 2014, to see their allocation in the Stokes’ framework. Results were far from celebratory: even in programs specifically oriented towards developing possible solutions from research activities, we found that half of them do not express a clear potential use, and only between 10% and 20% of them could be considered as part of the Pasteur Quadrant.

The experience from that research made us think about the possibility of expanding this approach to undertake, not only ex-post evaluations – as we did with the projects already financed – but also ex-ante evaluations, that could provide new information to guide research projects’ approvals. In order to do so, we needed to follow a two-step procedure: (i) analyzing the social participation in science as a structuring dimension of the assessment tool. We did not follow this path in the referenced diabetes exercise (Natera et al. 2019) because the available information did not allow it, and we deeply missed it, as a fundamental way for understanding the alignment of research projects with societal challenges; (ii) revisiting each one of the axes to reflect on social participation, aiming at incorporating it vis-à-vis the already specified variables.

3.1. The Pasteur Quadrant meets social participation in health issues

This proposal considers that in order to evaluate a research project oriented to attend national health problems, the Pasteur Quadrant structure should be increased by incorporating social participation in science as a complementary dimension. We would never state that research activity is independent of its social context, being that our point of departure is that science is a social process. However, we do want to emphasize the explicit consideration of the social components of the research activities when peoples’ wellbeing is at the core of the problem. Furthermore, by making this claim, we aim at fostering the interactions with social groups as a strategy to increase knowledge application.

Our consideration of social participation in science is a subset of the process of the democratization of science (Arocena 2014; Delgado 2010), in which we aim at distinguishing the interaction between the development of scientific projects and the inclusion of social groups, from the broad concept of reconfiguring the social system to strengthen the use of knowledge as a development tool. Since we are presenting a device for project evaluation, we preferred to take this avenue to limit its scope. In a nutshell, by social participation in science we mean the inclusion process of social groups in scientific projects’ structure and activities. It comprehends the actors involved, their beliefs, interests, and values, the social context in which they live, their capability building process, their possibilities to orient the research agenda and to benefit from the results of research projects.

When dealing with health issues, there is a delegation process that we need to consider: there is a limitation of the participant's agency when deciding how to interact with knowledge applications. Not all health products, services or policies emerge from autonomous decisions from the actors involved, or the actors in need. In health scenarios, we often need another entity that can translate the knowledge we have available into a specific solution or application: typically, healthcare providers (medical personnel, nurses, social workers) act as intermediaries between the available knowledge and the obtaining of its benefits; most of the time, we need a capable mind to decide what we should take to improve our health. This delegation of authority makes it necessary to distinguish between two dynamic actors (Natera et al. 2020):

• – Knowledge users: actors who will put health knowledge to use and experiment with it; they may be policy makers working along with scientists to change a particular rule, researchers using novel methods to measure blood sugar levels, or patients learning how to self-administer new medications.

• – Knowledge beneficiaries: the set of the population that will benefit from the produced knowledge. It includes the people having greater wellbeing from a policy measure, the scientific community that will have quicker access to publications on glucose thanks to a new express test, or the patient who is self-administering the new treatment.

Both actors’ specifications depend on the kind of issue we are trying to solve, because they are not a priori determined. These two categories, “knowledge users” and “knowledge beneficiaries”, are proposed as an input for research evaluation, in order to assess how oriented those are projects to collaborate with health issues. “Knowledge users” exhibit a higher level of capabilities, which allows them to interact with new knowledge in a more active position, being able to decode and understand its nature. “Knowledge beneficiaries” belong to the population that is in need of the solution that could be implemented from new knowledge; they could have a more passive interaction with it. This is why we think that these two categories are not mutually exclusive.

Structuring a social participation vision in scientific activities requires taking an inclusive approach. Cozzens and Sutz (2014) provide useful reflection to make science, technology and innovation (STI) a process and a tool to deal with social exclusion. They foster an inclusive agenda in which the necessities of marginalized people are placed at the center of the STI objectives and activities, particularly by taking two approaches: (i) generating solutions that are useful to their needs; and (ii) incorporating social groups as agents of the STI processes. In our view, this approach calls for a set of sub-processes that could structure social participation: incorporating heterogeneous actors in the process, detection of knowledge demand, the possibility of knowledge co-production with social groups, having a strategy for communicating scientific results to broad audiences, and, specifically, when dealing with the health sector, the consideration of the process of social determination of health and the alignment of the research project to the health issues identified in the public agenda. We will develop these points in the following section.

3.2. Revisiting the axes of the model in the light of social participation in science

Stokes’ framework is structured in two axes. In this section, we will discuss each of them, while explicitly incorporating social participation in science as a structuring dimension. Following the previous exercise, we have structured the axes using a process approach: we analyze them in terms of inputs (what they require as a starting point to develop a research project), process (a characterization of the research project development), and outcomes (results from the research project). We have named scientific production variables those items that encompass the vision already discussed in Section 2.3, in which processes closer to the application of a scientific method are present. We will make a lighter discussion on these variables, given readers can find a more detailed explanation in the previous paper (Natera et al. 2019). Consequently, we have labeled social participation of science the items that represent the STI democratization process, as discussed in Section 3.1.

3.2.1. The “consideration about use” axis

The first axis analyzes whether research is oriented to the application of and/or knowledge use; it refers to the concrete use of research activities. In the adaptation we made in the diabetes exercise (Natera et al. 2019), we followed De Sousa, Zamudio lgami and de Souza Bido (2009), using categories that investigate the nature of the problem, the nature of the research, and the perspective of direct benefit for the scientific production variables. One of the main changes we are incorporating here is reflecting on the difference between Knowledge users and Knowledge beneficiaries. As the focus of this axis is the usability of knowledge, we centered all questions around Knowledge beneficiaries, particularly, changing the definition of the outcome variable to specify the procurement of a direct benefit (in exchange to immediate use). Table 1 summarizes the description of the main variables for this axis.

Table 1. Definition of variables for the “consideration about use” axis.

	Scientific production variables	Social participation in science variables
Input	Problem’s nature: it focuses on the definition of research objects, evaluating whether they are closer to theoretical or to practical emphasis. Healthcare applications, feasibility constitutes the principal guidance to understand the level of practical applications (Graham and Tetroe 2007).	Knowledge demand detection: the inclusion of mechanisms that make explicit social knowledge demands in health as an indicator of the orientation of the research project (Arocena and Sutz 2010). National problems call for a clear knowledge demand identification, in which the interests of social groups are at the core of the research project (Natera, Rojas Rajs, and Suárez 2016).
Process	Nature of research: it identifies the type of research that is being proposed in the project. Following the criteria proposed by the Frascati Manual (OECD 2015), this dimension measures whether a project is closer to basic research or to experimental developments.	Consideration of knowledge co-production processes: in a research project for health issues, the inclusion of beneficiaries in the research process is a capability building process that increases the possibility of producing knowledge aligned with their conditions, and even more important: it generates a capability building process in the social groups (Cozzens and Sutz 2014).
Outcome	Perspective of direct benefit: the identification of knowledge beneficiaries of a research project’s outcome is a clear signal of its usability. Therefore, the more specific the level of identification of knowledge beneficiaries, the greater the level of this dimension (Natera et al. 2020).	Social communication of scientific results: the communication strategy developed to diffuse the outcomes of a research project, increases its possibility of impact, increases social awareness of the potential of knowledge production to solve societal problems, and is a source of inspiration for other social groups in need (Rojas-Rajs and Soto 2013; Rojas Rajs and Natera 2019).

Source: authors' elaboration.

Let us discuss the social participation in science variables. “Knowledge demand detection” is the input variable of this axis. We identify it as a fundamental starting point, to ensure that research projects are useful in attending societal needs: defining a research objective that is divorced from social demands hardly provides a sound opportunity for applying useful knowledge on current issues (Arocena and Sutz 2010). In fact, we believe that one of the reasons that half of Mexican research projects for diabetes failed to exhibit a feasible application is found in the definition of a research agenda based only on researchers’ particular interests. The process in this dimension is represented by the “Consideration of knowledge co-production processes”: the learning possibilities that come from participating in STI processes are placed as the central activity in a research project. This might generate a virtuous cycle, in which the involvement of those who benefit from knowledge application generates new possibilities of knowledge benefits (Yazdizadeh, Majdzadeh, and Salmasian 2010). Finally, the outcome variable is represented by the “Social communication of scientific results,” because this defines the diffusion of knowledge among society: it creates a shared language between actors that allows knowledge flows, increasing their capabilities over time (Rojas-Rajs and Soto 2013; Rojas Rajs and Natera 2019).

3.2.2. The “search for fundamental understanding” axis

In the axis “Search for fundamental understanding,” projects are categorized according to their proximity or contributions to fundamental knowledge. Leveraging on the diabetes exercise (Natera et al. 2019) we use the categories knowledge requisites, knowledge generation process, and knowledge progress, to express the characteristics of the scientific production variables. Again, the main adaptation we have made on this current proposal is to move the attention to Knowledge users, not necessarily because actors in health scientific activities are all Knowledge users, but because the generation of knowledge should take into account the possibility of having agents that translate it to useful solutions. The description of the main variables for this axis is presented in Table 2.

Table 2. Definition of variables for the “search for fundamental understanding” axis.

	Scientific production variables	Social participation in science variables
Input	Knowledge requisites: it denotes the kind of knowledge that needs to be integrated into each project; therefore, it refers to disciplinary background of participant researchers’ profile. Being health a complex process, the use of inter or transdisciplinary approaches is preferable as a way to comprehend the characteristics of research on a health problem (Rycroft-Malone et al. 2016)	Participation of heterogeneous actors: as proposed by Gibbons (2000), when dealing with complex problems the articulation of heterogeneous actors is key to include different visions on the research problem. In the case of healthcare activities, the participation of the academic sector, the productive sector, the public sector, the health sector, and civil society organizations is required (Natera et al. 2020).
Process	Knowledge generation process: it refers to the characterization of the ontology of a research object. The closer a research project is to understanding the nature of a health issue, the greater the possibilities of getting multiple directions for future studies and potential strategies to deal with it (Graham et al. 2006).	Consideration of social determination of health: health issues occur in a set of complex processes, that includes historical context, socio-political conditions, geographic determination, and economic structures (Rocha and David 2015). The inclusion of the characteristics of these processes in the research project development, opens the door to fully understand the nature of the problem by developing research contextualized activities.
Outcome	Knowledge progress: it establishes progress in relation to the type of object it could produce. This could imply the generation of new data for a specific purpose, with a concrete and limited contribution, or the creation of new scientific approaches to a health issue, giving rise to new research lines (Natera et al. 2019).	Articulation with health issues: It is desired that research outcomes are closely related to the agenda of social groups in need. The alignment between research outcomes and the agenda of specific (and particularly large) social groups increases the possibility of generating impact on people’s welfare (Rojas Rajs and Natera 2019).

Source: authors' elaboration.

The input dimension of the social participation in science is represented by the participation of heterogeneous actors in the research project; it is much inspired by the propositions of Gibbons’ Mode 2 (2000), that calls for the integration of academic research with actors that possess different types of knowledge, particularly those that are deeply involved with the problem. The central process for this case is the consideration of social determination of health: knowledge production needs to take a process approach to health, in which causes and effects tend not to be directly related, instead, they are embedded in a complex non-linear process of biological and social interactions, endogenous and self-reinforcing structures that configure the relationship between health and disease (Rocha and David 2015). Finally, the outcome variable is expressed by the articulation of research outcomes with health issues: the assessment of these types of projects calls for a direct check of the products’ relationship with the health agenda for social groups in need; we need to ensure that the final results obtained will be aligned with the policies' objectives (FOLEC-CLACSO 2020; Rojas Rajs and Natera 2019).

4. The proposal: an operationalization of the Pasteur model

We are now ready to present an alternative to assess each of the variables previously discussed. We have constructed a five level scale for each variable, from one to five, the higher the number, the more the dimension is developed; which would be the more desirable level. Based on the revision proposed by Gras (2022), we observe that there is a need for practical tools when assessing research projects. The big problem to propose such tools is that project evaluation needs to consider the specificities of the country/region, the historical process of the construction of the scientific culture, and the particularities of the research objects that are included in policy action. Additionally, the capability level of the evaluation group also has to be considered when designing the evaluation instrument. It means overall that a universal assessment tool for health problems, if presented, would be useless. However, we could outline some guidelines of what to expect in this type of projects, while proposing a set of qualitative indicators that should be revisited, adapted, and tested before being presented as a final assessment tool (FOLEC-CLACSO 2020).

In Table 3, the scale for the “consideration about use” axis is presented. In the construction of this scale, we followed the same logic as in the definition of the variables: including Knowledge beneficiaries as the main actors in the process and relating them with the concepts for scientific knowledge production, and for the social participation in science.

Table 3. A guide for measuring the variables of the “consideration about use” axis.

		Measurement scale
Input	Problem’s nature	(1) Entirely theoretical. Possibilities for healthcare applications are unforeseen or very remote, (2) More theoretical than practical. There are possible healthcare applications but they are not fully accessible or require intermediate complex processes (3) Balance between theoretical and practical. Healthcare applications are feasible, but have to be developed (4) More practical than theoretical. Healthcare applications are clearly attainable, and (5) Entirely practical. Healthcare applications are available.
	Knowledge demand detection	(1) Knowledge demand is based on the interests of the scientific community, (2) Knowledge demand is based on the perception of the scientific community of the public agenda, (3) Knowledge demand is based on the perception of the scientific community of social groups’ issues, (4) Knowledge demand emerges from the interaction with social groups, (5) Knowledge demand attends specific revealed needs of social groups.
Process	Nature of research	(1) Pure basic research, (2) Oriented basic research, (3) Oriented basic research with some scopes of problem solving or practical applications, (4) Applied research to solve problems and/or generate practical applications, (5) Experimental development about new products, processes, materials, equipment, and systems or their improvement.
	Consideration of knowledge co-production processes	(1) Research project does not include beneficiaries’ perceptions, (2) Research projects includes secondary information of beneficiaries’ perceptions, (3) Research project shows interactions with knowledge from beneficiaries, (4) Knowledge beneficiaries have particular tasks in the research project, (5) Knowledge beneficiaries are fully integrated and participating in the research project.
Output	Direct benefit’s perspective	(1) Knowledge beneficiaries are not identified, (2) There are potential knowledge beneficiaries, (3) There are specific knowledge beneficiaries groups, (4) There are specific knowledge beneficiaries groups and there is a knowledge application method, (5) There are specific knowledge beneficiaries groups and there is a method and a specific action plan for knowledge application.
	Social communication of scientific results	(1) Publication of scientific articles and/or scientific books, (2) Publication of articles and books in specialized media for broad audiences, (3) Publication of articles, books or multimedia content in open media, (4) Integrating articles, books, multimedia content in open media with social networks, (5) Setting an interactive communication strategy for broad audiences.

Source: authors' elaboration.

In turn, Table 4 shows the scale for the “search for fundamental knowledge.” In this case, we included Knowledge users as the reference actors during the process, always taking into account the dimensions of scientific knowledge production and social participation in science.

Table 4. A guide for measuring the variables of the “search for fundamental knowledge” axis.

		Measurement scale
Input	Knowledge requisites	(1) Knowledge in one specific area, (2) Knowledge in at least two specific knowledge areas, (3) Multidisciplinary knowledge areas, (4) Interdisciplinary knowledge areas and (5) Transdisciplinary knowledge areas.
	Participation of heterogeneous actors	(1) Only academic actors participate in the research proposal, (2) Academic actors are linked with the productive sector, health sector or public sector, (3) There is an interaction between academic actors, the productive sector, health sector, and government, (4) There is an interaction between academic actors, the productive sector, health sector, public sector, and social groups, (5) Academic actors, the productive sector, health sector, government, and social groups are fully integrated in the project’s development.
Process	Knowledge generation process	(1) Generating specific data about a phenomenon – What is a specific level variable? (2) Specifying phenomenon's analytical dimensions – What should be measured? (3) Determining causal relationships – Why does it happen? (4) Exploring the association of the phenomena with established knowledge – What is it related to? (5) Characterizing and describing the nature of a phenomenon – What is it?
	Consideration of social determination of health	(1) Research project does not include considerations of the social determination of health, (2) Research project considers secondary information of the social determination of health, (3) Research project includes first-hand information on the social determination of health, (4) Research project proposes activities linked to the social determination of health, (5) Research project is structured around the characteristics of the social determination of health.
Output	Knowledge progress	(1) Support: Confirms already existing scientific or technological knowledge, (2) Minor: Explores relations to already existing scientific or technological knowledge, (3) Moderate: Establishing / transforming new data or new standards to already existing scientific or technological knowledge, (4) Significant: Creates a new research line, (5) Extraordinary: Creates a new research area or theme.
	Articulation with health issues	(1) Research outcomes are not related to social groups’ health issues, (2) Research outcomes are related to social groups’ health issues as a secondary objective, (3) Research outcomes are articulated with social groups’ health issues but do not identify any specific social group, (4) Research outcomes are articulated with specific social groups’ health issues, (5) Research outcomes are articulated with health issues that affect large parts of the population.

Source: authors' elaboration.

The full operationalization of this device in a final evaluation questionnaire has to be tailor-made. There is no space for one-size-fits-all measures when dealing with health issues (Natera et al. 2020). Once again, science and health are both social processes, complex and dynamic, in which the history and the context play a structuring role. Nevertheless, we claim that this proposal serves as an orientation when defining the specific set of questions to ask, which might be useful when having the demanding task of assessing knowledge production oriented to collaborate with national health issues. Our proposal for the evaluation of research projects is only an alternative, and is based on concrete evaluation experiences of projects (Natera et al. 2019). For this reason, it is important that it is held as a flexible device, not as a closed or unique proposal. Issues such as the scale levels (from 1 to 5), or the specific characteristics of research projects, can exhibit great variability in the health field, making it necessary to adjust the device case by case. Having research projects that clearly express this type of criteria will facilitate the construction of more specific and useful devices for research evaluation.

5. Conclusions

In this paper, we aimed at presenting a device to generate assessment tools for research projects oriented to collaborate with health issues. We discussed the challenges around the research evaluation process oriented to societal problems, and made use of the Stokes’ model (1997) for knowledge production as an analytical framework that shows the interaction between two dimensions: the consideration of use and the search of fundamental knowledge. Leveraging on a previous experience when analyzing research projects for diabetes in Mexico (Natera et al. 2019), we present a device to operationalize the Stokes’ model, considering input, process, and outcomes as structural parts of each axis.

We made an explicit effort to incorporate social participation in science as a complementary dimension to scientific knowledge production in the Stokes’ model. Dealing with national health issues means considering taking part of the democratization of STI activities, particularly from an inclusive development approach (Arocena and Sutz 2010). Health is embedded in a process of social determination in which history, geography, political, and economic conditions should be part of the knowledge generation process. This also implies that STI activities oriented to intervene on this area require the inclusion of heterogeneous actors, the search of knowledge co-production processes, the definition of a social communication strategy, and the alignment with the health agenda for social groups.

We presented a set of orientating scales (from 1 to 5, for each variable), to show how the concepts could be measured and to provide guidance when constructing the assessment tools to evaluate research projects. We firmly believe that generating a universal questionnaire for research projects’ evaluation is a mistake, seen as it would neglect the existence of specific conditions linked to the health social process. Nonetheless, the consideration of the dimensions and variables – including a proposition for determining different levels – is useful when dealing with the complex task of research projects’ assessment. In this sense, we hope this paper can be useful for policy action.

We would like to present two reflections that could be developed in future research. First, when dealing with societal challenges, new methodological proposals for research evaluation are much needed. Discussing possible criteria (in our case, devices!) to generate research assessments tools, could lead to generate stable processes that, over time, could support policy learning. Second: any health device is insufficient to present a full diagnosis by its own. Meaning the proposal we made should be retained as one more source of information for the decision-making process; the group of evaluators will definitely play a key role; the definition of tools does not mean the objectification of a complex process. In both science and health, we cannot disregard collective reasoning and human interaction.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

We acknowledge the support of Consejo Nacional de Humanidades, Ciencia y Tecnología (CONAHCYT) through the project entitled “Marco analítico y metodológico para estimar distancias cognitivas, institucionales, sociales y geográficas en la movilización del conocimiento basado en investigación,” founded by the “Fondo de Ciencia Básica” (A1-S-22942).

Notes on contributors

José Miguel Natera

José Miguel Natera Researcher at the Instituto de Investigaciones Económicas – Universidad Nacional Autónoma de México. He holds a Ph.D (2014, European Mention) and a M.A. (2010) in “Economics and Innovation Management” from the interuniversity program of the Complutense University of Madrid (UCM), the Autonomous Metropolitan University of Madrid (UAM), and the Polytechnic University of Madrid (UPM). He also holds a Master's of Arts in Society, Science and Technology in Europe from the University of Oslo (2010). He is a Production Engineer from the Simón Bolívar University (2006) and Industrial Organization Engineer accredited by the Spanish Ministry of Education (2010). He is a member of the National System of Researchers (Mexico), Level II.

Soledad Rojas-Rajs

Soledad Rojas-Rjas CONACYT Researcher – Metropolitan Autonomous University, professor at the Department of Health Care. She holds a PhD in Sciences in Collective Health from the Xochimilco Metropolitan Autonomous University (UAM-X), a Social Communicator with a Master's degree in Communication and Politics from the same university. She is a member of the National System of Researchers, Level I.