Evidence through the lens of bibliometrics—the case of Finnish higher education admission reform

ABSTRACT

A major admissions reform was carried out in Finnish higher education at the end of the last decade. This paper focuses on three main policy papers connected to the reform and examines the use and production of evidence therein. Drawing on bibliometric research and research on educational policy, we aim to provide insight into how to utilize citation analysis when examining evidence in educational policymaking. In the three policy papers, domestic legislature was cited the most frequently, and research from higher education institutions was cited the least. Affirmational citations were the most prominent; perfunctory, assumptive, conceptual, persuasive, contrastive, and negational citations were found to a lesser extent. When cross-examined in relation to citation types, sources were mostly cited as affirmational, with the exception of research from higher education institutions, which was cited as conceptual information. We also found that the writers cited their own previously written ministry-affiliated policy brief as central information in one of the policy papers connected to the reform. Our study is in line with the earlier literature showing that certain sources, such as domestic publications, are favoured over others. It also illustrates the different citation strategies experts employ to substantiate and legitimize educational policymaking.

KEYWORDS:

• Bibliometrics
• evidence
• educational policy
• information
• legitimization
• expert knowledge

Introduction

Current policy-making systems in modern Western societies are characterized by the overproduction of evidence and scientification. Policymakers have been found to turn to scientific knowledge and academic expertise for epistemic authority in order to base their decisions on the best information and practices possible (Alasuutari, 2018; Christensen, 2018; Steiner-Khamsi et al., 2020; Wiseman, 2010). However, evidence and policies do not work in a linear fashion and attempts to identify research content in adopted policies have succeeded only in making the so-called evidence-policy gap more visible (Innvaer et al., 2002; Oliver et al., 2014; Wiseman, 2010). To go beyond this discussion on the evidence-policy gap, some researchers have turned to the symbolic and performative functions of evidence, where evidence is first and foremost understood as the substantiation and legitimation of policies and an ongoing process of co-production and persuasion between experts and policymakers (Boswell, 2009; Cairney et al., 2016; Jasanoff, 1990). From this perspective, the relationship between information, arguments, and claims, as well as any specific circumstances, becomes key in trying to better understand how policymakers and experts use evidence to convince others of their arguments and make those claims stand up against opposition (Boswell, 2008, 2009; Desrosières, 1998; Sanderson, 2011; Wiseman, 2010).

Recently, there has been growing interest in citations and their connection to the use of evidence in policymaking. By analysing citation counts, Christensen (2018) found that there was a marked increase in citations to economic literature and orientation towards prestigious outlets of international economic discipline in the Norwegian policy advisory system. In turn, Baek et al. (2017) found in their study on Norwegian school reform that policymakers and experts made excessive use of references, citing highly specialized and issue-centred domestic publications. Similarly, by examining the Finnish National Core Curriculum 2014 with bibliometric network analysis, Volmari et al. (2022) found that local experts with ties to national policy institutions had the power to select relevant information as policy evidence, and this evidence, in turn, strengthened expert power. In contrast, Steiner-Khamsi et al. (2020) found in their bibliometric network analysis of Norwegian school reforms that there was little correspondence between expert knowledge and political knowledge, as the Ministry favoured government-commissioned texts over scientific evidence produced by expert commissions. Earlier, by examining the key sources of evidence in the coalition government’s 2010 White Paper on Education in England, Morris (2012) found that the use of data was selective, with a propensity to mix and match sources of comparison to legitimize preferred policy options.

In this paper, we examine the use and production of evidence in the case of the higher education admissions reform carried out in Finland between 2017 and 2020 by asking whose knowledge mattered, what information was being relied on (Baek et al., 2017; Christensen, 2018; Steiner-Khamsi et al., 2020), and how that information was being used (see, Morris, 2012; Sanderson, 2011). To do this, we mapped out citations to understand the relationships between information (e.g. Aksnes et al., 2019; Van Raan, 2019) in three central policy documents that have been connected to the higher education admissions reform, which has had a significant impact on the education pathways of individuals, higher education institutions, and the Finnish educational system at a national level. We focused on the intellectual function of citations, i.e. the information that citations symbolize (Aksnes et al., 2019), and their social function as a tool of persuasion, where the reader is to be convinced of the value and importance of what the author has to say (Gilbert, 1977). We understand evidence as information that is used to substantiate and legitimize arguments or positions set forth by experts and policymakers and as a way of solidifying certain understandings of a complex social reality (Alasuutari & Qadir, 2014; Boswell, 2009; Cairney, 2016).

By focusing on in-text citations rather than just references, our aim was to consider how one reference might be cited multiple times in different ways. We examined citations quantitatively to understand what the most cited sources and citation types were. We also examined citations qualitatively within the context of the texts to understand the different ways citations were used in the policy papers and to look beyond the idea that the quantity of citations is synonymous with their impact or importance (e.g. Baldi, 1998; Bornmann & Daniel, 2008; Gilbert, 1977; Hanney et al., 2005). Instead of focusing on those who use evidence and their networks, institutions, the policy process, or the impact evidence has on policies (e.g. Boswell, 2009; Weible et al., 2012), we focused on the citations themselves and what they could tell us about the production and use of evidence in educational policymaking. Following this, we formulated the following research questions: 1) What are the most used sources and citation types in the policy papers connected to the Finnish higher education admissions reform? 2) How are citations used in the text? By answering these questions, this study provides insights and contributes to the discussion on the use and production of evidence in educational policymaking (e.g. Baek et al., 2017; Steiner-Khamsi et al., 2020; Volmari et al., 2022).

In this paper, we examine how citations are used to produce evidence in educational policymaking. We contextualize the use of evidence as a form of knowledge production in educational policymaking and use a bibliometric approach to analyse citations in policy papers to understand what sources are cited and how. Based on the results, we contribute to the ongoing discussion on evidence-based policymaking by showing how experts use citations to support policies (e.g. Morris, 2012; Volmari et al., 2022) by favouring specialized issue-centred publications and domestic sources of information (e.g. Baek et al., 2017).

Evidence in educational policymaking

For policymakers, a strong evidence base and the means to produce evidence are important in pursuing and ultimately attaining better educational quality and equality, for example, through the use of comparative data (Boswell, 2009; Piattoeva & Saari, 2018; Wiseman, 2010). Different countries have their own frameworks for evidence usage, in which the collection and utilization of evidence as part of the policymaking process is outlined. The Organization for Economic Co-operation and Development (OECD, 2015) has acknowledged the sophisticated frameworks for evidence-based decision making in Finland, including consultation procedures with stakeholders, impact analysis, robust foresight analysis, and the Finnish ministries’ heavy reliance on state research institutes to provide evidence to inform the policy process. According to Pollitt (2013), the issues in evidence-based decision making in Finland do not stem from the evidence itself but from the challenges of fostering a culture and processes that promote evidence-based decisions. Similarly, one of the aims of OECD is to foster epistemic communities with shared views and arrangements on knowledge creation and utilization in member states and inside the OECD itself (Kallo, 2021).

While evidence is needed to formulate ideas, create expectations, and direct actions towards one potential future over other possible ones (Simon, 1997), the conceptualization of policymaking as a clear-cut rational process where the best evidence produces the best results describes, at best, an ideal (Cairney, 2016). Common barriers that have been found to hinder the use of evidence include a lack of time and resources, communication, the relevance and timeliness of research, and differences between political and academic cultures (Boswell & Smith, 2017; Cairney et al., 2016). Meanwhile, personal experiences and oral testimonies might have an even greater impact on educational policymaking than just empirical research (Milton, 2007; Saunders, (2007). as cited in Wiseman, 2010). Furthermore, while policymakers and experts might call upon comparative data about policies and practices in other countries to support their own policy proposals (Santos, 2021; Santos & Centeno, 2021), the actual analysis and interpretation of data between different systems can be superficial, and discussions about reliability and generalizability may not be considered (Morris, 2012; Wiseman, 2010; Young et al., 2010).

The use and production of evidence in policymaking can be described as a ‘what works’ approach (Wiseman, 2010), with an emphasis on quantitative empirical data (Volmari et al., 2022). Debates focus more on technical details, implementation (e.g. Boswell, 2009), and dealing with uncertainty (2021) rather than furthering theoretical knowledge related to research disciplines or methodologies. Accordingly, policymakers can be seen as being pragmatic towards evidence, taking a wide stance on what counts as research and preferring a variety of sources (e.g. Boswell, 2009; Cairney & Oliver, 2017) while valuing generalizability, relevance to practical questions, and objectiveness (Jasanoff, 1990; Young et al., 2010). The need for such evidence also generates a political mandate for its production, where experts and working committees are commissioned by policymakers to curate information and provide alternatives for developing the educational system (e.g. Kauko et al., 2021; Steiner-Khamsi et al., 2020; Volmari et al., 2022).

Working committees can be seen as a hybrid activity: experts and policymakers combine elements of scientific research and reasoning with social and political judgment while competing to present evidence in a particular way to be considered at different levels of the government (Cairney & Oliver, 2017; Cairney et al., 2016; Jasanoff, 1990; Simon, 1997). The value of experts in working committees stems from their specializations and competences, which are crucial to translating and choosing information as evidence (Boswell, 2009; Cairney et al., 2016; Volmari et al., 2022). For example, expertise pertaining to legislature might be required, as educational reforms are to be embedded in the legal framework through which the education system is organized. In turn, expert committees may emphasize technical and scientific argumentation and analytical knowledge in the policymaking process rather than beliefs, experiences, or values (Boswell, 2009; Jasanoff, 1990; Weible et al., 2012). The underlying risk is that by relying on the same experts repeatedly, policymakers create a network of ready-at-hand experts and give them considerable influence over political decisions, thereby inviting doubts of bias from opponents (e.g. Jasanoff, 1990; Volmari et al., 2022). In addition, these expert commissions can start to act like an ‘echo chamber’ in which knowledge and ideas that support existing beliefs and agendas continuously circle (Jasny et al., 2015; Weible et al., 2012). For example, Stanziola (2012) found that experts close to the policymaking process favoured research in line with existing agendas and relied on evidence to confirm the validity of new goals and reduce ambiguity surrounding the topic.

Finnish higher education admissions reform

In Finland, secondary education has two stages, each commonly taking three years of full-time study: lower secondary education (grades 6–9) and upper secondary education, which is further divided into academically oriented general education (preparing students for the matriculation examination) and vocational education and training (Ouakrim-Soivio & Kupiainen, 2020). Higher education follows this dual model: the universities focus on academic research and teaching, while the universities of applied sciences have a closer vocational and work-life connection. Regardless of which track a student chooses in secondary education, they may apply for higher education; however, the division of tracks orientates the choices that students make after compulsory education, and admissions can be very competitive (Nori et al., 2020).

Higher education is recognized due to its importance in the knowledge economy, and the successes or failures of higher education are increasingly discussed from economic viewpoints, with policymakers focusing on how higher education might better serve the governments’ economic goals (Kallunki et al., 2015; Olssen & Peters, 2005). Accordingly, the slow transition from secondary education to higher education has received attention as a broader characteristic of the Finnish education system and as a question of the national economy. The latest statistics indicate that the transition to higher education became even more involved during the 2010s: only 28% of freshly matriculated upper secondary students immediately continued on to higher education, despite 82% of them applying for a degree place (Official Statistics of Finland, 2020). Furthermore, over a quarter (28%) of students started their higher education studies after more than a two-year delay, though the proportion of delayed starters in the rest of Europe was 14% (2018). Consequently, several working groups have been tasked with drafting plans for a swifter and smoother transition to higher education, and the facilitation of young people’s transition into tertiary education was included in all government programmes in the 2010s (e.g. Finnish Government, 2019; Finnish Government, Prime Minister’s Office, 2011, 2015).

This discussion surrounding the perceived challenges and economic importance of admissions resulted in a centre-left coalition of prime ministers Rinne and Marin to include a higher education admissions reform in their government programme, which was to be carried out between 2017 and 2020. Before that, higher education institutions had already created a quota for applicants that had not previously accepted a degree place in higher education (Universities Act, 2009/558; Universities of Applied Sciences Act 932/2014). In their 2014 proposal to Parliament, the government estimated that ‘when a sufficient quantity of degree places is reserved for first-timers, they will be admitted to a greater extent’ (Finnish Government, Prime Minister’s Office, 2015, p. 17). However, there are indications that the quota has not had a visible impact on admission results, as the majority of the applicants were classified as first-timers during the first years of the quota (Ahola et al., 2018).

In the admissions reform, higher education institutions committed to certificate-based selection, meaning the weight of the entrance examinations would be lessened, while that of the matriculation exam certificate would be increased, so that by 2020, more than half of the student places would be filled based on matriculation exam grades (Finnish Ministry of Education and Culture, 2017). The change was significant; a few years earlier, only 15% of degree places in universities were filled based solely on matriculation exam performance (Ahola et al., 2018). The aim of certificate-based selection was to lighten the laborious and prolonged preparation for entrance examinations, which was seen as decelerating the transition to higher education (Finnish Ministry of Education and Culture, 2022). In some high-status disciplines, such as medicine and law, competitive admissions and demanding preparation for entrance examinations led to a shadow education market and commercial preparatory courses (Kosunen et al., 2021). In addition, the matriculation exam scoring models in certificate-based selection have aroused heated debate in Finland, as the models accentuate certain subjects by awarding them considerably higher scores, thereby leading students to study, for example, mathematics, regardless of their interests or the field they want to pursue (e.g. Holmström, 2018; Lyytinen, 2019). Indeed, the discussion surrounding the higher education admissions reform seems to have intensified after the fact. There has been increasing pressure for policymakers to respond to the public’s claims that the reform has failed to bring about a fairer and more streamlined admissions process, and the evidence and arguments that speak for the reform are under scrutiny.

Methods and data

To understand the use and production of evidence in the higher education admissions reform carried out in Finland between 2017 and 2020, three policy papers and their citations were examined. The first two were programme papers from the Ministry of Education and Culture: Valmiina valintoihin I (‘Ready for the selections I’; henceforth referred to as PP1), which was published in 2016, and Valmiina valintoihin II (‘Ready for the selections II’; henceforth referred to as PP2), which was published in 2017. The working committee that produced PP1 was assigned to draw up suggestions for better exploitation of the matriculation exam in higher education admissions and give their recommendations. The working committee that produced PP2 was assigned to supplement the suggestions made in the first paper and give their recommendations on improving the utilization of vocational qualifications in higher education admissions. The third policy paper included in this study was an operational programme called Korkeakoulujen opiskelijavalintojen kehittämisen toimenpiteet (‘The actions for developing actions in higher education’; henceforth referred to as OP), which was based on the other two papers and outlined the main actions to be taken in reforming admissions. The working committees themselves were comprised of Ministry officials and researchers from research institutions. Following this, we use the term expert broadly to refer to actors who have specialized content knowledge, either through their educational background, their work as a researchers, or through their position in organizations as experts.

We examined citations to identify relationships between different sources of information in the policy papers and understand what sources were cited the most and how (e.g. Aksnes et al., 2019; Brika et al., 2021; Van Raan, 2019). This inquiry was based on the conventions of scientific writing, where citations and references are used to credit previous research, display influences, partake in the development and history of a particular research field, and convince others of the value and importance of what the author has to say (Aksnes et al., 2019; Baldi, 1998; Bornmann & Daniel, 2008; Frandsen & Nicolaisen, 2017; Gilbert, 1977). There has been some debate as to what citations and citation counts can actually reveal about the impact and quality of research or citation behaviour, as reasons for citing and the importance of the references vary (e.g. Aksnes et al., 2019; Bornmann & Daniel, 2008; MacRoberts & MacRoberts, 2010; Rodríquez-Ruiz & Fernandéz-Menéndez, 2009). However, citing is not random, and by examining citations closely, both quantitatively and qualitatively, we aimed to find variations and patterns within the relationships between different sources of information while being aware of and expanding the limitations of what citations could tell us when examined only from a quantitative viewpoint (e.g. Aksnes et al., 2019; Cronin, 2005; Van Raan, 2019).

In the quantitative analysis, we examined the most used sources and citation types, while in the qualitative analysis, we examined how citations were used in context. To understand what the most cited sources were, the citations in the three policy papers were first analysed and categorized through inductive content analysis (Hsieh & Shannon, 2005; Mayring, 2000). Attention was given to the title of the reference and place of publication, with writers, year of publication, and language being secondary information. Citations were identified, categorized, and analysed in-text based on the title of the reference and the place of publication. Of 405 citations, 7 statistical citations were excluded due to being citation only showing the publication year, and it was unclear which statistic source was being referred to. Metatexts that referred to chapters, tables, or pages in the same policy paper were not counted as citations unless they referred to external sources of information. Citations were not excluded when there was no corresponding reference in the reference list, which was especially the case with the OP. For further analysis, citations were classified into six thematic categories depicting the most used sources: (1) legislature; (2) statistical sources, i.e. numerical data, statistical databases, and indicators; (3) political sources, i.e. policy papers and government proposals and releases; (4) organization and network websites; (5) research from research institutions, i.e. reports; and (6) research from higher education institutions, i.e. academic research from universities and universities of applied sciences.

To understand the most frequently used citation types, we utilized a unified typology by Bornmann and Daniel (2008; Table 1) to increase the validity of the qualitative analysis and help expand the bibliometric theory (e.g. Hsieh & Shannon, 2005). The typology includes the types of citations, their functions, and the percentage range of citation types in citing behaviour studies.

Table 1. Unified typology of types of citing, adapted from Bornmann and Daniel (2008, p. 66).

Types of citations	Function	Percentage range
Affirmational	Citing work confirms cited work; citing work is supported by cited work; citing work depends on cited work; citing work agrees with ideas or findings of cited work; citing work is strongly influenced by cited work	The percentages for this type of citations range from about 10% to 90%.
Assumptive	Citing work refers to assumed knowledge that is general/specific background; citing work refers to assumed knowledge in an historical account; citing work acknowledges cited work pioneers	5% to 50%.
Conceptual	Use of definitions, concepts, or theories of cited work	1% to 50%.
Contrastive	Citing work contrasts between the current work and cited work; citing work contrasts other works with each other; citing work is an alternative to cited work	5% to 40%.
Methodological	Use of materials, equipment, practical techniques, or tools of cited work; use of analysis methods, procedures, and design of cited work	5% to 45%.
Negational	Citing work disputes some aspects of cited work; citing work corrects/questions cited work; citing work negatively evaluates cited work	1% to 15%.
Perfunctory	Citing work makes a perfunctory reference to cited work; cited work is cited without additional comment; citing work makes a redundant reference to cited work; cited work is not apparently strictly relevant to the author’s immediate concerns	10% to 50%.
Persuasive	Cited work is cited in a ‘ceremonial fashion’; the cited work is authored by a recognized authority in the field	5% to 40%.

According to Bornmann and Daniel (2008) there are several factors (both academic and non-academic) that determine citing behaviour. This behaviour can be analysed and categorized by examining the functions that citations have by using information derived from local and global cues in the citing article, allowing for consideration of the range of citation types within a publication (Bornmann & Daniel, 2008; Garzone & Mercer, 2000). Accordingly, we categorized citations based on the tone and style in which sources were cited by examining the wording, citational strategies, and context of the citation in the text. The caveat here is that researchers must rely on their own subjective judgment when analysing why a certain citation was made in someone else’s publication and categorizing that citation (Bornmann & Daniel, 2008). Following this, we aimed to give an accurate textual description of the different citation types to contextualize our analysis and make the process more open to examination. For example, in the case of affirmational citing, we sought citations that either supported or confirmed the citing work, or instances when the cited work was the basis for citing work. In contrast, in perfunctory citations, we searched for passing notes or remarks or suggestions to look up a source for a more complete account, while in conceptual citations, we looked for citations that were determined by the use of definitions, concepts, or more abstract knowledge related to research disciplines. In the qualitative analysis, we cross-examined sources with citation types to understand how the sources were used in the text. This enabled us to locate and consider the textual context in which citations were embedded and give an account of their use.

Findings

We identified a total of 405 in-text citations in the three policy papers. The majority (n = 376, 97.3%) of citations were in Finnish. PP2 included 217 citations, PP1 171 citations and OP included only 17 citations. In the following text, we present the different sources and types of citations, concluding with cross tabulation of the sources in relation to types of citations.

Sources in policy papers

In the three policy papers, legislature was the most cited source, forming almost half of all citations. The second most cited source was political sources, which included political documents, such as previous programmes and working committee papers, as well as government proposals for new legislature. Statistical sources was the third most cited source category. The statistics were mainly retrieved from established statistical actors, such as Statistics Finland and Vipunen – Education Statistics Finland, which is a statistics portal jointly managed by the Ministry of Education and Culture and the Finnish National Agency for Education.

Organization and network websites, research from research institutions, and research from higher education institutions comprised less than 10% of in-text citations. Organization and network websites were cited to illustrate the different admissions criteria and selection procedures that higher educations’ institutions have. Research from research institutions included published reports and research from research institutions, such as the Finnish Education Evaluation Centre and VATT Institute of Economic Research. Research published by these organizations was categorized as research from research institutions rather than research from higher education institutions to highlight institutional differences, where the former can be argued to focus on research for policy and the latter on basic research (Byrne & Ozga, 2008). The rarest in-text citations were from research from higher education institutions: there were ten publications from the field of health science, four from the field of education, four from the field of social sciences, and one from the field of economics. The different types of sources with examples and frequencies are presented in Table 2.

Table 2. Sources in the policy document in-text citations.

Types of sources	Used for	Frequency	Example
Legislature	Relating the reform to existing legislature	47.3% (n = 192)	Universities Act, 2009 (Finland)
Political sources	Developmental ideas, giving credit and considering previous work	17.2% (n = 70)	Finnish Ministry of Education and Culture, 2017). Ready for selections. Reinforcing the use of matriculation examination in higher education admissions.
Statistic sources	Quantitative data, eg. number of students applying to higher education	12.9% (n = 50)	Statistics Finland (2016). Nearly 70% of new passers of the matriculation examination remained outside education
Organization and network websites	Comparing admission criteria and procedures in higher education institutions	11.8% (n = 48)	National Selection Cooperation Network in the Field of Education http://www.helsinki.fi/vakava/
Research from research institutions	Societal and economic perspectives	6.4% (n = 26)	Pekkarinen, T. & Sarvimäki, M. (2016) A better way to choose higher education students. VATT Policy Brief 1/2016.
Research from higher education institutions	Societal, academic, and economic perspectives	4.7% (n = 19)	Heikkilä, T. (2016). On the path of a physician. What motivates physicians in their career related decisions and are they happy with them? Publications of the University of Eastern Finland. Dissertations in Health Sciences.

Types of citations in the policy papers

In the three policy papers, affirmatory citations, where citations were mostly done in support of the paper or were central information to the paper, were the most prominent. The second most frequently used citing style was perfunctory, which was mostly encouragement to seek the cited work for more information or passing remarks. Assumptive citations, meaning references to assumed general or historical knowledge, were the third most common way of citing external sources. Conceptual citations, which refer to certain concepts, theories, or abstract information, were the fourth most common way of citing, while persuasive, contrastive, and negational citations were found to a lesser degree. It should also be noted that the original typology had a category for citations to methods and methodology, which was omitted due to having no citations. The percentages of the types of citations in the three policy papers were mostly within the same ranges as in the original typology in affirmational, perfunctory, assumptive, and conceptual citations. However, persuasive, contrastive, negational, and methodological citations were well below the ranges of the original typology. This indicates that sources were cited one dimensionally, rather than engaging in a discussion with a multitude of viewpoints, methodologies, and critical stances. The types of citations are presented in Table 3.

Table 3. Types of citations.

Types of citations	Used for	Frequency
Affirmational	In supporting and confirming the citing work. Cited work has an influence on citing work.	65.8% (n = 267)
Perfunctory	As a passing note or remark. No explicit focus given to the cited work.	18.5% (n = 75)
Assumptive	In referring to assumed knowledge as background information.	5.2% (n = 21)
Conceptual	Referring to certain definitions, concepts, or theories of cited work.	4.9% (n = 20)
Persuasive	Referring to an authority.	2.7% (n = 11)
Contrastive	Contrasting and comparing relation to citing work and cited works	2.5% (n = 10)
Negational	Correcting, disputing, or questioning the cited work	0.2% (n = 1)

Contextual analysis of citations in the policy papers

Next, we examined the types of citations in relation to the sources used to better understand how different sources were cited, with consideration for the textual context in which the citations were embedded (Table 4). In four of the six sources, affirmational citations were the most common.

Table 4. Source * Types of citations.

	Affirmatory	Perfunctory	Assumptive	Conceptual	Persuasive	Contrastive	Negational
Legislature	n = 165	n = 19	n = 8	n = 0	n = 0	n = 0	n = 0
	85.9%	9.9%	4.2%	0.0%	0.0%	0.0%	0.0%
Political sources	n = 40	n = 6	n = 9	n = 0	n = 11	n = 4	n = 0
	57.1%	8.6%	12.9%	0.0%	15.7%	5.7%	0.0%
Statistic sources	n = 36	n = 13	n = 0	n = 0	n = 0	n = 1	n = 0
	72.0%	26.0%	0.0%	0.0%	0.0%	2.0%	0.0%
Organization and network websites	n = 9	n = 31	n = 3	n = 1	n = 0	n = 3	n = 1
	18.8%	64.6%	6.3%	2.1%	0.0%	6.3%	2.1%
Research from research institutions	n = 12	n = 6	n = 1	n = 5	n = 0	n = 2	n = 0
	46.2%	23.1%	3.8%	19.2%	0.0%	7.7%	0.0%
Research from higher education institutions	n = 5	n = 0	n = 0	n = 14	n = 0	n = 0	n = 0
	26.3%	0.0%	0.0%	73.7%	0.0%	0.0%	0.0%

Legislature pertaining to higher education institutions was cited to emphasize the autonomy of the institutions in relation to their admissions criteria and procedures, for example, ‘According to legislature (YoL 36 §, AMKL 28 §), higher education institutions decide on student admission criteria. It can be said that the autonomy higher education institutions have over admission criteria in Finland is especially extensive’ (PP1, p. 14). It should be noted that in many cases, the affirmatory citation of legislature was not about legislature being supportive of the reform itself, rather that legislature was the foundation and main source of the policy paper exploring legislative changes, such as in the following: ‘according to the legislature (630/1998) 2§, the goal of vocational education is to increase the vocational competence of citizens, develop work life and react to its competence needs, promote employability and entrepreneurialism, and support lifelong learning’ (PP2, p. 14).

Political sources were also mostly cited in an affirmational way. For example, the OP cited PP1 and PP2 to underline that ‘the working group report(PP1) that proposed increasing the weight of certificate-based admissions had a positive reception from commentors in January 2016. Likewise, the working group paper(PP2) that was published on 3.5.2017 that examined how to better consider vocational education was received mostly positively’ (OP, p. 4).

Statistical sources were cited in an affirmational way to provide information upon which to act. One table of figures was cited and examined closely to illustrate how different ways of selection had different admissions percentages, paving the way to the argument at the end of the paragraph: ‘it can be said that certificate-based selection is not represented enough, especially considering the well-known fact that success in matriculation examination correlates with academic success in higher education (e.g. 4.1.)’ (PP1, pp. 28–29). Here, the last citation, e.g. 4.1., was in reference to Chapter 4 of P1, where the conceptual premises were examined more closely. This also shows that considering the text before and after the citation itself is important in understanding the role it plays in the text.

Organization and network websites were mostly cited as perfunctory information, usually to notify the reader of admissions criteria in different institutions or of projects that were linked to developing admissions. Political sources were also cited as perfunctory information, and one government statement was cited as perfunctory, as shown by the use of ‘e.g’.: ‘The educational supply of higher education institutions is sized according to predictions regarding the needs of the labour market and industry (e.g. for example, OKM 2015)’ (PP1, p. 7). However, using abbreviations like ‘e.g’. did not always imply perfunctory citing. A policy paper from the Ministry of Education that was written a decade earlier was cited in an assumptive way to illustrate a more general historical premise behind admissions: ‘one of the basic principles in Finnish educational policy is the possibility for lifelong learning where no educational cul-de-sacs exist (e.g. for example, OPM 2008, p. 52)’ (PP1, p. 14).

Research from higher education institutions, in particular, was cited as conceptual information. This could be seen in reference to theoretical perspectives, such as ‘Keltikangas-Järvinen (2016), for example, has brought up that personality tests are irrelevant when examining aptitude’ (PP1, p. 45), or in the case of non-academic research as reference to concepts: ‘This argument is founded on vast literature examining the so-called two-sided matching markets’ (PP1, p. 53). However, in contrast to research from higher education institutions, research from research institutions was mostly cited as affirmatory information. For example, the working group members who wrote PP1 cited their own 2016 ministry-affiliated research institution policy brief as the main premise for the paper and as the overarching goal of the admissions reform itself, giving that one source considerable weight.

Discussion

In this study, we sought to discover what the most used sources and citation types in the policy papers connected to the Finnish higher education admissions reform were and how citations were used in the texts to better understand the use and production of evidence in educational policymaking. To do this, we turned to citation analysis, which enabled us to identify and map out information in the policy papers and locate the textual context in which the information was used. Our results highlight the domination of legislature in the citations, while research from higher education institutions was marginalized in the policy papers. The categories of the different sources, produced through inductive analysis, illustrate the variance that was present in the three policy papers, which is in line with previous research (eg., Boswell, 2009; Cairney & Oliver, 2017) regarding how all kinds of information are appreciated in the policymaking process. Yet, the prominence of legislature shows that policy papers can have a clear emphasis on certain viewpoints. Another explanation is that legislature was cited as evidence of the political mandate that the Ministry has over the educational system and as proof of the extensive autonomy of higher education institutions in Finland to highlight different institutional responsibilities and to emphasize the need for co-operation in changing the education system.

In the three policy papers, the types of citations we identified were mostly in line with the typology proposed by Bornmann and Daniel (2008), with affirmational citations being the most common. This was because the citations either constituted core information for the paper or because they were supportive of the arguments set forth by the writers of the policy papers. Overall, the lack of persuasive, contrastive, negational, and methodological citations, as well as an affirmational emphasis on legislature, political sources, and statistical sources, implies that sources were referred to one dimensionally and focused on implementing the higher education admissions reform rather than furthering theoretical research, engaging in critical discussion with a multitude of viewpoints, or genuinely exploring alternatives. Interestingly, even as different countries were referenced through organization and network websites regarding higher education admissions and what kinds of evaluative tools and methods they used, a clear majority of citations were in Finnish. This contrasts with previous findings underlining the importance of international data (Wiseman, 2010) and implies that local sources might be more important in the matter of national educational reforms (e.g. Volmari et al., 2022).

When examining the citations contextually, it became clear that the same sources of information played different roles in the text. While research from research institutions was mostly cited in an affirmational way, it was also cited conceptually. For example, a ministry-affiliated research institution’s report was cited for economic concepts stemming from the field of economics. This also shows that, while research from higher education institutions were mostly cited conceptually, they did not have a monopoly on information related to different research fields. One explanation is that conceptual information is learnt, internalized, and filtered by experts who might have a background in both worlds. These experts also refer to sources known to or made by them, rather than using new information, to better serve the paper. Related to this, we found that the working group members behind PP1 cited their own ministry-affiliated research institution policy brief in the goals for the paper and as economic and societal viewpoints supportive of the reform. The position of the writers, the cited paper as the basis for the goals of the working group, and the inclusion of multiple paragraphs based on the same source led us to believe that the source in question was central to the policy paper rather than just peripheral information. This confirms previous findings that local experts with ties to certain national institutions who have the power to choose the evidence (Volmari et al., 2022) might turn to information that is in line with the reform and supportive thereof (e.g. Stanziola, 2012), and that citation counts do not necessarily equate to importance, as said policy brief was only cited a few times (e.g. Baldi, 1998; Bornmann & Daniel, 2008; Gilbert, 1977; Hanney et al., 2005).

As the focus of this study was the use and production of evidence from the viewpoint of citations, it is not possible to infer the influence these publications have had on policies. This is especially true when considering that while experts and expert commissions have the power to curate and produce evidence, the government that asked for said evidence might ignore or include it at their own discretion (e.g. Jasanoff, 1990; Steiner-Khamsi et al., 2020). Furthermore, even as the current study builds upon and, to some extent, confirms previous research in the field of bibliometrics, care must be taken to not transfer the methods or results of bibliometric analysis made in the context of academic research to the context of policymaking, or to impose the logic of scientific writing on the writing of policy papers.

It should be noted that we examined the three policy papers together rather than as single publications, and that different decisions regarding categorizations would have produced different results. For example, categorizing research from research institutions the same as research from higher education institutions would have given a different picture of the role of research knowledge in this study. Moreover, the lines between the different citation types were not always clear, and some citations could have been placed in multiple categories. There were also differences between the policy documents with regard to the accuracy of their citations: PP1 cited statistics in a way that enabled checking the corresponding statistics from the database, but in PP2, statistics cited only the database in general, which does not enable the reader to follow up and check the statistics on their own.

More research is needed on the citing behaviour of experts who take part in policy processes and their possible motives for including or excluding certain pieces of work. An inherent limitation of citation analysis is that it focuses on text and does not consider other ways of using and producing evidence, which relates to the idea that personal experiences and social networks might influence the use of evidence more than just empirical research.

Conclusions

Our paper shows that examining the knowledge base of educational reforms through bibliometric lenses can provide insight into the use and production of evidence, where citations help us map the relationships between and understand the roles of different sources of information. Our study shows that evidence does not come to being on its own, but rather in the interactions of experts and information, where certain sources are favoured over others, and different citation strategies are employed to substantiate and legitimize educational policymaking. More specifically, our study supports the findings of previous research showing that domestic publications are extensively cited to support policy proposals and that experts play a central role in selecting information as evidence. At the same time, our paper gives an important account of the change of the Finnish education system and how the flow of information between experts and written text was used to choose a future where a certificate-based selection favouring first-time applicants replaced the previous admissions model that was said to require extensive preparation from applicants. While the use and production of evidence may be important in supporting a democratic system, over-reliance on sources of information requiring specific expertise can narrow the possibilities for non-experts to participate, where a deliberative process involving multitudes of voices is replaced with issue-specific and technical discussions.

Declarations and ethics statements

No potential conflicts of interest were reported by the authors. The data that support the findings of this study are available from the corresponding author, [JF], upon reasonable request.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

No external funding was received by the authors.