A systematic review of CDIO knowledge library publications (2010 – 2020): An Overview of trends and recommendations for future research

ABSTRACT

The Conceiving – Designing – Implementing – Operating (CDIO) Initiative identifies itself as a global educational framework for producing the next generation of engineers. The purpose of this systematic review is to provide an overview of trends and to consider how these may be optimised for the continued evolution of the initiative. This systematic review follows PRISMA guidelines and is preregistered on The Open Science Framework (OSF). The review includes all publications within the CDIO knowledge library between 2010 and 2020 (N = 898). Each of the publications was categorised as Advances in CDIO, CDIO Implementation and Engineering Educational Research. The initial screening identified the popularity of publishing articles discussing CDIO implementation and the notable decline in CDIO publications. A second screening took place that included all the publications in the Engineering Educational Research category. Some of the findings include: 1) 43% of publications have links with Nordic institutions. 2) Sweden is the most active country. 3) 81% of the publications are completed collaboratively; however, only 22% are cross institutional collaborations. The paper concludes with three main suggestions for future research: 1) Enhancing evidence-based practice 2) Support of blended learning research and 3) Further development of collaboration & replication efforts.

KEYWORDS:

• CDIO
• Engineering education research
• Systematic review
• PRISMA
• Evidenced-based practice

1. Introduction

The Conceive, Design, Implement, Operate (CDIO) initiative was first conceptualised in the late 1990s with the primary goal of equipping 21^st century engineers with the necessary skills to meet the demands of modern industry. To this end, the CDIO initiative introduced its first syllabus in 2001 (CDIO Syllabus 1.0), later followed by the CDIO standards (CDIO standards 1.0) and an initiative comprising 10 renowned schools of engineering in 2004. Over the course of this period, the initiative rapidly grew in size and began holding annual CDIO conferences, the first of which was held in 2005. As of 2022, the CDIO initiative has held 17 international CDIO conferences in 12 different countries. Since its inception the CDIO initiative has continued to evolve to meet the needs of practitioners (Crawley et al. 2011; Malmqvist, Edström, and Rosen 2020). This has occurred alongside fundamental reconceptualisation of engineering education on a global level (Case 2017).

However, the CDIO community might be faced with new challenges in the coming years that can require significant developments to new and current pedagogical strategies (Graham 2018; Meikleham et al. 2018). For example, societal changes, such as increasing cohort sizes, flexible, on demand delivery methods, increasing running costs, advancements in science and technology, as well as the developments in teaching and learning are expected to accelerate these ongoing changes in the coming decades (Graham 2018; Kamp 2021). Graham (2018) anticipates that the future of engineering education will include (i) the blending of resource-intensive, on-campus, active learning experiences with off-campus online learning, (ii) an increased flexibility, choice and diversification offered to students, and (iii) curricula that bring together cross-disciplinary learning, human-centred engineering and global experiences. In addition, the recent COVID-19 pandemic has added an increased urgency underlying such developments. This has resulted in the rapid adoption of online and blended practices with limited time to provide supporting resources and to develop specialist blended pedagogies. In turn, these ongoing changes have highlighted the need for a suitable evidence based on which to base our evolving practices within engineering education (Power 2021a).

In order to integrate evidence into practice it is essential to consider the many different types of evidence. This presents a particular difficulty in engineering education, as the complexities of engaging with competing methodologies and research designs require considerable specialist knowledge. This challenge is especially relevant to engineering educators who typically have little formal education in the social science disciplines that compose the evidence base (Power 2021b). Furthermore, scholars such as Buckley, Hyland, and Seery (2021) have raised concerns about the replicability and methodological rigour involved in developing the evidence base in engineering and technology education research. In order to guide practitioners in this initial engagement, Glover et al. (2006) developed an evidence pyramid, for use in the medical fields, as presented in Figure 1. We have modified this figure to represent broad categories of publications that are typically disseminated within engineering education. The modifications to the evidence pyramid include the introduction of a section of the pyramid labelled ‘Implementation reports’ which is situated between ‘case-controlled studies’ and ‘Expert opinion’. This category was added to differentiate between opinion pieces, implementation reports and research efforts on clearly defined point/s of interest. The second modification was the removal of two sections labelled ‘critical appraisal individual articles’ and ‘critical appraisal topic’ as these aren’t commonly used within engineering educational research and are more applicable to the medical field within which the pyramid was originally designed.

Figure 1. Evidence pyramid.

The base of the pyramid is composed primarily of expert opinions. These opinions serve as a useful entry point for researchers as this means of dissemination typically eschews complex terminology. Implementation Reports are primarily descriptive and would typically describe the experience of engineering educators and the perceived effectiveness of curriculum reform. Case–Control Studies typically seek to evaluate the impact of an educational intervention, where the case may represent the new experience group and the control may include a group that is taking the traditional version, or even a group who has taken the traditional version in a preceding academic year. Cohort Studies are slightly more sophisticated in their design and frequently include repeated measures within one cohort with a view to explaining performance changes within that group. It should be noted that these approaches include quantitative, qualitative and mixed methods approaches. Randomised Control Trials are rarely seen within engineering education. These very sophisticated designs can control for many confounding variables, resulting in robust conclusions. However, they require very large sample sizes across a wide range of contexts and are logistically challenging. Moreover, educational settings present a myriad of contextual variables that cannot always be controlled.

As outlined by Kamp (2021), CDIO has provided a wealth of supporting resources in the past that could primarily be described as Implementation reports. Kamp goes on to posit that in order to support engineering educators to enhance their practice, we require an increase in resources that could be classed as Case–Control Studies and Cohort Studies using the classification system originally developed by Glover et al. (2006) presented in Figure 1. In the context of this review, the modified evidence pyramid is posited as a useful system to consider past, current and future CDIO publications so that potential research avenues could be identified and built upon. Using this system to consider CDIO publications has the potential to contribute to the progression of evidence-based practices in engineering education.

Previous reviews of CDIO knowledge library publications (Li and Bennedsen 2014; Malmqvist et al. 2019) reported unique insights into the publishing trends of the CDIO community. These trends included inter alia reports on aspects, such as the number of annual conference publications, author/s involved, the degree of co-authorship, institutional links, reference usage and keyword co-occurrence. However, these reviews have not yet provided insight into the classification of research publications, methodology used and research topics investigated by the community. Such classification has the potential to identify holes in the existing literature base and to inform areas of future improvements and points of exploration for the community. For the purposes of this study all items extracted from the CDIO Knowledge Library will be categorised as either Advances in CDIO, CDIO Implementation or Engineering Education Research as previously used by the CDIO initiative in the presentation of articles in their annual conference. These categories are considered relative to the evidence-based pyramid shown in Figure 1.

• Advances in CDIO include updates/developments to guiding standards, policies and curricula. This would align with the foundational layer of the pyramid as an expert opinion.

• CDIO Implementation focuses on the practical design and delivery of CDIO efforts. Reports are typically descriptive and as such align with the second level of the evidence-based pyramid.

• Engineering Education Research focuses on articles that include novel analysis or synthesis. These are primarily represented within the evidenced based pyramid by the remaining upper layers.

In order to take a current ‘snapshot’ (McLure, Tang, and Williams 2022, 10) of the CDIO landscape and building on past research efforts (Li and Bennedsen 2014; Malmqvist et al. 2019), while also addressing some of the anticipated challenges outlined by Graham (2018) in light of future developments in engineering education research, we are limiting the review between the years 2010 and 2020.

1.1. Aim and objectives

As the CDIO initiative continues to evolve, it is essential that past trends are considered so that future efforts can meaningfully support practitioners in a rapidly changing global context. As such, the aim of this systematic review is to identify and evaluate research trends within the CDIO conference.

This systematic review has the following objectives:

1. To provide an overview of methodological designs within ‘Engineering Education Research’ items hosted within the CDIO Knowledge Library (2010 – 2020)¹

2. To identify potential limitations in the existing literature base

3. To inform future research directions associated with the CDIO initiative

By realising these objectives, this review might identify potential opportunities to enhance the support of practitioners and ensure the continued development of the initiative.

2. Methodology

2.1. Methodology overview

This systematic review follows the PRISMA 2020 Guidelines. PRISMA is an evidence-based approach requiring a minimum set of items for reporting in systematic reviews and meta-analyses (Page et al. 2021). All authors of the article have employed the PRISMA guidelines for this systematic review, including the search database, strategy, inclusion and exclusion criteria, data extraction procedure, and PRISMA flowchart. In addition, the review adheres to an open science format to make scientific processes more transparent and results more accessible (Crüwell et al. 2019; Power 2021a). The Open Science Framework (OSF) database is used to facilitate this need.

The study includes two main screening processes. The first screening process (Part One) included categorisation of all documents within the CDIO knowledge library between the years 2010 to 2020. The second screening process (Part Two) included an in-depth review of all the documents categorised in the Engineering Educational Research category from the initial screening.

2.2. Eligibility criteria

All publications within the CDIO Knowledge Library between the years 2010 to 2020, that meet the eligibility criteria, were included in the study. Each of the publications were screened using the inclusion and exclusion criteria. Please note that all excluded documents are highlighted within the open dataset on OSF, with an accompanying note to outline justification for exclusion.

2.2.1. Inclusion

• All CDIO conference papers need to be between the years 2010 and 2020.

• All papers included needed to be accessible within the CDIO Knowledge Library.

• To be included in the systematic review the paper needs to be in the English language or translated into English.

2.2.2. Exclusion

• Conference workshop documents stored within the Knowledge library are to be excluded.

• Conference slideshow presentations stored within the Knowledge library are to be excluded.

• Conference papers inaccessible within the CDIO Knowledge library at the time of search are to be excluded.

2.3. Information sources

The CDIO Knowledge Library database was searched between 1 June 2021 and 4 March 2022. All of the publications screened had been published in the CDIO conference proceedings between 2010 and 2020 (inclusive), as seen in Table 1. The CDIO Knowledge library database is available at http://www.cdio.org/knowledge-library.

Table 1. CDIO conference proceedings.

Year	CDIO Conference Proceedings:
2020	Proceedings of the 16th International CDIO Conference, hosted online, Chalmers University of Technology, June 8–10 2020 (Volume 1 & 2)
2019	Proceedings of the 15th International CDIO Conference, Aarhus University Denmark, June 25–27 2019, with activities on 24 and 28 June
2018	Proceedings of the 14th international CDIO Conference, Kanazawa, Japan, June 28 - 2 July 2018
2017	Proceedings of the 13th International CDIO Conference, Calgary, Canada, June 18–22 2017
2016	Proceedings of the 12th International CDIO Conference, Turku, Finland, June 12–16 2016
2015	Proceedings of the 11th International CDIO Conference, Chengdu, China, June 8–11 2015
2014	Proceedings of the 10th International CDIO Conference, Barcelona, Spain, June 15–19 2014
2013	Proceedings of the 9th International CDIO Conference, MIT, USA
2012	Proceedings of the 8th International CDIO Conference, Queensland University of Technology, Canada
2011	Proceedings of the 7th International CDIO Conference, DTU, Denmark
2010	Proceedings of the 6th International CDIO Conference, École Polytechnique de Montréal, Canada

2.4. Search strategy

All publications between 2010 and 2020 were exported from the CDIO knowledge library. Each publication was then compiled within Excel for screening, categorisation and analysis.

2.5. Selection process

2.5.1. Part one

The initial screening process involved screening each record twice by the first author. This screening process was to identify records to be excluded, such as irretrievable documents and items that did not meet the inclusion criteria. This was done by reviewing the full paper in its published CDIO proceedings format. The second screening was to determine the appropriate categorisation of the remaining papers. These categories included 1) Advances in the CDIO, 2) CDIO Implementation and 3) Engineering Educational Research. The first author extracted the aforementioned categories from the categorisation included by the CDIO Initiative in the 16^th to 12^th CDIO conference proceeding documents (Table 2). The criteria were then inductively created from the publications included within each category. Where classification was initially uncertain the first author marked the relevant entries for in depth review and discussion with the second author. This was done periodically so that a clear understanding of classification criteria was actively developed and utilised for subsequent reviews. At the conclusion of the first author review process, the second author reviewed each entry and where uncertainty arose an additional evaluation and discussion was completed by both the 1^st and 2^nd author in order to ensure quality and agreement.

Table 2. CDIO categories.

No.	Categories	Criteria (Outlined/Discussed in the Paper)
1.	Advances in CDIO	Changes to the CDIO Suggested improvements to the CDIO CDIO Initiative/Policy Research
2.	CDIO Implementation	CDIO is being implemented Improvements to implementation of CDIO Research carried out on the effectiveness of CDIO implementation
3.	Engineering Education Research	Research on engineering education outside the prior two categories (Advances in CDIO & CDIO Implementation) Reviews (Narrative, systematic, etc.) Opinion Articles Implementation Articles Case Studies

2.5.2. Part two

The second screening process involved a full document review of the research publications categorised as Engineering Education Research. This screening process examined Engineering Education Research category as the authors’ focus was placed primarily on reviewing publications on the upper levels of the evidence-based pyramid as per the aims of the review (Figure 1). The first author screened each publication categorised as Engineering Education Research to gather various data points, including research paper type, research approach, data collection method/s, sample size, cohort, country of origin, author/s and keywords. This additional screening process was implemented to provide a greater level of granularity regarding the types of research conducted by the CDIO community within the higher ranks of the hierarchy of evidence as shown in Figure 1.

2.6. Data

Excel was used to compile information from each of the publications included in the systematic review. The open data file is available at [https://osf.io/k3nzq/?view_only=c4292e44d248430798a3c7694d340fb3].

2.6.1. Part one

Information collected on each publication includes conference location, conference year, document title, 1^st Author’s name, categorisation of documents (Advances in the CDIO, CDIO Implementation and Engineering Educational Research) and abstract.

2.6.2. Part two

Additional information collected in relevant publications includes research paper type, research approach, data collection method/s, sample size, cohort, country of origin, all authors names and keywords.

3. Results

3.1. Study selection (flow of studies)

The PRISMA flow chart (Figure 2) represents the screening and analyses processes for documents at each phase of the systematic review. The flow chart has five main phases: Identification, Screening, Included (Part 1), Screening and Included (Part 2).

Figure 2. PRISMA flow diagram.

3.2. Study selection (excluded studies)

During the first screening (Part 1), 19 publications in total were excluded from the dataset. The publications were excluded according to the inclusion and exclusion criteria. An example of an exclude article is the publication Exploration and practice of the CDIO engineering education reform control system. The publication was included within the CDIO database; however, the full document wasn’t available for retrieval. Therefore, this publication was excluded from the study.

3.3. Risk of bias in studies

It’s worth noting, that for review type publications, such as narrative reviews, systematic reviews and meta-analyses, the biggest treat to validity is publication bias (Banks, Kepes, and Banks 2012; Rothstein, Sutton, and Borenstein 2005). To ensure this review avoided common issues such as absent publications (Banks, Kepes, and Banks 2012; Dickersin 2006; McDaniel, Rothstein, and Whetzel 2006), all publications were included, from within the CDIO Knowledge Library between the specified years. Any excluded papers were outlined clearly within the open data file available on the OSF.

Additionally, the authors acknowledge that the studies included within this systematic review could have a risk of publication bias. Since 2009, papers submitted to the CDIO conference follow a peer-review process before acceptance (Edström and Kolmos 2014). However, it is currently not known whether the CDIO could potentially apply limits, such as the number of accepted conference publications in each track or in total, due to practical constraints and as such this potential limitation is worth considering in the context of the current data.

3.4. All CDIO conference papers published between 2010–2020 (Part one)

A breakdown of the 879 CDIO conference publications included in the initial section of the systematic review can be found in Figure 3. The majority of the yearly publications are within the category CDIO Implementation. This category accounts for an average of 76% of yearly publications since 2010. This is expected as CDIO are primarily practitioners with a formal background in engineering disciplines, but with an immediate concern for increasing the effectiveness of their teaching (Kamp 2021). The minority of yearly publications are within the category Advances in CDIO, accounting for an average of only 6% of yearly publications since 2010. The number of papers published by the CDIO conference in each category annually is visually represented in Figure 3 and shows that the number of publications peaked in 2011 and 2016. The lowest number of publications being included in 2010, however closely followed by 2017 and 2020. The yellow dashed trend line represents the overall decrease in publications in CDIO publications from 2010 to 2020. This finding, combined with the 2005 to 2009 data from a 2014 methodological analysis paper (Li and Bennedsen 2014, 4), shows that publication numbers increase by 11 papers yearly from 2005 to 2011 before beginning to slowly decline by 5 papers per year from 2011 to 2020. Although the argument could be made that 2020 could likely have been affected by the COVID-19 pandemic, it’s worth noting that when 2020 is excluded the decrease is still visible with a decline of 4 papers yearly.

Figure 3. Visual representation of publications (annually).

3.5. All engineering educational research CDIO conference papers published between 2010–2020 (Part two)

A total of 159 CDIO publications out of 879 publications (18%) were categorised as Engineering Educational Research using the inclusion criteria (Table 2) between 2010 and 2020. The publications are categorised into their publication type: 1) Research Article, 2) Opinion Article, 3) Implementation Article (Not to be confused with CDIO implementation, which primarily focuses on the planning and implementation process of the CDIO teaching and learning strategies), 4) Review Article, 5) Teaching Strategy/Resource Article, 6) Framework Article and 7) Other. Each of the categories shown in Figure 4 is in order of quantity from left to right, excluding the category entitled Other. The category Other represents nine publications that weren’t easily categorisable. Two examples of paper categorised in Other include a PhD thesis summary and an article discussing future research.

Figure 4. Publication type (2010 – 2020).

Over half the publications within Engineering Education Research contain sample sizes ranging between 100 and 500 participants. The publications categorised as Research Article (90) can be divided into three research approaches Quantitative (26), Qualitative (22) and Mixed Methods (42). Figure 5 shows the yearly breakdown of Research Articles research approaches. The yellow line displays variation in publications yearly within this category, although the dashed yellow line represents a slight increase in publications over time. The most popular quantitative and qualitative measurement methods was surveys, questionnaires, interviews, and focus groups. It’s also worth noting that the vast majority of these publications report solely on data taken from student cohorts with very few reporting on data from a combination of stakeholders (Students, teachers, technicians, teaching assistance, etc).

Figure 5. Research articles - research approaches (annually).

A total of 13 (8%) of the 159 publications from the category Engineering Education Research discussed forms of online and blended learning. While research that investigates cross-disciplinary learning, human-centred engineering and global experiences (Graham 2018) were identified in 14 (9%) of the 159 publications. Associated words found include Interdisciplinary (Fouw, Klaassen, and Tang 2020), Interdisciplinarity (Kans and Gustafsson 2016; Klaassen et al. 2020), Transdisciplinary (Spooner, Raynauld, and Lalande 2011), Personas (Yström et al. 2010), Global change (Rosén et al. 2018), Sustainability, Sustainable development (Malmqvist, Rådberg, and Lundqvist 2015), Social responsibility (Brodeur 2012b), Cross-cultural awareness (Le and Le 2014), Intercultural communication (Josefsson 2010), Globalisation (Brodeur 2012a), Industry linkages (Roza 2010) to name a few. These papers for the most part discuss engineering education research conducted in a wide variety of curricula designs that bring together a mix of cross-disciplinary learning, human-centred engineering and global experiences. Although, the most common area of research among the 14 publications were forms of interdisciplinary research, which was visible within 8 (5%) publications. Additional publications discussing these three anticipated challenges can be found under Advances in CDIO and CDIO Implementation. However, the focus of this analysis is on Research Articles due to their higher rank within the hierarchy of evidence as shown in Figure 1.

All 159 publications can be linked back to 26 countries. Some publications have links to multiple countries due to co-authorship. 43% of these publications can be linked to Nordic institutions (Denmark, Finland, Norway, Sweden and Iceland). Sweden is the most active of any country, holding links with 42 (26%) of the publications, this is consistent with existing bibliographic research (Malmqvist et al. 2019). Figure 6 shows that Europe has the most significant number of links to these types of publications compared to the other continents. 70% of Engineering Education Research publications include an author from a European institution.

Figure 6. No. of linked with publications.

When reviewing all 159 publications, we identified the number of authors per publication: one (n = 30), two (58), three (34), four (15) and greater or equal than five (22). This outlined that 129 publications (81%) were completed in collaboration with two or more authors. However, only 35 publications (22%) were completed in cross institutional collaboration, most of which are produced by Nordic author (22).

4. Discussion

4.1. Discussion (limitations of evidence)

It should be noted that papers included in this review have been taken from a source that facilitates reports of practice (Edström 2017) and that alternative sources, including multiple highly regarded international journal articles, which contain additional examples of ‘Research’ category papers were omitted from this review. However, the CDIO is one of the most established engineering education collectives and arguably presents a unique insight into the focus of practitioners.

Furthermore, the systematic review only includes papers maintained by the CDIO knowledge library, potentially, additional papers were included in the CDIO annual conference proceedings, which haven’t been included in the database.

Additionally, the impact of the COVID-19 pandemic could explain a variation in publications during the 2020 period. The increase in academic workload on CDIO members and the move to a digital conference format could potentially have prolonged or ceased the completion of planned CDIO publications.

Finally, it has been observed that geographical location of CDIO conferences has an effect on contributions to the CDIO knowledge library (Malmqvist et al. 2019). Four of the 10 CDIO conferences were held in Nordic countries; this could be a significant factor in these countries’ sizeable levels of engagement.

4.2. Discussion (limitations of review processes)

Identifying the appropriate categorisation for the remaining 879 articles was determinant by the author’s criteria developed from past CDIO conference proceedings publications (16^th to 12^th CDIO proceedings). This criterion was designed to help categorise the earlier conference proceedings which didn’t include the categorisation of articles. Missing a predefined criterion by the CDIO opens the study to the potential risk for miscategorisation of some papers. However, to control for miscategorisation, the primary reviewer identified items that were not clearly within one category. These items were then re-reviewed by the first and second author and consensus was reached prior to final inclusion.

4.3. Discussion (implications)

4.3.1. CDIO and evidence-based practice

The CDIO and other comparable educational development approaches within organised international communities, such as problem/project-based learning (PBL) rose in popularity within engineering education in part by strong drivers from stakeholders, such as higher education institutions, graduates, political pressure and industry (Edström and Kolmos 2014). As pointed out by Edström and Kolmos (2014) the CDIO itself is easier to define than many other approaches, as its community have developed a coherent collection of supporting documentation, such as the syllabus and standards, which are updated and discussed regularly within the community (Malmqvist, Edström, and Rosén 2020; Malmqvist et al. 2022, 2019). In addition to providing the community with these documents the CDIO initiative also hold an annual CDIO conference, which we can see from the finding, focuses primarily on CDIO implementation. However, the CDIO is currently at a crossroads, as outlined in the recent publication by the CDIO Co-director Albert Kamp. Kamp (2021) states that it is optimistic to think that the CDIO’s continued focus on the syllabus, standards, and lasting emphasis on classroom implementation of conceive-design-build-operate projects will provide sufficient value for long-term community members. He adds that the goals these members had set for adapting the CDIO framework in their programmes have been fulfilled, and that new supports will be required as they advance their practice beyond implementation. This review further highlights an emphasis on publications dealing with CDIO implementation. Based on the conclusions outlined by Kamp (2021) and the trends identified within this review, we posit that a future focus on practitioner evidence-based practice would simultaneously support the ongoing global developments as identified by Graham (2018) and provide a valuable new direction for the CDIO initiative. This would see practitioners build on existing pedagogical strategies in the CDIO context, while also adding to the evidence base in the form of Case-Control and Cohort Studies. The notion of ‘evidence-based practice’ is well established in disciplines such as medicine, agriculture, transportation and technology, within education it is a comparatively new concept (Bauer and Prenzel 2012; Sackett et al. 1996; Slavin 2002). Evidence-based practice provides practitioners with evidence of ‘what works’ (Ratcliffe et al. 2005). Ratcliffe et al. (2005) adds that if research findings are to impact classroom practices, they need to be convincing findings that resonate with, or acknowledge, educators’ professional experience in their translation into practical strategies for classroom practice and are widely disseminated through professional networks. The CDIO already has a well-established network for effective dissemination of practice, but produces relatively few publications related to engineering education research. A considerable research base is a necessary precursor to broadly adopted evidence-based practice (Thomas and Pring 2004). The authors suggest that within this need, there exists an opportunity for the CDIO to assist practitioners to incorporate advanced research practices as they evaluate their own practice, and in turn continue to support new developments within engineering education.

4.3.2. Anticipated challenges

Engineering education as a whole has developed significantly since the beginning of the CDIO Initiative in 2004. New challenges such as increasing cohort sizes and blending of on-campus and off-campus learning (Graham 2018) require novel approaches to delivering effective engineering education. Graham (2018) outlines that concerns are rising around the capability of engineering institutions to deliver quality educational programmes to increasing cohort sizes, particularly when operating under limited budgets. Graham (2018) further suggests that the learning experiences of these large cohorts can be improved by employing a range of student-focused strategies. It is suggested that this could best be achieved through a combination of remote personalised online learning and on-campus hands-on experiential learning also known as blended learning. Blended learning post-COVID-19 has become an essential part of teaching and learning in the engineering classroom and has only further solidified its future role in engineering education (Saboowala and Mishra 2021). While it is encouraging that a considerable number of CDIO Engineering Educational Research are examining large cohorts, a further focus on large-scale blended learning approaches could inform the optimisation of these new approaches. When blended learning is carefully and thoughtfully implemented, it offers advantages for students and teachers, such as increased flexibility and engagement (Millson and Wilemon 2008), while maintaining a cost-effective program (Lefoe and Meyers 2006; Yigit et al. 2014). Additionally, blended learning research shows that it has positive effects on reducing dropout rates (López-Pérez, Pérez-López, and Rodríguez-Ariza 2011), while achieving higher student performance when compared to traditional face-to-face approaches (López-Pérez, Pérez-López, and Rodríguez-Ariza 2011; Means et al. 2013; Vo, Zhu, and Diep 2017). Using blended learning as a strategy to reducing dropout rates in engineering could be beneficial, when considered alongside increasing student cohorts this approach has considerable potential value within the discipline (Casanova et al. 2021; Min et al. 2011; Paura and Arhipova 2014). Active learning approach, such as the CDIO’s, to education has been shown to work well with the blended learning format were the benefits from both online and face-to-face environments are experienced (Bhadani et al. 2017; Mielikäinen 2021).

4.3.3. Blended learning

A US Department of Education meta-analysis found that students in blended classrooms outperformed students who only received face-to-face teaching (Means et al. 2009). Furthermore, research also shows that motivation and satisfaction are positively affected by blended learning (Mosca et al. 2010; López-Pérez, Pérez-López, and Rodríguez-Ariza 2011; Francis and Shannon 2013). However, it must be acknowledged that blended learning doesn’t come without its own new set of challenges. Redesigning, preparing and supporting online elements successfully in a blended learning model is resource and time intensive (Dykman and Davis 2008; Francis and Shannon 2013). Francis and Shannon (2013) further highlights the vast level of research individual staff members need to perform and the tools that must be implemented/learned before adoption occurs. An additional drawback outlined by Okaz (2015) is the lack of immediate support from the teaching team compared to the traditional face-to-face classroom environment. In the context of the CDIO, research focusing on forms of online and blended learning strategies is quite limited, with only 13 publications being present within the Engineering Educational Research category between the years 2010–2020. Focused research around the implementation of blended learning has the potential to alleviate some of the issues previously outlined by informing practice. If blended learning is to play a part in a new and innovative evidence-based practice orientated within the CDIO framework, further supporting research is needed.

4.3.4. Engineering education research

Although already present within the CDIO conference, Engineering Education Research was added as a standalone conference track in 2016 (Edström 2017). Edström (2017, 2020) states that Engineering Education Research as a dedicated track has the potential to further sharpen tools for educational development, while also increasing the available tools by adding new perspectives. However, on review of the finding’s, publications catogorised under Engineering Education Research since 2016 has plateaued if not fallen.

Based on the policies, papers and findings presented within this review the authors suggest that increased engineering education research is essential as engineering education continues to evolve in response to the various challenges outlined previously. This should not be conflated with a devaluing of practitioner experiences and reports. This can be complemented by journals in supporting engineering educators as they shift their focus from teaching and curriculum development, to exploring fundamental questions about engineering education, remains essential for our community (Borrego 2007). This issue is also echoed in A Systematic Review of Canadian Engineering Educational Research 2004–2017 by Brennan (Brennan et al. 2018). The inclusion of a diverse range of research practices can prove extremely beneficial to teaching and learning as outlined by le Roux et al. (2021).

When analysing the publications categorised as Engineering Educational Research, a distribution of 29% Quantitative, 24% Qualitative and 47% Mixed methods can be observed. This analysis identified several quantitative and qualitative measurement methods, the most popular methods being surveys, questionnaires, interviews, and focus groups. Other methods of such as observations, protocol analysis and task analysis have received limited attention. The diversity of research approaches seen in these publications can be considered as a strength, but also as a potential weakness (Power 2021a). Power (2021a) suggests that a wealth of methodological perspectives can be interpreted as a strength; however, the variety in approaches might limit the grouping of large amounts of studies to form more concrete conclusions. In the context of the results of this review, with a majority of studies coming from Northern Europe, there is a need to examine the results of these studies in different socio-economic settings. To achieve this ‘conceptual replication’ (Schmidt 2009) of studies needs to occur. ‘If education research is to be relied upon to develop sound policy and practice, then conducting replications on important findings is essential to moving towards a more reliable and trustworthy understanding of educational environments’ (Makel and Plucker 2014). The wide variety of approaches followed by CDIO research makes consistent replication of studies difficult in various sociocultural and pedagogical circumstances to form reliable and trustworthy understanding of educational environments. Conceptual replication also provides opportunity for early researchers to engage with researching their own practice and has the potential to increase collaboration internationally (Benson and Borrego 2015). Replication studies themselves are important inputs to systematic reviews and meta-analyses, which build certainty around conclusions (Frieler et al. 2013; Benson and Borrego 2015). Publications with well-developed instruments and good generalisability were seen as examples of best practice by the CDIO when examining particular pedagogical strategies. In that case, this could provide a foundation to collate rigorous reviews with strong conclusive results.

Cohorts reported on by CDIO papers categorised as Engineering Educational Research tend to be students solely. This isn’t unexpected, as the CDIO is a practitioner’s collective at its core. The community consists of practising members with a common interest in engineering education (Kamp 2021). Kamp (2021) describes the CDIO as a place of exploration, experimentation, evaluation and reflection for engineering practitioners. Although, it is worth noting that effective evaluation and reflection involve both the student and teacher perspectives (Agouridas and Race 2007), especially in tertiary education where multiple educators are present in individual modules. To this end, the CDIO can benefit from the inclusion of teachers, students and other relevant stakeholders in the research process for a more holistic view of the pedagogical strategies being investigated.

4.3.5. Cross-institutional collaboration

The majority of sample sizes represented by CDIO papers categorised as Engineering Educational Research are reasonably large cohorts, ranging between 100 and 500 participants. Medium to small effect sizes can be detected within this range, which reduces the likelihood of type II error (Serdar et al. 2021). Although these large samples are beneficial, it is important to note that many of these publications are being carried out by a small group of high performing northern European institutions. Although most articles include multiple authors, they primarily maintain collaboration within the same institution or country. Geographical location strongly determines how engineering education researchers collaborate (Jesiek, Borrego, and Beddoes 2008; Xian and Madhavan 2014). Xian and Madhavan (2014) states that the diffusion of research can be influenced by national and international organisations. The CDIO has the unique potential of using its existing platforms such as the website (CDIO Initiative 2022) and conference to promote cross-institutional collaborations by introducing workshops to develop capacity and grow international networks. Evidence of success in this area is evident within a host of CDIO publications, but further leveraging of this strength could help to develop practice in other regions. Although face-to-face meetings are preferred by most (Borrego, Froyd, and Hall 2010), online communication provides not only a budget-friendly substitution to engage in such workshops, but also eliminates the barriers caused by physical distance (Xian and Madhavan 2014) and limited budgets. Collaboration also supports replication and supports larger-scale studies leading to seminal publications as outlined by Borrego (2007). It’s also important to note that similar recommendation is also being made in other collectives (Xu et al. 2020)

4.3.6. Replication

Conceptual replication is also key when trying to develop conclusive research findings, often going hand in hand with collaboration. More recently, Buckley, Hyland, and Seery (2021) have emphasised, the crisis in educational research in general, and Technology and Engineering Education research in particular as an area of concern. As such, promoting collaboration efforts might support replication, which supports large-scale studies leading to seminal publications as outlined by Borrego (2007). Seminal publication will increase the associated rigour of pedagogical strategies implemented within the CDIO framework. Conceptual replication is limited within the CDIO and publications tend to follow a plethora of different methodologies. As Buckley, Hyland, and Seery (2021) notes, it is paramount that education researchers are aware of the challenges in relation to replicability and so that they can adopt appropriate methodological and reporting practices to ensure that a merited degree of trust can be placed in published results. This ultimately affect the opportunity of larger seminal publications being initiated around pedagogical approaches being investigated by the community. The promotion of collaboration can only be beneficial to the long-term goals of the CDIO initiative and community.

5. Conclusion

In conclusion, this systematic review posits three main areas for future development within the CDIO initiative. These have the potential to provide a clear purpose in the coming decade and further support practitioners; 1) Enhancing evidence-based practice 2) Support of blended learning research and 3) Further development of collaboration & replication efforts.

5.1. Enhancing evidence-based practice

While CDIO implementation reports remain the dominant form of publication, overall there has been a decrease in total outputs in the preceding decade. Kamp (2021) suggest that this is due to a lack of relevance once new members complete CDIO adoption within their programmes. The authors propose that a focus on supporting these new adopters in implementing more advanced research methods in the ongoing evaluation of the CDIO approach could encourage long-term engagement, enhance international collaborations and make substantial contributions to the evidence base.

5.2. Support of blended learning research

The second recommendation by the authors of this review is a continued focus on blended learning. Evidence-based practices in Blended learning is a critical pedagogical strategy that needs future research in the CDIO context as previous studies have shown that blended learning strategies can increase student performance, satisfaction, engagement, and flexibility while simultaneously decreasing student dropout rates. International trends suggest that blended will continue to grow as a delivery mechanism of engineering education. It is likely that this will be accelerated by the COVID-19 pandemic. By incentivising practitioners to conduct research around blending learning it could inform future digital/traditional classroom practices for teaching and learning.

5.3. Further development of collaboration & replication efforts

The final recommendation of the authors of this review is to promote cross-institutional collaboration and replication. Cross-institutional collaboration assists in conducting larger studies in different socio-economic settings, which leads to stronger research findings. Collaboration is common in the CDIO; however, it is prominently within a limited geographical location. The excellent practice within these institutions has the potential to act as a model for developing researchers and groups. The CDIO also has the unique opportunity to reach a large group of engineering education practitioners using its well-established network. The introduction of annual or biannual workshops, focused on engineering educational research, could provide community members with the opportunity to present potential research ideas to facilitate cross-institutional collaboration and replication.

It is important to note that the CDIO initiative is one of the most successful efforts of its kind. It has influenced the delivery of engineering education programs around the globe. It has become a bastion of innovative practice and a source of exceptional support for its members. The suggestions outlined here should be considered in light of these strengths. By aligning with emerging needs, the CDIO initiative will continue to influence the development of the next generation of engineering professionals globally. At a time where we increasingly turn to these professionals to combat existential threats, it is more important than ever that the initiative remains vibrant and at the core of engineering education development.

Open scholorship

All raw data and visual representations of raw data used within the study will be available on the Open Science Framework (OSF) website.

Registration

Open Science Framework (OSF) Preregistration. 2 September 2021 [https://osf.io/k3nzq/?view_only=c4292e44d248430798a3c7694d340fb3].

Disclosure statement

Competing Interest: Both the first and second authors are associated with the University of Limerick. The University of Limerick is a member of the CDIO initiative. The second author is an active member and proponent of the CDIO initiative.

Additional information

Funding

This work was supported by the Irish Research Council under Grant GOIPG/2021/352 and European Commission under Grant 2020-1-IE02-KA226-HE-000765

Notes on contributors

Sean O’Connor

Sean O’Connor presently is a PhD student in the School of Education at the University of Limerick. His current research is on examining team-based learning in blended and online environments. The research effort is being funded by the Irish Research Council (IRC) under the 2021 Government of Ireland Postgraduate Scholarship award.

Jason Power

Jason Power currently leads an EU funded study that aims to enhance evidence-based practice within third-level STEM learning environments. Within this, and related nationally funded projects, heis leading an international team in the creation of professional development programs, synthesised evidence bases and associated supporting resources. His previous research has focused on non-cognitive factors and their relationship to performance within STEM learning environments.

Nicolaas Blom

Nicolaas Blom is a lecturer in the School of Education at the University of Limerick where he lectures in design and communication graphics and engineering education pedagogy modules. He completed his PhD at the University of Pretoria in 2019, which explored the design cognition of students involved in integrated STEM tasks. Nicolaas’ research interests include learning and teaching in integrated STEM environments and exploring the nature of organisational relationships in post-primary schools.

Notes

1. Please note that 2020 was the last CDIO conference proceedings included in the study and that it is likely that the COVID-19 pandemic affected data gathered from this year. However, this is taken into consideration within the review findings, discussion and conclusion.