What keeps FLAME lit? Comparing two modes of implementation of a physical education-based intervention to improve motor competence among Irish adolescents

ABSTRACT

Background

Adequate levels of motor competence (MC) have been associated with multiple health outcomes. Despite the preponderance of effective MC interventions, globally, levels of MC in children and adolescents are low. There is a gap in understanding what leads to effective implementation of MC interventions into routine practice, to benefit the wider population.

Purpose

This study aims to compare implementation outcomes of two versions of Project FLAME: one group of teachers implementing Project FLAME as per the original efficacy trial (‘Original FLAME’), a second group of teachers implementing Project FLAME incorporating three additional implementation strategies (‘Modified FLAME’).

Methods

A mixed method, two-group pre-and-post design, lasting six weeks during the period of September to November 2021. Three implementation evaluation outcomes were assessed: (i) PE teacher’s self-efficacy in delivering Project FLAME; (ii) Fidelity and adaptation to the project protocol; and (iii) Teachers’ and students’ responsiveness to the project. Data were collected at student and teacher levels using online survey and interviews. Nine PE teachers and their classes from eight schools consented to participate. Descriptives were reported for quantitative online survey data, and qualitative data were analysed thematically.

Results

The final analytical sample included data from 9 teachers and 127 students pre- and post-study. Irrespective of implementation group, teachers with low levels of self-efficacy at the baseline improved after the six-week intervention. Teachers’ fidelity to the use of pedagogical external cues and error identification were high in both groups, with more adaptations made in the Modified FLAME group. Students’ satisfaction towards the intervention was high in both groups, with the use of pedagogical external cues reported as highly preferable.

Conclusion

Findings provide evidence on the ‘non-negotiable’ features of Project FLAME that have the potential to be implemented for a longer-term in Physical Education settings (e.g. external teaching cues). The documented implementation of Project FLAME provide knowledge on what adaptations may be needed to translate an effective MC intervention into real-world practice. The study reaffirms that documenting the implementation (especially fidelity and adaptation) of MC interventions is beneficial.

KEYWORDS:

• Motor skills
• fundamental movement skills
• implementation science
• children
• adolescents
• translation

Introduction

Motor competence (MC) refers to children’s and adolescents’ proficient performance in a broad range of motor skills (Robinson et al. 2015; Utesch and Bardid 2019), including object control (e.g. throwing and catching), locomotor (e.g. running and jumping), and stability skills (e.g. balancing and coordination) (Gallahue, Ozmun, and Goodway 2012; Rudd et al. 2016). Having adequate levels of MC is associated with multiple health outcomes, including improved physical fitness, perceived competence, and healthy weight status (Barnett et al. 2022). Globally, children and adolescents do not reach standards of MC that are expected for their age (Bolger et al. 2020). This is also the case for Irish children and adolescents (Belton et al. 2014; da Silva et al. 2022; O’Brien et al. 2018), where even in adolescence less than 50% (aged 12–16 years) demonstrate proficient levels of motor skills (Philpott et al. 2020).

Given the low levels of MC globally, interventions to improve MC have become a research priority (da Silva et al. 2022; Lopes et al. 2021). A diverse range of strategies that are developmentally and instructionally appropriate have been used in interventions (Jiménez-Díaz, Chaves-Castro, and Salazar 2019), with empirical evidence showing that the provision of structured activities (da Silva et al. 2022), quality instruction (Invernizzi et al. 2019), and feedback (Meester et al. 2022)positively influence individuals’ MC development. As such, implementation of Physical Education (PE) curricula, instruction, and pedagogy can be modified or enhanced to optimise students’ learning and practice of motor skills (Lawson 2018). PE-based MC interventions are often accompanied by teacher training and intervention delivery manuals (e.g. lesson plans, teaching styles) to meet the intervention aims (Lander et al. 2017a; Tompsett et al. 2017). Naturally, teachers need to make ad-hoc adaptations or modifications in their practice, which may or may not be aligned with a predefined intervention protocol (i.e. intervention fidelity). Such adaptations may be reflective of effective teaching, for example, differentiating a lesson by adjusting content to suit the learning needs of individual learners (Tomlinson 2014). The impact of such adjustments and documentations are under-reported in MC intervention evaluations(Ma et al. 2021b), yet this information is important to identify educational practices and policies relating to MC development (da Silva et al. 2022; Lopes et al. 2021).

Whilst there is an increasing interest in developing and testing the efficacy of MC interventions(Eddy et al. 2019; Engel et al. 2018; Hulteen et al. 2023; Jiménez-Díaz, Chaves-Castro, and Salazar 2019; Logan et al. 2012; Morgan et al. 2013), implementation evaluation is also important to elucidate the processes, factors, and strategies that impact effective and sustained use of MC interventions in real-world settings (Eccles and Mittman 2006). In a recent expert statement commissioned by researchers in the UK and Ireland (Duncan et al. 2022), it was stated that measuring and reporting on the implementation of MC interventions is urgently needed to translate effective interventions into routine practice, such as teaching (da Silva et al. 2022).

Project FLAME (Fundamental and Functional Literacy for Activity and Movement Efficiency) is a six-week intervention that has proved efficacious in improving MC of Irish adolescents (aged 12–16 years old) in post-primary school settings (Lester 2020; Philpott et al. 2021). However, in the Project FLAME efficacy trials, there was no assessment of if and how the intervention could be implemented over time and in different school contexts (Lester 2020). Given that the intention of Project FLAME is for it to be rolled out in school settings in Cork, Ireland, understanding real-world implementation of the intervention is necessary (Ma et al. 2021a).

Following recommendations in the PRACTIS guide (PRACTical planning for Implementation and Scale-up) (Koorts et al. 2018), stakeholders of Project FLAME (i.e. researchers, PE teachers, and local health promotion officers) were invited to engage in a group consultation workshop (i.e. the Collective Intelligence workshop, which has been described in detail and published elsewhere; Ma et al. 2021a). In this workshop, a set of implementation barriers were identified, and strategies were developed to inform modifications to the original Project FLAME intervention that aimed to address the implementation barriers (herein called ‘Modified FLAME’). These modifications include an online knowledge hub (i.e. website) that hosts all Project FLAME resources, encouragement for teachers to adapt the delivery, and build a community of practice.

The current study aims to describe and compare the two modes of implementation of Project FLAME: with one group of teachers implementing Project FLAME as per the original efficacy trial (‘Original FLAME’) (Lester 2020; Philpott et al. 2021), and the other group of teachers implementing Project FLAME incorporating the additional implementation strategies (‘Modified FLAME’). Specific objectives are to measure and compare implementation outcomes (later described in ‘data collection’ section), including PE teacher’s self-efficacy in delivering the intervention, fidelity and adaptation to the intervention protocol, teachers’, and students’ responsiveness to the intervention across the two versions of Project FLAME.

Methods

Study design and context

This study was a non-hybrid pilot implementation study which has a sole focus on implementation outcomes (Pearson et al. 2020 ), based on the established effectiveness of Project FLAME (Lester 2020; Philpott et al. 2021). The process we followed in this regard follows recommended guidance on conducting pilot studies for implementation trials (Person et al. 2020). The study used a mixed method, quasi-experimental design, lasting six weeks during the period of September to November 2021. This design was chosen as a flexible approach to compare the implementation process in real-world practice (Geldsetzer and Fawzi 2017). Ethics approvals for conducting this trial were obtained from Ethics Committees of Coventry University (P116006), Deakin University (HEAG-H 52_2021), and the Social Research Ethics Committee in University College Cork (Log: 2021-010). Three evaluation outcomes were included based on the evaluation priorities identified in the group consultation with Project FLAME stakeholders. These outcomes included: (i) PE teacher’s self-efficacy in delivering Project FLAME; (ii) PE teacher’s fidelity and adaptation to the project content and delivery; and (iii) PE teachers’ and students’ responsiveness to the project. Data were collected at student and teacher levels via online surveys, teacher’s log, and interviews. Questions in the survey and log did not differ by implementation groups.

Recruitment of participants

Irish secondary schools and PE teachers that had never been exposed to Project FLAME in previous efficacy trials were eligible to take part. Recruitment of schools and PE teachers was overseen by two members of the research team who were qualified specialist PE teachers, as recognised by the Teaching Council of Ireland. PE teachers from schools deemed eligible were contacted by the research team via phone, to invite their school to participate. Following initial approval granted by school principals and PE teachers, consent forms and information sheets were distributed to class groups. Informed parental consent and child assent were required before any adolescent could participate in the student data collection. All participating PE teachers were given a choice of two timeslots within the same week to attend the pre-trial training (i.e. Training One and Training Two; details of the training are reported in Lester 2020). Teachers who attended Training One were allocated to the Modified FLAME group, and teachers who attended Training Two were allocated to the Original FLAME group.

Description of Project FLAME and two modes of implementation

Project FLAME is a multi-component school-based intervention aiming to improve Irish adolescents’ MC levels (Lester 2020; Philpott et al. 2021). The Project FLAME intervention includes four critical components: (1) Movement activities and resources for students, (2) PE, (3) Digital resources, and (4) Classroom. A detailed description of intervention activities in the Original FLAME and Modified FLAME groups are provided in Table 1. Consisted with previous Project FLAME trials, Project FLAME activities were recommended to be integrated into a minimum 20-minute section within one 80-minute PE lesson weekly over six weeks (Philpott et al. 2021). Both groups of teachers received a total of two hours of training via online meetings from the research team in delivering Project FLAME, which involved a comprehensive overview on how to utilise the Project FLAME resources in lesson planning and in-class. This training protocol was consistent with what was reported in Philpott et al. (2021). Teachers from the Modified FLAME group received additional trainings on accessing and using the project website.

Table 1. Intervention description of two modes of implementation of Project FLAME∼.

	Intervention description
Intervention components	Original FLAME	Modified FLAME
Movement activities and resources for students	- Developmentally appropriate movement-based activities in an authentic learning environment in school (available in a manual). - Clear and specific process-oriented criteria (available in a manual).	- Developmentally appropriate movement-based activities in an authentic learning environment in school (available in a manual). - Clear and specific process-oriented criteria (available in a manual). - - Access to the project website (project-flame.com); Students are encouraged to leave questions and comments on the website regarding skill learning.
PE	- Appropriate PE teacher training (available online). - Comprehensive subject and pedagogy content (available in a manual). - Teaching instructional practices, and teaching quality (available in a manual). - - External cues (available in a manual).	- Appropriate PE teacher training (available online). - Comprehensive subject and pedagogy content (available both online and in a manual). - Teaching instructional practices, and teaching quality (available both online and in a manual). - External cues (available both online and in a manual). - Access to the project website that includes all project resources, project aims, and benefits. - Teachers are encouraged to adapt the project delivery, and document and share local knowledge among the group. - Teachers are consulted regarding the project delivery to increase student engagement.
Digital resources	Digital resources (QR codes/YouTube videos).	Digital resources (QR codes/YouTube videos/Website).
Classroom	This component was not delivered in the current trial, due to various school circumstances post COVID-19 lockdown.

*The additional intervention activities implemented in the Modified FLAME group are bolded. ∼Contents are adapted from Philpott et al. 2021. Details on the intervention descriptions are reported in Philpott et al. 2021 and Lester 2020.

Data collection

Data were collected during the 2021/2022 Irish academic year (September 2021–November 2021). The measures taken at different time points are presented in Table 2.

Table 2. Schedule of evaluation data collection methods.

Study Measure	Week 0	Week 1	Week 2	Week 3	Week 4	Week 5	Week 6	Week 8–10
PE teacher survey	x	-	-	-	-	-	-	x
Student survey	-	-	-	-	-	-	-	x
Teacher log*	-	x	x	x	x	x	x	-
Semi-structured Interviews (PE teachers)	-	-	-	-	-	-	-	x

*Teachers log were collected weekly.

PE teacher’s self-efficacy

Self-efficacy was assessed by asking PE teachers to complete an online survey pre- and post-trial. The self-efficacy scale used for the assessment is a modified version of The Primary School Physical Education questionnaire (Morgan and Hansen 2008), which was used in a previous MC intervention trial to measure perceived teacher competence on assessing and teaching motor skills (Lander et al. 2017b). Teachers were invited to state their level of agreement within 14 self-efficacy statements, on a 0 (cannot do at all) to 100 (highly certain can do). The self-efficacy scale used in the current study is included in Supplementary File 1. Example statements include ‘Planning effective lessons to achieve MC related student learning outcomes’, and ‘Apply appropriate content knowledge in my teaching (e.g. motor development theory) to achieve learning outcomes’. To obtain the face validity of the scale prior to data collection, the questionnaire was piloted with a group of five non-participating PE teachers working in Irish post-primary settings, with suggested minor modifications to phrasing made thereafter to enhance clarity and relevance.

Fidelity and adaptations

Throughout the six-week intervention, teachers were asked to complete a weekly log via a web link to record implementation fidelity and intervention adaptations. To determine the extent to which Project FLAME was implemented as intended, teachers were asked to report against the core principles of Project FLAME (see Table 3). The assessment and analysis procedure followed the recommendations for fidelity measurement development (Schoenwald et al. 2011). Relevant components for monitoring (i.e. core principles of Project FLAME, referred to as fidelity component hereafter) were firstly identified, followed by the development of questionnaire items on the weekly log. Table 3 describes all fidelity components and how fidelity ratings were defined. A summary score for the rating was created for each fidelity component. Questions that elicited information on fidelity components were included in the student questionnaire to validate teacher’s self-report responses (Creswell and Clark 2017).

Table 3. Fidelity components of Project FLAME and their definitions and ratings.

Fidelity Component	Item question on a weekly log	Fidelity rating defined according to response options
Motor skill focus	Was the lesson focused on MC for this week? (e.g. focused on throw in Week 1)	0 – Fidelity inconsistent* adaptation made 1 – Fidelity consistent adaptations made 2 – Implemented with no adaptations
Integration	Was MC developed further as part of the curricular strand?	As above
Teacher demonstration	Were students visually shown the correct performance of the movement skills by you, to the best of your understanding?	As above
Digital demonstration	Were students visually shown the correct performance of the movement skills by digital resources (e.g. videos accessed by QR codes)?	As above
Criteria	Did you share and teach the performance criteria/features of quality, as relevant to the selected movement skills (e.g. when throwing, wind-up is initiated with downward movement of hand/arm)?	As above
External Cue	Did you share and teach the movement through the use of external movement-based cues (e.g. throwing like the NIKE logo)?	As above
Error identification	Did you identify potential errors among students when they perform the movement?	As above
Digital provision	Were students provided with the digital resources (e.g. QR codes) to practice this week’s movement skill in their own time?	As above
Duration of Project FLAME activities in one PE lesson	On average, I included Project FLAME for () in one PE lesson	1–0–5 min 2–5–10 min 3–10–15 min 4–15–20 min 5–20+ minutes 6 – a whole lesson∼

*Fidelity-consistent adaptations refer to those that preserve core elements of an intervention that are needed for it to be effective in improving student outcomes, identification of this was made in consultation with input from the project developers (WOB and DL). ∼a standard PE lesson in Irish post-primary education is 40mins

Participant responsiveness (teacher)

Information on teacher’s responsiveness was elicited via questionnaires and interviews. Teachers’ perception on acceptability (i.e. the perception that the intervention is satisfactory), feasibility (i.e. the extent to which an intervention can be successfully used), and appropriateness (i.e. the perceived fit of the intervention) of the programme were assessed at the end of Week 6 using the Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM), and Feasibility of Intervention Measures (FIM) (Weiner et al. 2017; definitions adapted from Proctor et al. 2011). Three measures used in the current study are included in Supplementary File 1. A sample AIM item is ‘Project FLAME is appealing to me’. A sample IAM item is ‘Project FLAME seems applicable’. A sample FIM item is ‘Project FLAME seems easy to use’. Psychometric properties of three scales were previously assessed with a group of international implementation scientists and health practitioners (n = 296) (Weiner et al. 2017). Construct validity exhibited factor loadings that ranged between 0.79 and 0.94, an acceptable model fit, i.e. CFI = 0.98, RMSEA = 0.08, and three-week test-retest reliability coefficients ranged from 0.73 to 0.88.

Interviews were conducted to elicit information on the delivery and adaptation of Project FLAME, self-efficacy, perceived feasibility of implementation strategies and intervention components, and perceived project impact on student responsiveness. Prior to each interview, teachers’ responses in the survey were studied to develop prompts used in the interviews, to connect the analysis of quantitative data with the subsequent qualitative data collection (Creswell and Clark 2017). The interviewer had no prior involvement in the outcome evaluation of Project FLAME, therefore was able to conduct the interview with objectivity that reduces potential biases arising from analysing outcome data (Moore et al. 2015).

Participant responsiveness (student)

Regarding student responsiveness, students were asked to complete an online survey at the end of Week 6. Questions included if they thought Project FLAME was fun, useful, and introduced them to new activities, as well as the frequency of their access to project resources and skill practice. Students were also asked to provide free text responses about the highlight of a Project FLAME lesson. Example questionnaire items include: ‘How often were you given feedback on your skill performance?’, ‘In a sentence or two, describe a Project FLAME PE lesson that you liked the most.’

Data analysis

Descriptive statistics were reported to summarise the characteristics of continuous variables. Given the small sample size of teachers (n = 9), self-efficacy data were presented in full for each teacher using a univariate scatterplot to show the data distribution (Weissgerber et al. 2015). For the analyses of the three scales (i.e. AIM, IAM, FIM), responses for each scale item corresponding to the indicator (i.e. appropriateness, acceptability, and feasibility) were presented for each teacher. Scores for each fidelity component were also determined for each weekly log received for each school. An average of the score was calculated for each school.

For qualitative data analysis, interviews were audio recorded and transcribed verbatim. Along with open-ended survey responses, qualitative data were compiled into a word document for deductive coding. Coding processes were dependent on the implementation outcome of the data reported on. To describe the extent, type, and reasons for adaptations of the project, adaptations reported from the weekly logs were collated and coded using the Framework for Reporting Adaptations and Modifications-Enhanced (Stirman, Baumann, and Miller 2019). The purpose of the coding was to systematically examine the common adaptations made during the trial using a taxonomy of classifying adaptations that is consistent with the wider implementation research literature. The number of adaptations was reported for each adaptation category. Students’ text responses were analysed thematically. The analysis outcome was presented via descriptions of composite key emergent themes.

Results

Sample characteristics

Overall, 12 schools and 13 PE teachers were invited to take part in the study, of which 8 schools (n = 9 teachers) provided consent to participate. Four teachers did not provide consent due to work commitments and personal reasons. The final analytical sample included data collected from 127 students (consent rate: 72.5%) and 9 teachers’ pre- and post-trial. Among these teachers, 2 teachers were from the same school delivering the intervention to different classes. As mentioned, teachers who were available to attend Training One were allocated to the Modified FLAME group (Participant IDs: M-1, M-2, M-5, M-6, M-7), and those who were available for Training Two were allocated to the Original FLAME group (Participant IDs: O-1, O-2, O-3, O-4). Table 4 provides a summary of the characteristics of participating teachers and data availability of each participating teacher/school. Of the 127 students who responded to the survey, 74 were from schools in the Modified FLAME group (65 boys, 4 girls, 5 prefer not to say) and 51 were from the Original FLAME group (26 boys, 23 girls, 2 prefer not to say).

Table 4. A summary of data availability and characteristics by participating schools/teachers.

	School/Teacher ID	Gender	Years of teaching experience	Logbook Returned/Expected	Teacher survey (pre-trial)	Teacher survey (post-trial)	Student survey	Teacher Interview
Modified FLAME	M-1	Female	>10	4/6	✓	✓	✓	x
	M-2	Male	1–3	4/6	✓	✓	x	✓
	M-5	Male	1–3	3/6	✓	✓	✓	✓
	M-6	Male	1–3	5/6	✓	✓	✓	✓
	M-7	Male	1–3	4/6	✓	✓	✓	✓
Original FLAME	O-1	Female	1–3	4/6	✓	✓	x	x
	O-2	Female	4–10	4/6	✓	✓	✓	x
	O-3	Female	1–3	6/6	✓	✓	✓	x
	O-4	Male	4–10	4/6	✓	✓	✓	x

*As requested by the principals of M-2 and O-1, students from these schools were not asked to take part in the student survey.

Comparing implementation outcomes

To address the objective of the current study, that is, to describe and compare two modes of implementation of Project FLAME, results are reported by implementation outcomes, respectively.

PE teacher’s self-efficacy

For teachers in the Modified FLAME group, self-efficacy scores were similar pre- and post-trial (pre-trial median = 86.07, Interquartile Range (IQR) = 66.43–90.00; post-trial median = 81.07, IQR = 80.71–86.57), with the IQR pre-trial wider than that of post-trial. All teachers’ self-efficacy in the Original FLAME group increased from pre-trial (median = 75.04, IQR = 74.32–75.75) to post-trial (median = 88.89, IQR = 88.39–91.02). All teachers, irrespective of group, that had relatively low levels of self-efficacy at baseline, improved after the six-week delivery of Project FLAME.

Fidelity

High agreement between teachers’ and students’ responses on questions related to fidelity components was found, which established trustworthiness of teacher’s self-report responses. Fidelity to the use of external cues and error identification were high across all schools. The inconsistent use of digital demonstration was due to issues related to technology and facilities, which supports the data reported in the weekly logs and interviews. Table 5 presents fidelity scores for schools implementing Project FLAME.

Table 5. Scores for the fidelity component of Project FLAME, by each school.

Fidelity component (maximum score = 2)	Modified FLAME					Original FLAME
	M-1	M-2	M-5	M-6	M-7	O-1	O-2	O-3	O-4
Motor skill Focus	0.75	2	1.3	2	1.5	1.75	1.25	2	2
Integration	1.5	2	1.3	2	1.5	1.75	2	2	2
Teacher demonstration	2	1.25	2	2	1.5	2	2	2	2
Digital demonstration	0.5	0	0.67	2	0	0	0	0	0.5
Criteria	2	0	2	2	1.5	2	2	2	2
External cues	2	2	2	2	2	2	2	1.67	2
Error identification	2	2	2	2	1.5	1.75	2	2	2
Digital provision	0	0	0	1.2	2	0	1	0	0.5
Total (maximum score = 16)	10.7 (67.2%)	9.2 (57.8%)	11.3 (70.8%)	15.2 (95%)	11.5 (71.9%)	11.2 (70.3%)	12.2 (76.6%)	11.6 (72.9%)	13.0 (81.2%)

Adaptation

A total of 71 adaptations were made during the six-week trial. As shown in Table 6, 43 adaptations were reported by teachers in the Modified FLAME group and 28 adaptations by teachers in the Original FLAME group.

Table 6. Adaptations during the six-week Project FLAME trial, coded using the Framework for Reporting Adaptations and Modifications-Enhanced (FRAME) framework.

Adaptation categories	Code (number of adaptations)*	Example adaptations reported in the teacher log (teacher ID)
Who participated in the decision to adapt?	Teacher (70)	I used the lessons as a warmup activity. It is really useful to identify weakness in 1st year students. (O-4)
	Student (1)	I adapted it by using cues that the students came up with themselves. (O-2)
For whom is the adaptation made?	The whole class (67)	I created an obstacle course at the end which got the students using all 6 of the movements learned at different stages in teams as a race against the other teams. (O-3)
	Individuals (4)	I have one student on wheelchair so her control over the ball was much less than others. We did not do one handed dribble with her instead I focused on catching the ball after each bounce with 2 hands. (M-1)
Were adaptations proactive or reactive?	Proactive (50)	I changed it to a more simplified version of dodgeball (Using tennis balls to practice skill) then their engagement was much better. So it was great to be given the initial game and then being able to adapt it to the needs of my group. (O-4)
	Reactive (20)	Didn’t use QR demo due to lack of facilities. (O-2)
What was the goal	Improve feasibility (47)	Relay/tag teams were used to compensate for shortage of basketballs. (O-1)
	Increase engagement and satisfaction (24)	I made the activities more game like. I find the students engage better and they find it more enjoyable. (M-2)
	Improve outcomes (15)	Most students were inclined to not use their legs to get power in the throw at all – forcing them to stand side on enforced this. (O-2)
	Improve fit with students (13)	For ‘catching’ the person activity, students were matched according to ability. This ensured that every student had the opportunity to succeed. (M-1)
What is adapted?	Contextual (format) (49)	Skip focus was included in the warm—up and linked to the catching/fielding skill in GAA. (M-5)
	Content (24)	I didn’t use the cone game because it would have taken too long to set up and I thought the students may have found it boring. (M-2)
What is the nature of the adaptation?	Substituting/skipping (37)	Didn’t use QR demo during the class due to lack of access to phones, the videos have been added to their online class to look at themselves. (O-2)
	Integrating parts of the intervention into another framework (14)	I included this as part of the students’ gymnastics routines as it hit the area of balance and stability by encouraging a wide base when landing. (M-5)
	Tailoring/tweaking/refining (13)	I altered it by asking the students to perform the skip differently, e.g. low, high, sideways, backwards, quietly, loudly. (M-2)
	Adding elements (11)	Showing students examples of the movements of athletes in sport helps with application and interest. (M-5)
Reasons for adaptation?	Available time and resources (29)	Impractical to set up projector/iPads for a video I could demonstrate efficiently myself. (O-1)
	Student motivation and readiness (20)	Students seemed to enjoy experimenting with what position enabled them to throw further. We tried throwing the ‘wrong way’ e.g. Wrong foot forward/standing face on/etc. Students could discover themselves what their technique should be. (O-1)
	Provider competence and experience (18)	Didn’t use QR demon because I can demonstrate efficiently myself. (O-1)
	Student physical capacity (4)	Adapted games and personal ques to students due to physical disabilities e.g. no focus on stance for wheelchair user, softer balls. (M-1)
	Existing curriculum (11)	We were carrying out other small sided games so we used the dribbling activities as a warm up. As we were doing Gaelic Football this week it tied in nicely at the beginning of the lesson. It was perfect to use as a warm up activity. (O-2)

Participant’s responsiveness (teacher)

Teachers’ perceived acceptability was consistently high in the Original FLAME group. The Modified Project FLAME was more appealing compared to the Original Project FLAME, with more teachers in the Modified FLAME group indicating the highest level of agreement on ‘I welcome Project FLAME’, ‘I like Project FLAME’, and ‘Project FLAME is appealing to me’. During interviews, all teachers mentioned the benefits of developing MC when being asked about the appeal of Project FLAME, particularly for students with lower MC levels:

I suppose it’s the focus on FMS (that appealed to me) … It’s something different as well for the guys who are in the class, I think particularly the weaker guys benefited from it. (Teacher M-5)

Teachers from School M-2 and School M-6 reported lower levels of agreement on ‘Project FLAME meets my approval’, compared to other teachers in the Modified FLAME group. This may be explained by the varied perceptions on the project contents and structures:

It (Project FLAME) was really like kind of too linear. It was kind of isolating the skills away from the games, whereas when I was doing, for example, dribbling and stuff, I was trying to kind of doing through the games that dribbling being a key skill in and to get them to make decisions on when to dribble and how to dribble rather than dribbling for the sake of it. (Teacher M-2)

Teacher M-7 commented on the usefulness of resources provided in Project FLAME, for both teachers and students:

It’s nice to have the kind of bank of resource there as well just to lean back on, as kind of a tool for the students as well, they can, if it didn’t go well in class they had, they know where to go to get help or to revise themselves. (Teacher M-7)

Teachers from both groups tended to agree that ‘Project FLAME seems applicable’, which was confirmed by the high fidelity to the project components reported previously. When teachers’ responsiveness was discussed during the interviews, there were differences in opinions about the perceived fit of the programme. For instance, Teacher M-7 found Project FLAME relevant and applicable in developing students’ skills so that they can benefit from PE in the long run:

I was aware that if the students were able to do these skills confidently like that, they would be able to do a lot of other things … And I will hopefully have them (take part in Project FLAME) for the next two years. It probably makes my life a bit easier as well. (Teacher M-7)

Similarly, Teacher M-2 commented on the applicability of Project FLAME (performance criteria) in identifying students’ skill proficiency levels:

It was nice to identify potential areas where the students where they might have a bit of difficulty, or where they need improvement. (Teacher M-2)

By contrast, teachers agreed less on ‘Project FLAME seems fitting’, mainly for its suitability to the existing curricular strands that need to be covered in PE as well as the school-specific PE planning. For example, Teacher M-5 reflected on how Project FLAME could fit into the school curriculum:

I was actually kinda happy with some of them. Some of the ways that it slotted in. There were other aspects that (did) not particularly fit well, that’s just because of how we operate the schedule of modules. That might be something worth looking at, how it will fit in with the way that (school) PE department might run. (Teacher M-5)

Another contributing factor to teachers’ perspectives of the perceived fit of the programme was pedagogical preference. For instance, Teacher M-2 reported lower levels of agreement on the appropriateness items, due to the misalignment between Project FLAME and their own teaching philosophy and practice:

I suppose over the last few years, I’ve kind of moved towards more teaching the skills through the games. I’ve been doing a lot of reading like ecological dynamics and how you can apply their skills through a game. (Teacher M-2)

The tight and busy work schedules of teachers also created barriers to implementing other activities, such as leaving comments on the website. All four teachers interviewed have teaching responsibilities in addition to PE, with the planning and teaching hours for PE and other subjects usually split in half. PE teachers often needed to go from one class to another, if it were PE they would need to prepare and bring equipment and organise students to get to the site which leaves very little time. As Teacher M-5 commented,

I don’t think the online thing really works too well with PE teachers particularly, because we are very active and practical and like to see things being done and how they work.

Participant’s responsiveness (student)

There were 107 out of 127 students who wrote comments regarding their participation highlights. Across both implementation groups, a common theme was students’ reported enjoyment in skill-specific tasks, games, and external cues. In total, 94 students mentioned specific motor skills in their responses and described what they liked about the motor skill-focussed activities (e.g. ‘I liked the Christiano Ronaldo jump.’ (School O-4, M)). Horizontal and vertical jumps were the most preferred lesson by students because they involved a diverse range of games and competitions (e.g. ‘I enjoyed horizontal jump because we did a hoola hoop game which I enjoyed.’ (School O-3, F)). Students also found Project FLAME useful in terms of learning a new skill (e.g. ‘I enjoyed new methods of kicking’ (School O-3, F)) and improving their skill proficiency levels (e.g. ‘I got to learn how to skip properly and fix what I was doing wrong.’ (School O-3, F)). Many students highlighted the transferability of skills to their preferred sports (e.g.‘I liked kicking and throwing because they can be carried out in different activities and sports.’ (School O-3, F)). Irish national sports Gaelic football and hurling were the most mentioned in the responses.

Discussion

This study aimed to describe and compare two modes of implementation of Project FLAME: Project FLAME as per the original efficacy trial (Original FLAME), and Project FLAME incorporating additional implementation strategies (Modified FLAME). The present study addresses a noted research gap in the literature describing MC interventions, namely a lack of consideration of intervention implementation metrics and intervention fidelity (da Silva et al. 2022). A key finding is that integration of Project FLAME into routine teaching practice requires adaptations to meet contextual and student needs, and these adaptations represent key mechanisms and contexts to consider for the refinement of Project FLAME for future translation.

Self-efficacy is a key influencing factor of quality PE delivery and teachers with high self-efficacy are more likely to overcome challenges faced in their teaching (Bandura 1993; Hutzler et al. 2019). In the current study, all teachers with low level of self-efficacy at baseline improved after 6-week Project FLAME interventions, irrespective of the group. The elements of Project FLAME, including the resource pack with pedagogical cues, performance criteria, and games, were accessed in the teaching preparation to get inspiration and increase confidence in delivering Project FLAME activities.

In the current evaluation, in parallel to the measurement of fidelity, adaptations were also documented to expand on the context of fidelity (Bopp, Saunders, and Lattimore 2013). This model of assessment acknowledges that fidelity to the intervention and adaptation co-occur (Toomey et al. 2019), and in the context of this evaluation, it assisted in recognising which fidelity components are more context-sensitive than others. Combined with the adaptations notes that captured the multi-facetedness of fidelity, the findings generated a series of real-world implementation cases of under what circumstances, adaptations were made and what impacts they had. For instance, utilisation of pedagogical cues and provision of feedback seemed to be implemented across all teachers and schools, despite the individual and contextual variations. This may suggest that these two fidelity components are ‘non-negotiable’ to the project delivery, it may also indicate that these are essential elements of quality PE delivery recognised by all teachers involved in the intervention. The ‘non-negotiable’ features provide important implications for researchers and practitioners when scoping and planning a version of Project FLAME that can be sustained in teacher practice. These features may be tested further in a hybrid-effectiveness trial to investigate their impact on student skill outcomes. Conversely, fidelity to the digital aspect of the intervention (e.g. using digital demonstration) was largely contingent on the technology and equipment available to the teachers or teacher’s perceived need of the digital demonstration, demonstrated by its poor fidelity adherence. This revealed some fundamental barriers to adhering to this fidelity component, which posed questions on the necessity of including digital demonstration in the project, and if the fidelity was to be achieved, more support would be needed (e.g. providing digital equipment in addition to the project resource pack).

As a result of the implementation strategy that encouraged adaptation, teachers in the Modified FLAME group may have felt more flexible in the delivery and reported more adaptations. Meanwhile, adaptations also occurred in the Original FLAME group without being prompted by the implementation strategy. This suggests that adaptations are inevitable, especially in a dynamic PE environment, whereby teachers need to make proactive decisions to ensure learning objectives are achieved by the students (O’Brien et al. 2023). Putting in the context of effective teaching, differentiated teaching requires diverse content, delivery, assessment, and experiences to suit individual student needs. Indeed, one of the common goals for adaptations reported by the teachers was to improve the feasibility and fit with the students. Teachers in the Modified FLAME group reported more adaptations to suit student needs and different skill levels. The reported adaptations showcased a diverse range of teaching models and styles, and potentially elucidated some pedagogical mechanisms on how to teach to a diversity of skill levels and interests. For instance, games and practice included in the project resources (that has basic and advanced levels of difficulty) were often set up by the teachers as different stations and let students decide which is the most appropriate for their abilities and motivations. Some teachers would also print out the project resources as skill cards, so that students with lower proficiency MC levels can practice and identify errors and improvements among themselves. By contrast, in the stronger group, students tend to engage in competitions, teachers would then give little feedback in this instance and encourage students to discover alternatives to improve the performance (e.g. change the task and ask students to find a way to jump higher). Capturing these adaptations through documentation is indeed a way to explore effective teaching strategies and under which circumstances these strategies work.

While there are no specific teaching models or styles suggested in the Project FLAME resources, teachers creatively availed of existing resources (cues, games, criteria) and integrated them into different approaches catering to the different ways in which students learn (MacPhail et al. 2018). Even though some adaptations skipped or substituted the fidelity component (e.g. teacher demonstration), they were considered appropriate by teachers in the lesson and helped achieve the learning outcomes. This reinforces the importance of documenting fidelity and adaptation, which builds the understanding of how, when, why, and to what extent programme are adapted, what new things are added, or whether parts of a programme are omitted (Ma et al. 2021b). This understanding can subsequently be translated into future refinements of project content and resources for better usability.

Teachers in the Modified FLAME had mixed responses to the additional implementation strategies. For example, some teachers found the ‘bank of resources’ provided on the website appealing and useful. However, not all teachers were used to accessing online resources regularly or had time/equipment to do so. Additionally, online Continuing Professional Development for teachers needs to be tailored to teacher’s need and regularly updated for it to be more beneficial (Lander et al. 2020; Lonsdale et al. 2021); this is reflected in teachers’ suggestions to include video examples of effective delivery of Project FLAME. Student responses suggested that Project FLAME appealed equally to students from both groups. However, this needs to be interpreted with caution due to the imbalance of gender distribution between groups (i.e. more male respondents than female). Encouragingly, Project FLAME activities were perceived as fun and enjoyable, which may have increased students’ motivation (Deci and Ryan 2012). Students’ motivation was not only driven by the ‘fun’ aspect of the projects, but they also found Project FLAME useful in learning a new skill, improving performance, with recognition of these skills’ transferability to other contexts. Specifically, students mentioned behavioural components of skills that helped them learn and improve skills, which showed that the project elements (performance criteria, cues) likely had a direct impact on students’ engagement.

Strengths and limitations

One strength of this study was the use of mixed methods and measures from multiple data sources, to provide a comprehensive understanding of the intervention implementation. The triangulation of data from multiple sources helped to minimise the social desirability bias (Creswell 2013).

Potential selection bias is acknowledged. Schools volunteered to take part in the current trial and therefore participants may already have a greater interest in Project FLAME and consequently report a positive experience. Although teachers’ choice to attend one of the two training days and the consequent group allocation were not restricted to any criteria, this was not a strict randomisation procedure given the pragmatic nature of the evaluation. An additional limitation concerning sampling is the imbalance of the sample size of boys and girls between groups, which could have been minimised by stratification, e.g. by matching the numbers of all boys and all girls’ schools. Lastly, COVID-19 presented significant challenges for recruitment and engagement of teachers and students, for example, teachers’ fatigue to the online environment post lockdown, during which teachers have intensively engaged in online teaching and meetings. Findings need to be interpreted with caution considering the small sample size, especially of teachers (n = 9). Further, the instrument that measures teacher’s self-efficacy, although it has been applied in similar MC intervention contexts, has limited validity and reliability evidence. Findings therefore need to be interpreted with caution.

Conclusion and implications – what keeps the FLAME lit?

Findings on implementation outcomes highlighted that Project FLAME is an adaptable programme that can be taken as a whole or in part. Triangulation of implementation evaluation data allowed for the disaggregation of different components of Project FLAME and the identification of ‘non-negotiable’ features that have the potential to be implemented for a longer-term. These features include pedagogical cues and games that can be integrated into PE lesson planning. For the planning across a series of PE lessons, Project FLAME helped identify MC levels among students who just entered post-primary education and direct goals and strategies to help them improve. Most importantly, Project FLAME provided a bank of resources that inspired teachers to include a variety of activities that enable effective teaching and learning to help students continuingly improve MC.

The current study reaffirms that documenting the implementation (especially fidelity and adaptation) of MC interventions is beneficial. The documented adaptations of Project FLAME highlighted the conditions under which adaptations occur and what impact they had, thus providing knowledge on what may be needed to translate an effective MC intervention into real-world practice.

Authors’ contributions

JM led the design of the study with all authors contributing to the study design and ethics submission. HK advised on the evaluation design. WOB and DL recruited study participants. JM led the data collection, with assistance from WOB and DL. JM conducted the data analysis, with assistance from EE, NL, LB, and MD in data interpretation. JM drafted the manuscript with all authors contributing to the review and edit. All authors have critically read and approved the final manuscript. MD and LB provided senior supervision and mentorship on research activity planning and execution.

Acknowledgements

We sincerely thank all the schools, teachers and students for taking part in this study. JM was supported by a Cotutelle Doctoral Studentship of Coventry University and Deakin University

Disclosure statement

No potential conflict of interest was reported by the author(s).