Large-scale, design-based research facilitating iterative change in Irish schools – the Trinity Access approach

ABSTRACT

Trinity Access (TA) is a post-primary initiative, in the authors’ university, that offers programmes for students, teachers and schools in areas of socio-economic disadvantage. The aims of the programme include helping students develop the knowledge, networks and skills needed to make informed decisions about future educational opportunities, and supporting teachers and schools to foster collaborative and reflective learning environments, which help students develop the academic capital they need to attain their aspirations. This is achieved by delivering programmes for students and professional development for teachers, embedded in an iterative cycle of data collection/analysis and feedback. A distinguishing feature of TA is its long-term commitment to its partner schools (20 years in some cases). This sustained engagement enables the depth and breadth of longitudinal research described herein. The research follows an observational study design within an overarching design-based research methodology. Data are collected annually from consenting students, teachers and management in 20 schools, enabling the researchers to establish students’ levels of exposure/non-exposure to the intervention and to identify correlations between exposure and behaviours/aspirations. The goal of this paper is to provide an overview of the research design and methodology, illustrating its potential to deliver insights into practices that may alleviate educational inequalities.

KEYWORDS:

• Design-based research
• widening participation
• post-primary education
• cohort study

Introduction and background

The Irish education system, as with systems in most OECD countries, has become considerably more accessible in the last half-century. Secondary school completion rates have risen to 90% and the numbers progressing to higher education now typically exceed 65% (Clancy 2015). However, a number of surveys show that there are significant, and persistent, disparities in social class participation in higher education (Clancy 1982, 1988, 1995, 2001; O’Connell, Clancy, and McCoy 2004; Flannery and Cullinan 2014) with the increases in student numbers progressing to higher education appearing to have principally advantaged the professional and managerial classes. In Ireland, only 27% of the typical age of entry, low socio-economic status (SES) students progressed to higher education in 2017 (Higher Education Authority 2018).

A number of reasons have been put forward for this. These include an argument that dominant educational practices disadvantage those from low SES groups, with low teacher expectations, and ability grouping practices, playing a particularly destructive role (Devine 2011; Keane 2013; Lodge and Lynch 2002; Lynch 1998, 1999; McCoy and Smyth 2011). Keane (2013) argues that a lack of family experience of educational success and higher education participation negatively affects students’ academic self-confidence and their expectations relating to academic achievement and progression to post-secondary education. The research indicates that a range of deficits in information, pedagogy and trusted relationships at the school-level militate against many students reaching their full academic potential.

In addition, Smyth and McCoy (2009) point out that schools in areas of low SES have a higher proportion of ‘newcomer’ (immigrant) students, students with disabilities, and Traveller students than schools in more affluent areas and have a higher incidence of serious literacy and numeracy problems, emotional and behavioural problems, absenteeism, lower student motivation, problematic student–teacher relationships and less parental involvement.

In 2005, the Irish Government (Department of Education and Skills) launched an action plan for ‘Delivering Equality of Opportunity in Schools (DEIS)’. Under the scheme, a range of additional supports are provided to schools in areas of low SES provided they meet certain criteria and in 2020, out of the total 723 post-primary schools in Ireland, 197 have DEIS status.¹ One of the goals of the DEIS scheme is to increase the participation rate of students from DEIS schools in HE and while some progress has been made the figure of 27% quoted above shows that serious inequality of opportunity still remains.

The endeavours of HE institutions to widen the participation rates of low SES students are typically run through Access Offices who oversee the institution’s educational outreach programmes. The standard outreach model provides a range of educational opportunities to low SES students. These typically include visits to HE’s campus, learning supports, foundation courses to help bridge the gap between school and HE and alternative admission routes to access places in HE institutions.

The broad research areas that motivated the work presented in this paper arose from the identification of two key problem areas. Problem area 1 highlights that, in addition to various barriers including financial constraints, lack of resources and a dearth of networks and support to inform post-secondary options, there is also a perception that some pathways in education are only open to certain strata of society. The combination of actual and perceived barriers leads to educational inequality at a systemic level. Problem area 2 identifies that historically, the post-primary education system in Ireland could be viewed as inflexible and primarily focused on terminal exams. This has been identified as not being well suited to the preparation of students for full participation in a rapidly changing society, and in line with international trends, recent curriculum reforms aim to address this issue (Banks et al. 2018; Department of Education and Skills 2015; O'Sullivan et al. 2015). However, such changes are not always easy to mainstream, requiring sustained professional development for practitioners in order to ensure that they understand and value the reform (Kärkkäinen 2012). Consequently, despite the intended curriculum reflecting the values of the reform, a traditional, exam-focused approach remains in evidence in many classrooms.

The overarching goal of Trinity Access (TA) is to address these two problem areas through the delivery of a suite of programmes for students, and the provision of professional development for teachers, both of which are embedded in an iterative cycle of data collection and analysis, feedback to schools and consultation with stakeholders.

The goal of this paper is to provide an overview of the research study design and research methodology and to show the potential for this research to deliver insights into practices that may alleviate the problems outlined above.

Programme details

The university’s outreach programmes began in earnest in the 1990s and since then the Trinity Access Programmes (TAP) has developed a range of outreach activities, university ‘foundation’ courses and alternative admissions routes to Trinity College Dublin. The admission scheme piloted in Trinity – HEAR² – has been mainstreamed nationally (Trinity Access Programmes (TAP) 2010), and the pre-entry foundation course was adapted for use in one of the colleges in Oxford. A suite of supports, including scholarships, are provided for those who enter by non-traditional routes. In 2014, 48% of students from TA-linked schools progressed to higher education. Over time, this number has gradually increased and in 2020, the schools with high levels of engagement are reporting up to 74% of their students progressing to HE.

Until 2014, however, TAP outreach activities mainly adhered to a standard outreach model of providing some students from linked schools with opportunities to visit the university campus, primarily aimed at students in upper secondary school (grades 11 and 12). In 2014, thanks to a significant external grant, the model changed to focus on building the social, academic and human capital of all students within partner schools (grades 7–12) and providing professional development support for teachers to foster collaborative and reflective learning environments that will help students in that regard. The first, pilot phase of the initiative – Trinity Access 21 (TA21) – involved a three-year, quasi-experimental, intervention-style study, which ran until 2017 and followed a cohort of 1100 s year (grade 10) students from 11 treatment and 4 control schools (2 from areas of similar SES, as well as 2 with high progression rates to higher education). Results from this three-year pilot intervention indicate that the TA21 programme has a positive effect on participants’ aspirations to continue in education after completing post-primary school (Hannon 2018; Hannon et al. 2017), with evidence of increased aspirations and capabilities in the intervention group, with respect to the control groups (O'Sullivan et al. 2017).

Re-named TA in 2019, the project aims to support the development of specific forms of ‘academic capital’ under the umbrella of the Capability Approach (Sen 1992). It is underpinned by a ‘widening capability’ (Walker 2008) model of widening participation and aims to shift from a focus solely on student progression, to one that includes student potential and capability. TA draws on a US intervention, CFES Brilliant Pathways³, which has three components derived from the theory of Academic Capital Formation (St John et al. 2010) that aim to increase students’ understanding of college application and support services (Pathways to College), provide individual mentoring of students (Mentoring) and develop students’ leadership skills (Leadership through Service) (Hannon et al. 2017). A fourth aspect – twenty-first century (21C) teaching and learning – was added to the original CFES model, with the goal of supporting teachers to transform their pedagogical approaches to help develop students’ key 21C skills (Figure 1). This final component was combined with Leadership through Service, as there was significant overlap between many of the skills that were developed through participation in both practices. The new core practice has been termed Leadership in Learning.

Figure 1. TA intervention model (Tangney et al. 2021).

The Pathways to College core practice involves coordinating activities that provide students with a chance to investigate their post-school options. Activities include projects relating to college/courses choices, mock CAO/HEAR/DARE⁴ applications, campus visits, careers fairs, talks by experts and workplace visits. The mentoring core practice involves college-focused mentoring, peer-to-peer mentoring, and career-focused mentoring. Mentors include teachers, past pupils, community leaders, peers, adults and/or college students. Leadership in Learning involves providing professional development opportunities to teachers to assist them in adopting pedagogical approaches in their classrooms to promote the development of 21C skills. The focus of the CPD is on a particular pedagogical model developed in TCD (Lawlor et al. 2018) and ranges from workshops to an accredited postgraduate certificate. Schools are also encouraged to facilitate students to engage in a substantial service project as a means of developing their 21C skills. Cutting across all three practices are a series of student workshops held on campus. These are from 1 to 4 days in duration and cover a range of topics. The activities are described in more detail in (insert ref to overview report).

There are 20 ‘leader’ schools associated with the project. These are all post-primary schools located in the greater Dublin area, either with official DEIS status or situated in areas of low SES. They engage, to varying degrees, in all the activities offered by the project. In addition, there are 40 linked ‘network’ schools associated with the project. These are spread over a wide geographic area and receive less direct support from the team. In particular, their students do not get offered places in the workshops and other intensive on-campus activities. The ‘leader’ and ‘network’ schools approach allows the project to scale both deep and wide and to use learning from the scaling deep in scaling more broadly within the ‘network’ schools. An overview of the student activity provision with the TA team, for the 2020/21 academic year, is shown in Table 1.

Table 1. Student Activity 2020/21 (Tangney et al. 2021).

Programme type	Duration	Engagement hours	Participants
Campus visits, talks and Educational Awards	1 d	1–5	6815
Junior Cert & Leaving Cert Tuition	1–3 months	8–10	1402
Exploratory weeks and Summer Schools	1–2 weeks	20–50	673
Persistence Programmes	6–18 months	20–110	373

The degree to which the three core practices are implemented will vary from school to school. For example, implementation of the pathways to college core practice can range from a low level, in which at least one-year group visits at least one HE campus or careers fair each year, to a level of engagement in which all students in a school might visit more than one HE each year, while a highly engaged school would expand upon this to also involve parents in at least two school-based pathways to college activities in a given year. On one end of the spectrum, mentoring might involve a small number of sessions of peer-to-peer mentoring at junior cycle, or it could include multiple mentoring sessions with past pupils, members of the local community or teachers. The number of contact hours spent with mentees tends to range from three sessions a year to weekly check-ins with their mentors. Schools’ engagement with the Leadership in Learning (LiL) ranges from 30% of students being provided with the opportunity to engage in at least one student LiL project each year, to all students across the school being provided with the opportunity to engage in at least one student LiL project each year. In highly engaged schools, there are systems in place for sharing best practices in the classroom including co-teaching, peer observation and encouragement of professional development activities.

The Access Team in the university consists of ∼30 people of whom over half are involved in aspects of this project ranging from programme delivery to data collection and analysis. The team is led by one of the co-authors of this paper. The team is supported by a number of academics in the university (two of whom are co-authors of this paper).

Funding for the 2014–2017 phase of the project was provided by Google.org, the Corporate Social Responsibility arm of Google. In conjunction with significant and sustained fundraising by TA, Rethink Ireland, the social innovation fund of Ireland⁵, financed by the Irish government and philanthropic sources, has provided matched funding for the long-term ambitions of the TA project. It is anticipated that this funding stream will continue until 2023, and possibly beyond. Ethical approval has been obtained from the relevant Trinity College Dublin Research Ethics Committee.

Research questions and methods

With the exception of Smyth et al. (2011), most Irish studies, that specifically examine the effects of interventions aimed at widening participation in HE, have been relatively small-scale qualitative case studies, such as Hannon (2018) and Keane (2011).

In the case of this research, the methodology involves a large-scale observational cohort study of schools that are participating in the programme (Mann 2003; Parab and Bhalerao 2010; Song and Chung 2010). This is situated within a design-based research (DBR) paradigm (Anderson and Shattuck 2012), with a goal of helping to bring about positive change in partner schools. It involves a primarily school-led programme in which participating schools engage in TA activities as they see fit, with support from the university team when appropriate. The research aims are broad, involving evaluation of students’ experiences of the TA core practices; teachers’ integration of the core practices; the schools’ facilitation of supportive, collaborative and reflective environments; changes in students’ educational aspirations. The research also involves evaluation, and improvement, of the programme offerings at student and teacher level and development of structures to support schools in informed targeting of resources and interventions within their own context.

In order to achieve this, we are collecting a combination of students’ demographic data, self-reported levels of exposure to TA activities and information relating to their sense of self-worth, active engagement with education, sense of purpose, student voice, student–teacher relationship, self-efficacy in relation to key skills, educational aspirations, etc. Teacher data relating to their sense of self-efficacy, 21C teaching practices, perception of student behaviour, etc., will also be collected, as well information relating to school management structures and the levels of intervention in the school. The scope of data collection and analysis allows for the monitoring of changes over time at individual student, class group and school levels, as well as an exploration of the relationship between exposure/non-exposure to the TA programme and any changes that occur. In particular, the research questions that the study aims to address are as follows.

• Research Question 1: what is the impact of participation in TA on each of the primary stakeholders: students, teachers, school culture/management?

• Research Question 2: What are the reasons for change – what strategies, level of support from TA, level of engagement of school management, teachers and students, context, etc., have led to which changes?

• Research Question 3: How can this information be used to support schools in targeting interventions and resources where they are most needed?

Our hypothesis is that higher levels of exposure to the TA practices will be effective in increasing students’ future aspirations and goals, as well as their academic attainment, self-efficacy, confidence with key skills and so on. In order to develop some insight into the accuracy of our hypothesis, and thus answer the research questions outlined above, schools are requested to provide detailed information about the TA activities offered to the students in their school. Self-reported student data is also collected in order to gain the students’ perspectives on their participation with the programme.

In order to engage in an effective change process with schools, the goal of the data collection and analysis is not merely to identify what works at a general level but also to provide tools for schools to target interventions where they are most needed. The large-scale data collection permits the researchers to provide a snapshot of participating schools for a given year, and an in-depth view of changes over time for a particular year group (e.g. a comparison of 3rd years over the last five years) or cohort (e.g. an exploration of the progression of the 2015 cohort over the last four years).

The purpose of this paper is to provide an overview of the research design that is allowing us to address the far-reaching goals of TA.

Research approach: DBR

The TA project is pragmatic in nature, being primarily concerned with improving the situation in a way that is feasible and implementable (Johnson and Onwuegbuzie 2004; Johnson et al. 2007; Morrison 2007). It is situated within an overarching DBR paradigm, which is a methodology

designed by and for educators that seeks to increase the impact, transfer, and translation of education research into improved practice. In addition, it stresses the need for theory building and the development of design principles that guide, inform, and improve both practice and research in educational contexts. (Anderson and Shattuck 2012, 16)

An outline of how the TA intervention and associated research programme aligns with the seven principles that underpin DBR, as outlined by Anderson and Shattuck (2012), is as follows.

Authentic context

TA is an outreach programme, in which the widening participation practices are integrated into the partner schools with support from the university team. The authentic context of the intervention provides validity to the research, ensuring that the results can be used to assess, inform and improve practice, with the potential for scalability.

A focus on the design and testing of interventions

The design of the intervention is informed by relevant literature and theories and is grounded in local contexts. The cohort-study approach supports the identification of aspects of the intervention that are (and are not) having significant effects (Mann 2003), thereby facilitating an iterative approach to the refinement of the practices.

Mixed methods

The TA research design uses a quant-led, mixed-methods approach to data collection. Quantitative data are generated through closed-response survey items in the instruments that are distributed to all (consenting) participants. Open-ended survey items are included in each of the data collection instrument, providing scope for all stakeholder and beneficiary groups to provide depth and nuance to their responses where appropriate. Descriptive and inferential statistics are used for the analysis of the quantitative data, providing scope for presentation of findings to participating schools along with the development of models to explore relationships between variables, with thematic analysis (Clarke et al. 2015) used to generate themes from qualitative responses.

Iterative

Design, in whatever field, generally involves the generation of prototypes followed by an iterative process of testing and refining. The TA programme design and theory are continuously evolving, in an authentic educational setting. Annual reports are developed at project level and for each participating school, in order to continually inform practice. Changes made over the past few years include the introduction of a dedicated stipend for schools to facilitate teacher engagement in CPD; aligning our reporting on data collected from schools with data schools need to report as part of the DEIS scheme and the recruiting of teachers from linked schools to work directly as members of the project team.

Collaborative

DBR involves collaboration between practitioners and researchers. In DBR (unlike action research), the researchers take initiative in the process with respect to both the research and the design, while working with practitioners at a local level to develop and support suitable local practices and programmes.

Evolving design principles

The design of successful interventions is grounded in, and leads to, the development of practical principles, programmes and theories. These are not de-contextualised principles or grand theories that function with the same weight in all scenarios but reflect the context in which they are utilised. TA aims to support each school in identifying and addressing priority areas of development.

Practical impact

One of the fundamental tenets of DBR is that it is at least partly conducted in the natural environment and that it generates practical guidelines, principles and theory. The TA research programme is designed specifically for this purpose, aiming to effect positive change at local, system and policy levels.

While the main purpose of the TA research design is to shed light on the research questions posed above and to assist schools with the process of evaluation and improvement, the wide-ranging nature of the TA dataset will be useful to researchers and practitioners in various fields including education, sociology and economics. Data are being collected on an annual basis from all consenting students in 20 schools (with a total intake of ∼12,000 students), and include demographic information, educational background, subject choices, experience in school, future attitudes and aspirations, confidence with key skills and wellbeing (Table 2), as well as exposure to the three TA core practices.

Table 2. Summary of data collected using the Trinity Access instruments.

Student instrument	Teacher instrument
Personal information: DOB, Gender, School, Grade Demographic data: Cohabitees and Siblings Ethnicity and language Parents’ qualification and job Background: Behaviour in school Parents’ relationship with academic issues Subjects: Current subjects Subject levels and grades Future subject choices and reasons. Experience in school: Self-worth Active engagement Sense of purpose Student–teacher relationship Student voice 21C Key skills: Exposure to 21C practices in school Confidence in relation to: collaboration, communication, creativity, critical thinking, self-direction and technology for learning. Mentoring: TA21 process questions Usefulness of advice from various sources Pathways to College: TA21 process questions Aspirations and goals Future plans Confidence to achieve goals Leadership through Service: TA21 process questions Community and education Wellbeing	Personal information: DOB, Gender, School Teaching experience: Years teaching Grade levels Subjects CPD Participation in TA21 Continuous Professional Development (CPD), such as Postgraduate Certificate in twenty-first Century Teaching and Learning, workshops, conferences, etc. Participation in other CPD CPD needs Level of school support Community of Practice Level of engagement with in-school professional learning community Level of engagement with online professional learning communities Teacher self-efficacy Student engagement Classroom management 21C teaching practices Frequency of integration of key skills in class Perception of success of integration and assessment of key skills Assessment Methods Usage of Technology Self-efficacy Value Job satisfaction
School Instrument
Background Size of school Management structure Planning and organisation TA21 team	TA21 core practice evaluation Description of activity Frequency Participants

Sample

The TA sample was selected using opportunistic sampling, in that all of the schools participating are ‘Leader Schools’ as described above. Participants in the study are the students, teachers and principals of the project schools. Information sheets and consent forms have been distributed to all students and staff in the participating schools with the final sample made up of those that provide explicit consent to take part.

Owing to the voluntary nature of engagement with TA, there are diverse levels of engagement with the project among participating schools. As such, the research programme will focus on measuring the level of implementation of the intervention in these schools, as well as outcomes at student, teacher and school level, and aim to draw correlations between the two. This obviates the need to include control schools in the study as the variation in implementation provides the control data.

Instrument design and testing

The TA student questionnaire includes process questions relating to participants’ experiences with the core practices of Pathways to College, Mentoring and Leadership in Learning. In addition to these process questions, demographic information and what we term information relating to outcomes such as post-secondary school plans and aspirations, self-efficacy in relation to key skills, wellbeing, etc., are also collected. A summary of the data collected for the study is presented in Table 2.

In an effort to ensure validity and reliability of results, the vast majority of scales used in the student and teacher questionnaires were drawn from instruments that had already been validated and whose internal consistency had been established. Rigorous validation of scales that were developed for this study has also been carried out (Bray, Byrne, et al. 2020).

Student instrument

wIn order to establish construct and sampling validity, the student instrument was discussed with a panel of experts, as recommended by Rubio et al. (2003), including academics and practitioners from the fields of sociology and education. Having agreed that the selected items were good choices to capture the desirable concepts, a pilot instrument was developed. Once the instrument was agreed upon, it was piloted with a class of grade 10 students (ages 13/14, n = 15), from a co-educational school that has been designated as ‘disadvantaged’. The reason for this was to ensure that the language of the instrument could be easily understood by a group of young people who were roughly representative of our target participants. Some minor adjustments to language were made and the instrument was re-tested with another similar cohort (n = 20). No further issues emerged.

Teacher instrument

Development of the teacher instrument followed a similar trajectory to that of the student questionnaire. Initial design decisions were made in consultation with various stakeholders in order to establish construct and sampling validity. The instrument was piloted with 20 teachers in a small post-primary school. Feedback from the teachers was acted upon, resulting in a revised survey, which was again piloted with another similar sample of teachers.

School instrument

The primary goal of the schools instrument is to gather information and evaluate schools’ implementation of the three TA core practices. It is a straightforward instrument used to record the number of TA activities in each school, the number of participants, and which classes they were drawn from. This high-level survey is also used to collect information about the school management and organisation, and the level of support provided to the TA team in the school.

Procedure

Ultimately the goal is to survey all students and all teachers in all participating schools. However, it is unlikely that this level of coverage will be achievable due to logistical difficulties and the challenges in obtaining consent from all the students’ parents/guardians. To address this challenge, and in consultation with the schools, a number of innovative strategies to maximise the percentage of participants completing the surveys have been devised.

• From experience in the pilot phase, it became clear that the use of paper-based surveys, managed and administered by teachers, was not appropriate for data collection of this scale; however, many of the link schools do not have the computing and network facilities to allow for online data collection. For this reason, the decision was taken for the TA research team to collect the data in person, by travelling to the schools with a bank of up to 100 tablet devices with the surveys pre-loaded on them. The data collection process involves cycling through the class groups, facilitating the completion of the survey by each group of consenting students within a single class period and with no requirement for network access. The survey software Qualtrics⁶ is used as it allows for the data collection to be conducted offline and the results to be securely uploaded later to a central server on return to the university. The process was piloted in December 2018 with 22 students in one school and was shown to be an effective and efficient method of data collection.

• Staff are requested to complete the teacher survey annually, during a whole-school staff meeting or other allocated time.

• The school data is provided by the TA project officer in each school, or by the school principal.

In order that such large-scale data collection is not seen as an additional burden, but is rather integrated into best practice in a school, a clear crossover between the Quality Framework for Post-Primary Schools (Department of Education and Skills 2016), and the TA student and teacher instruments have been identified (Table 1). The Quality Framework is a departmental publication that supports schools in the evaluation of their practice, with the goal of improving teaching and learning. If our hypothesis holds and the project aims are achieved, the schools will be able to use the data from this research as evidence when they are completing the School Self-Evaluation reports that are required by the Department of Education and Skills. By analysing the combined data from the three instruments identified above, we will be able to identify what schools are doing (outputs), and we will be able to report on the impact this is having in relevant areas at school, teacher and student levels (outcomes).

TA is using the results of the analysis of these data as a kind of ‘diagnostic tool’ for schools to pinpoint where relevant interventions would be of most benefit to the school and to which student cohorts. The data is being used to create interactive ‘dashboards’ for each institution that enables the school to identify the data that is relevant to them and to easily visualise it in a meaningful way. An example of what this will look like, based on a trialling of the whole-school data collection with two schools in December 2018, is provided in the next section.

Quantitative data are being collected annually in March/April. The rationale for administering the surveys at this point during the school year is that the participants will be well advanced in the academic year and should have been exposed to a number of TA activities. Collecting data earlier in the academic year, when they return to school refreshed after the summer holidays, could provide an unrealistic picture of participants’ experience, and later in the year tends to be very hectic, and staff and students are more likely to be fatigued.

While student and staff surveys will be administered on an annual basis, the schools instrument is designed to be used as a record-keeping document and as such, should be updated throughout the year in order to provide a comprehensive picture of students’ exposure/non-exposure to the intervention.

Trialling of the whole-school approach to data collection

A trial run of the whole-school approach to student data collection was conducted during December 2018. This involved members of the research team visiting two schools, bringing with them all of the required resources and surveying every student who had provided the required level of consent. The primary purpose of this data collection was to explore the logistics of this scale of data collection, as well as to further test and inform refinement of the student instrument. In addition to this, baseline reports were created for the two schools.

Logistics

In terms of logistics, the team conducted two whole-school data collections in 2 days. This amounted to a total of roughly 300 students. The data collection was administered by TA staff, and all consenting students in each school were surveyed on a single day, with all resources provided by TA. The data collection pilot was very successful, in that it clearly exposed what worked well as well as areas that needed to be focused upon.

What worked

From a technical point of view, the trial run was extremely successful. There was an adequate number of devices, they maintained a charge for the full day, and the surveys were easy to access and complete. The consistency of the approach to data collection across schools and class groups should bolster the reliability of the results.

The first school was an all-girls school in Dublin. The process in this school was particularly streamlined. The week before the TA researchers conducted the data collection, they were provided with a list of all of the students who had provided consent (∼50% of all students in the school) and a timetable of which classes would participate at what time. Each student was assigned a unique identification number and ID cards with the student’s name and number were generated for all participants. In the school, the gym hall was laid out with individual desks to which the ID cards were affixed by class group and in alphabetic order. Before each session, the list of participants was called over the intercom and they made their way to the gym hall, where they found their allocated desks. After a brief introduction, the students began the survey, and in most cases were able to complete it within half an hour.

In this school, the principal had asked for extra questions relating to formative assessment to be included in the survey. This request was easily accommodated, strengthening buy-in from the school management.

Where were the issues?

The main issue that has emerged with data collection on this scale is obtaining written consent from the parents/guardians of such a large number of participants under the age of 18. We envisage that this will become less of a problem as the project progresses, as the schools have agreed to include the consent forms with the introductory material provided to the parents/guardians of incoming students at the beginning of their child’s time in the school. This should lead to higher response rates as the project moves forward.

The second school was a co-educational school on the outskirts of Dublin. The coordinating teacher experienced difficulty collecting signed consent forms from the students’ parents/guardians and for this reason, a list of participants was not provided until the evening before the scheduled date. The room set-up was not particularly conducive to the data collection, as it was situated in a building off the main campus. Other teachers in the school were not fully aware of the data collection and were not always amenable to letting their students out of class. On a number of occasions, the students did not have adequate time to complete the survey.

In both schools, some of the younger groups needed assistance with the questions and did not manage to complete the full survey in the given time.

Refinement of the instrument

By testing the survey instrument with samples of students from each age-category, it became clear that some adjustments were required. The core survey was quite long and while there is benefit in collecting information that may not be immediately useful, it was necessary to cull some of the questions. Following the trial run, the team spent some time revising the survey by aligning it more closely with the research questions. In this way, we should be comfortably able to facilitate the inclusion of questions that are of particular interest to a school.

In addition to this, it was noted that having questions relating to culture, ethnicity, levels of parental education, etc., could potentially skew the responses negatively for students who may feel ‘less’ than others (Hoff and Pandey 2004; Wilkinson and Pickett 2011). For this reason, such sensitive demographic questions were repositioned to the end of the instrument.

Baseline reports

The data collected in the trial run were analysed and interactive reports were created using the Tableau data visualisation platform.⁷ This allowed the data to be visualised so as to provide a summary picture of the whole school or as a selection of groups of interest (Figure 2). Figure 3, which highlights the future plans of each of the year groups, provides an example of the potential diagnostic power of this approach to data visualisation.

Figure 2. Whole school and by year views on data.

Figure 3. School A – future plans.

In these graphs, it is clear that in five out of the six year groups, over 50% of students intend to progress to a 3–4-year Higher Education degree. However, in the 3rd year cohort (15–16-year olds) approximately 75% plan to progress to a trade or apprenticeship programme. This is an interesting finding from the school’s point of view and could indicate that it may be beneficial to engage in more college-related activities with the 3rd year group in order to broaden their aspirations, or indeed, that further-education options should be explored more with the other year groups.

As more data are collected, the value of the interactive element of the visualisation will become increasingly evident, affording the capacity to view the progression of certain cohorts over time, as well as the comparison between year groups.

Large-scale data collection

The first implementation of the large-scale data collection process took place in March/April 2019. A total of 3863 post-primary students, from 17 of the 20 TAP-linked post-primary schools, participated in the first iteration of TA data collection. This represents approximately 42% of students across the 20 schools. Principals in the 20 schools were asked to share a link to the teacher questionnaire with their teaching staff, and a total of 510 teachers responded to the survey from 14 schools.

Analysis of these quantitative data has involved both descriptive and inferential statistics. The illustration of the descriptive results forms the main body of the individual school reports, with inferential results from correlation and regression analysis, comparison of means, differences between groups, etc. informing the high-level report and providing a lens through with to view, understand and act upon the descriptive findings.

The second cycle of large-scale data collection began in early March 2020, but after the first school visit by the research team, the Covid-19 pandemic necessitated a sudden shift in focus. The surveys were adapted for distribution via online channels only, and the data collection instruments were edited to allow for data to be collected in relation to student and teacher experiences of online schooling. Open-ended questions were included in the surveys such that the respondents could provide depth to their answers, particularly in relation to their perception of any positive or negative changes as a result of the move online. Quantitative responses were analysed as before, with thematic analysis (Clarke et al. 2015) used to explore the qualitative responses. Results and recommendations from the analysis of these surveys have been published in a series of reports and a journal article (Bray et al. 2021; Bray, Ní Chorcora, et al. 2020; Devitt, Bray, et al. 2020; Devitt, Ross, et al. 2020).

The findings of the data analysis have also been presented to the participating schools and other stakeholders. They have been used to inform the development and refinement of offerings from TA, as well as how the schools are working with TA to target particular cohorts who have been identified as being most at need. Formal reporting and write-up of the data are ongoing (Bray and Byrne 2019; Bray, Ní Chorcora, et al. 2020; Devitt, Bray, et al. 2020).

Strengths and limitations

A number of lessons regarding the research design were learnt from the 2014–2017 pilot study. Weaknesses identified were the degree of participant attrition and the difficulties in administering the surveys in a systematic manner across the schools. In particular, it was challenging to obtain responses from groups in the control schools as they were not benefitting from participation in the programme and had little motivation to take part in data collection. There were also problems with the identification of the levels of exposure of students to the intervention. The revised design attempted to address these issues.

Strengths

The implementation of the research design described in this paper is having a very positive effect on the quality of the data collection. It is an efficient, streamlined process that results in little disruption for the schools, teachers and students. The administration of the student instrument by the research team has reduced the potential for administration mode bias (Bjarnason 1995), by increasing the perceived anonymity of the instrument by the students. Reduced potential for student attrition, as well as higher levels of consistency in the administration of the survey, have also been observed, as the students are all required to complete the survey in a single sitting, under the supervision of the researchers.

Even after just one year, the scale of the data collection has resulted in an important bank of information relating to various demographic, attitudinal and aspirational attributes of the participants allowing the researchers to look at correlations and interplay between these constructs. The inclusion of parental consent forms in the induction packs for incoming first years should increase the level of student participation each year. It is expected that an increase of ∼2000 students per annum is not unreasonable, although this is subject to the changing impact of Covid on schools.

The collection of self-reported and school-level responses relating to the exposure of the students to the aspects of the intervention has permitted triangulation of the data, providing a clear picture of what is actually going on in the schools in relation to TA. This has facilitated an accurate analysis of any correlations between the intervention activities and the various student/teacher attributes. Identification of the level of exposure/non-exposure of students to the intervention has also provided an alternative to the use of control groups for comparative purposes.

It is envisaged that the design of the new research instruments will improve the validity of the responses. The instruments themselves have been reduced in length, which should lessen levels of participant fatigue and non-completion. The use of validated scales in the construction of the surveys provides weight and reliability to the findings.

Weaknesses

Inevitably, with any research study, there are some weaknesses in the design. In the TA study, while some of the schools to date have engaged at a very high level with the programme, others have had much lower levels of participation. This can be related to levels of support from management, staff buy-in or school culture and infrastructure. We aim to address this issue by attempting to gather valid information about the intervention in each school.

Issues around collecting consent from students’ parents/guardians emerged as a problem. This should become less of an issue as time goes on, as consent will be collected from parents/guardians when their children join the school, as part of the formal entrance process. For students currently enrolled in the school, every effort will be made to ensure the return of new consent forms with each round of data collection. Other issues around the logistics of the day in individual schools have also been identified. It is evident that having whole-school and management support is an essential component for the success of the data collection process. Once again, we believe that time will play a positive role with respect to schools’ participation; as the value of the data produced becomes more evident, schools will be more positively disposed to come on board.

The student and teacher instruments both include a high number of self-reporting measures. This could be viewed as a weakness as responses can be easily influenced by external factors. The scale of the study should go some way to balance this, as a collection of such a high number of responses should offset any such problems. It is also anticipated that the data collection will be broadened to include formal attainment information for the students, permitting analysis of the impact of the programme on concrete levels of academic achievement. In addition, the repositioning of sensitive demographic questions to the end of the student questionnaire should also have a positive impact on the reliability of the data.

The TA programme is being rolled out during a period of systemic changes in education at a national level. The phased introduction of a new Junior Cycle making up the first three years of post-primary school began in 2017, with an emphasis on key skills and new approaches to assessment (Department of Education and Skills 2015), and a review of the Senior Cycle is currently underway. While these curriculum reforms are broadly in line with the pedagogical goals of TA, it will make it less easy to identify causal relationships. Once again, collection of accurate information relating to levels of student exposure to the core practices should go some way to address this.

Discussion and conclusion

The TA post-primary outreach initiative offers a range of programmes for students and teachers in areas of socio-economic disadvantage with low progression rates to higher education. All aspects of the programme are backed by rigorous research that aims to find out what really works for all of the beneficiaries.

The aims of the TA research study are far-reaching and are underpinned by an ambitious design involving a longitudinal, DBR approach, with annual data collection at a whole-school level across twenty schools. The goal is to survey all students, teachers and management, in all participating schools; while achieving this level of data collection is challenging, owing to difficulties in obtaining consent for student participation and a potential lack of uptake among teachers, a number of innovative strategies to maximise participation have been devised in consultation with the schools. The resulting research design and approach to data collection have led to an efficient, streamlined process, that results in minimal disruption for the schools, teachers and students. The administration of the student instrument by the research team has limited the potential for administration mode bias, provided high levels of consistency and reduced the potential for student attrition. A database storing the demographic, attitudinal and aspirational attributes of the participants has been created and will be updated annually, allowing the research team to track change over time at the individual student level. This has led to a first cycle of data collection with responses from ∼4000 students and ∼510 teachers across 17 of the 20 participating schools. Analysis of these data is ongoing and will be used to identify intervention effects, levels of exposure and associations between exposure and these effects. Going forward the richness of the data will allow for interrogation of different characteristics such as gender, ethnicity, etc., to see to what extent they are significant factors in increasing aspirations.

In order to encourage schools to take part in such large-scale data collection, clear parallels have been identified between TA research aims and departmentally mandated evaluation of practice. As a result, schools are able to use the data from this research as evidence when they are completing the School Self-Evaluation reports that are required by the Department of Education and Skills, thereby reducing the overall burden of data collection and evaluation. Furthermore, the nature of the data being gathered by through this research has an immediate benefit for participating schools by enabling them to identify patterns of behaviour in the school, as well as year groups or cohorts that require targeted intervention.

It is clear that the TA research will have significant benefits for the participating institutions. It is our belief, however, that the success and impact of this particular research design have scope for being replicated by other institutions engaging in widening participation or indeed any outreach activities.

Acknowledgements

The most important acknowledgement is to the participants of the study: the teachers, students and management in the schools who have worked with us to develop the programme and the research design; to the members of the TA research team and wider TA team; and to all the funders and supporters of Trinity Access.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Aibhín Bray

Dr Aibhín Bray is an Assistant Professor in the School of Education in Trinity College Dublin. She previously worked as the co-ordinator of research for Trinity’s widening participation programme: Trinity Access. She holds a B.A. (Int) in Mathematics and Italian and an M.Sc. in Computer Science from the National University of Ireland (University College Dublin), and a H.Dip. in Education and a Ph.D. in Technology and Mathematics Education from the University of Dublin.

Cliona Hannon

Dr Cliona Hannon was a director of Trinity Access at Trinity College Dublin from 1999 to 2021 and a visiting fellow at Lady Margaret Hall, Oxford University (2016-2021). She has a B.A. and Ph.D. from the University of Dublin, an M.A. (Higher Education and Training Awards Council) and an M.B.A. from the Open University. Her role as director in Trinity Access involved the development and implementation of strategic priorities relating to widening participation and lifelong learning. She led the development of TCD’s widening participation offerings, which now comprise a continuum of programs for primary school pupils right through to post-graduation, engaging socio-economically under-represented groups, adult learners and students with intellectual disabilities, and involving transformative educational processes through the use of new technologies.

Brendan Tangney

Dr. Brendan Tangney is a Professor in Computer Science & Statistics and a Fellow of Trinity College Dublin. He holds an M.Sc. from the University of Dublin and a Ph.D. from the University of Bolton. His research focuses on the overlapping areas of technology & learning, teacher professional development, educational reform and equality of access. He is the academic champion of Trinity Access. He has held visiting positions in the Universities of Sydney and Kyoto. He is a member of the Editorial Board of Computers & Education and the Journal for STEM Education Research.

Notes

1 https://www.gov.ie/en/policy-information/4018ea-deis-delivering-equality-of-opportunity-in-schools/#deis-schools-2021-2022

2 http://accesscollege.ie/

3 https://brilliantpathways.org/

4 These are the various national centralised schemes for applying to HE.

5 https://rethinkireland.ie

6 https://www.qualtrics.com/uk/

7 https://www.tableau.com/