Social-emotional learning intervention effects: the role of universal versus indicated implementation settings

Abstract

We assessed the role of implementation setting (universal or indicated) on the effects of social-emotional skill (SEL) intervention, focusing on school climate and psychosocial adjustment. Taking Topper Training as a case intervention, we used pretest-posttest data from 444 eight to twelve-year-old children across three conditions: universal implementation, indicated implementation, and no-intervention control. School climate improved in the universal implementation setting and psychosocial adjustment improved in both implementation settings. There were no significant differences between the implementation settings, however, larger effect sizes on psychosocial adjustment were found in the indicated implementation setting. Universal implementation of SEL interventions seems most effective when targeting the broader school environment (i.e., school climate), but also has small effects on psychosocial adjustment. Indicated implementation seems to yield larger effects on psychosocial adjustment. This suggests that practitioners should carefully align implementation settings with the behavior they aim to change when selecting interventions.

Children with social-emotional skill deficits generally show more problem behavior (Jewell et al., 2010). In turn, this is related to more adverse well-being outcomes later in life, such as substance use, financial and health problems, and delinquency (Moffitt et al., 2011; Von Stumm et al., 2011). Thus, if left unattended, social-emotional skills deficits can set off a chain of detrimental events and consequences (Boutin et al., 2020; van Lier et al., 2012). So how can we best help children improve their social-emotional skills?

Many programs are available to stimulate children’s positive social development and social-emotional skills. Social-emotional skills (SEL) interventions generally focus on improving individual children’s skills to improve psychosocial adjustment and improve school climate. Psychosocial adjustment reflects a construct of a child’s mood, self-esteem, and behavior toward others. School climate refers to children’s thoughts and feelings elicited by the school’s environment and reflects children’s perceived quality of social relationships with teachers and peers. These factors all contribute to children’s well-being in a context that is an important part of children’s lives. A school’s climate is considered positive when the environment supports an individual’s perception of social, emotional, and physical safety (Kutsyuruba et al., 2015; Leadbeater et al., 2015).

In our collective effort to better understand the development of social-emotional skills and ways to improve them, until now little attention has been paid to the contextual factors that may influence SEL intervention effects. One of these contextual factors relates to the implementation setting for SEL interventions. Specifically, we know less about how universal versus indicated prevention settings of SEL interventions matter for the effects we find—both on individual children’s psychosocial adjustment and on school climate.

Universal and indicated implementation settings for SEL interventions

Universal interventions target an entire group of children (e.g., a classroom) regardless of individual risk factors. In practice, this means that children with detectable symptoms of problem behavior or social-emotional skill deficits and children with adjusted social-emotional skills are addressed in the same group in such interventions. An advantage of the universal approach is that these programs can focus on multiple behavior problems across various experiences and settings and that peers can model opportune behavior (Greenberg & Abenavoli, 2017).

Indicated interventions are directed specifically at those children with detectable symptoms of disordered behavior but without a formal clinical diagnosis (Greenberg & Abenavoli, 2017). Indicated interventions are provided in smaller groups outside the classroom and may also be provided in a mental healthcare facility (i.e., equivalent to outpatient care; Sundararaman, 2009). An advantage of the indicated approach is that it allows for a more targeted intervention that addresses only those children in need of support. Also, due to the smaller group size, it may be easier to uphold treatment fidelity in indicated programs.

Meta-analyses have shown that universal SEL interventions are moderately effective in improving children’s social-emotional skills in general (de Mooij et al., 2020; Durlak et al., 2011; Taylor et al., 2017) and for children with internalizing and externalizing problem behavior specifically—those children that might be selected for an indicated intervention (Cook et al., 2008). In other studies, indicated SEL interventions have also been proven to significantly improve children’s social-emotional skills and reduce internalizing and externalizing behavior (e.g., Stoltz et al., 2012). However, these previous studies have not actively compared any universal versus indicated implementation settings when examining SEL intervention effects. Based on a close reading of the literature, however, this does seem crucial. Here, we argue that the implementation setting will very likely impact the effectiveness of SEL interventions on children’s perception of school climate and their psychosocial adjustment.

Impact of implementation setting on SEL intervention effects

There are two reasons to believe that universal and indicated interventions yield differential effects on how children adjust psychosocially and how they feel at school. For one, interventions in these two implementation settings may differ in the extent to which they match children's individual needs (Bonta & Andrews, 2007). Universal interventions are mostly implemented in contexts with only a few at-risk children, which means that substantial effort is spent on children who do not experience psychosocial maladjustment and thus will not likely show much improvement in this domain. Also, indicated intervention programs may be better suited to tackle psychosocial maladjustment because of the increased opportunity for individual attention in a smaller group intervention and more optimal dose-delivery of intervention components (de Mooij et al., 2020; Stoltz et al., 2012). To illustrate, a recent study demonstrated that children categorized as being at risk for moderate to severe behavioral problems did not respond to the universal intervention being assessed (Spilt et al., 2013), probably because program intensity was too low to change at-risk children’s behavior. Thus, we expect that indicated interventions will yield significant effects on children’s psychosocial adjustment, and universal interventions will not—or hardly.

Another reason to suspect universal and indicated interventions may yield differential effects is that they differ in emphasis or intervention content. Interventions provided in both settings aim to improve children’s social-emotional skills and to yield a more pleasant day-to-day life, which is mostly centered around school. It is plausible, however, that universal interventions are more effective in improving children’s experiences at school, as these are more focused on the self in connection with others, whereas indicated interventions focus more on the children’s inner world of belief. Taking an ecological perspective (Bronfenbrenner, 1979; Trach et al., 2018), it seems most effective to address children’s social-emotional behavior within the peer and school context, because specific (negative) social behavior may be driven by within-group processes and individuals’ motivations to fit in with their peers (Ellis & Zarbatany, 2007). Consequently, we expect that universal interventions will yield significant effects on children’s perceptions of school climate, and indicated interventions will not—or hardly.

Until now, no convincing evidence has emerged from our literature about the relative impact of universal versus indicated implementation setting on SEL intervention effects on school climate and psychosocial adjustment. One recent meta-analysis of SEL interventions did assess the moderating role of intervention structure but did not find a difference between universal and indicated program effects on psychosocial adjustment—although in this study no findings regarding school climate were reported and only a relatively small number of indicated prevention programs (12 vs. 37 universal interventions) were tested, across interventions that were variable in content (de Mooij et al., 2020). To our knowledge, no other studies have previously compared the effects of SEL intervention on children’s perception of school climate and psychosocial adjustment in an indicated versus universal (i.e., classroom) setting.

The present study

The core question of this study was: “Are social-emotional learning interventions implemented in a universal or indicated setting differentially effective on school climate and psychosocial adjustment?”. To examine this question we assessed secondary data of a social-emotional learning (SEL) intervention implemented across universal and indicated implementation settings. We hypothesized that the universal and indicated intervention would yield differential effects: SEL interventions in universal (i.e., classroom) settings would improve children’s perceptions of school climate (school enjoyment, social acceptance, and relationship with teacher) but not their psychosocial adjustment (depressed mood, self-esteem, narcissism, negative intentions, and disruptive and prosocial behavior), and SEL interventions in indicated settings would lead to improvements in psychosocial adjustment but not school climate.

Methods

Participants

Our dataset consisted of 444 children from grades three to six in Dutch elementary schools that participated in the social-emotional learning (SEL) intervention Topper Training (called “Kanjertraining” in the Netherlands) or a control group. Each condition (i.e., universal condition, indicated condition, and control condition) contained 148 children. The mean age of the sample was 9.75 years (SD = 1.54; range 7–16 years), and 55% (n = 244) of the participants reported their sex assigned at birth was female. Data on children’s ethnicity was only available for the participants that were matched from the trial implemented in Dutch schools (Vliek, 2015). Ethnicity was defined based on the standards of the Dutch Bureau for Statistics. In the universal condition, 79.7% (n = 118) of the children in the control condition were Dutch (i.e., two Dutch parents), 3.4% (n = 5) of the children had at least one non-Dutch, Western parent, and 11.5% (n = 17) of the children had at least one non-Dutch, non-Western parent. For eight children in the universal condition information on ethnicity was missing. In the control condition, 77% (n = 114) of the children in the control condition were Dutch (i.e., born to two Dutch parents), 6.1% (n = 9) of the children had at least one non-Dutch, Western parent, and 8.8% (n = 13) of the children had at least one non-Dutch, non-Western parent. For 12 children in the control condition, this information was missing. Data on ethnicity was also missing for all children in the indicated condition.

Based on norm scores for the Topper Questionnaire (Vliek et al., 2012), 6.4% of the total sample (n = 28) children reported at-risk or problematic levels of disruptive behavior, 37.6% (n = 167) children reported at-risk or problematic levels of depressed mood, and 3.7% (n = 16) children reported at-risk or problematic levels of negative intentions. Taken together, 39.4% (n = 175) of the children reported at-risk or problematic levels of behavior in one or more of these outcome variables.

Procedure

We pooled data from three trials that assessed the effects of SEL intervention Topper Training (Vliek, 2015; Vliek et al., 2014, 2019). These trials were performed in line with the principles of the Declaration of Helsinki and were approved by the Ethics review board of the Faculty of Social and Behavioral Sciences of the University of Amsterdam, The Netherlands (number 2014-CDE-3827).

The datasets were merged to create a new dataset with three conditions: one universal condition, one indicated condition, and one control condition. The first original trial from which we used data (n = 690, labeled “universal trial”) assessed the effects of Topper Training in classrooms characterized by disruptive behavior and peer problems using a quasi-experimental design (Vliek, 2015). Schools were recruited for participation through advertisements in local newspapers, journals for professionals in education, or through their professional social network. After applying to the study, a psychologist assessed the levels of problems experienced in a classroom based on discussion with the classroom teacher and head of the school. Schools that participated in the study implemented Topper Training due to a climate in the classroom characterized by distrust and problematic social interactions. In this trial, the universal intervention as described below (see “Intervention”) was implemented.

The other two original trials were taken up into one condition (n = 712, labeled “indicated trial”) and assessed the effectiveness of Topper Training as an indicated intervention in groups of children with heightened disruptive behavior and peer problems using an RCT design (Vliek et al., 2014, 2019). Children came to the training through school advice boards, referred by their family doctor, through advertisements in local newspapers, or referrals from their social network. Children who participated in these trials were signed up for Topper Training by their parents, who completed the Child Behavioral Checklist (CBCL; Achenbach, 1991) to assess children’s level of problem behavior. Children were accepted into the intervention if they experienced externalizing and/or internalizing problem behavior (Vliek et al., 2014, 2019). In these trials, the indicated intervention as described below (see “Intervention”) was implemented. The control groups from the original trials were combined into one control group for the current study (n = 399). In all trials, data was collected from children, their parents, and teachers under the supervision of the lead researcher on these trials (LV). In the current study, only child self-report data was included, as parent- and teacher-report data could not be harmonized.

Due to differences in design, recruitment, and screening of participants in the original trials, it was vital to match participants for the current study. Figure 1 presents a flowchart of the merging and matching phase of this study, resulting in the final dataset. First, the datasets of the different trials were merged. In this process, cases were excluded using listwise deletion if they participated in the 3-day intervention, if the period of measurement was more than 10 weeks, or if there was no posttest data available. Only self-reported data on the School Attitude Questionnaire and the Topper Questionnaire (see paragraph “Outcome Measures”) we retained. Other measures that were included in the original trials were not included in the new dataset, as these were not available for all trials. For example, the indicated trials also included parent-report data from the Child Behavioral Checklist (Achenbach, 1991), but this data was not available for the universal trial.

Figure 1. Participant flow chart.

Once the datasets were merged, children with incomplete pretest data on the School Attitude Questionnaire and the Topper Questionnaire were excluded using listwise deletion. This was the case for n = 77 children (9.2% of 827 participants). This resulted in a pooled dataset containing n = 750 children, n = 200 of which were in the universal condition, n = 211 in the indicated condition, and n = 339 in the control condition.

In the next step, we used propensity score matching (Pan & Bai, 2015) to balance the distribution of covariates in the intervention and control groups, to create the final dataset. This method reduces bias in the estimation of the intervention effect (Stuart, 2010). Logistic regression analyses calculated a propensity score used to create three groups (universal implementation, indicated implementation, and control) that had comparable pretest scores on all outcome variables. Data from children in the indicated implementation served as a reference point. Propensity scores were matched using a match “tolerance” value (also referred to as a caliper) of 0.1 (Bartak et al., 2008). The match tolerance determines how close the propensity scores of two participants must be for them to be considered a suitable match. In this study, children’s calculated propensity scores could differ by 0.1 from each other at the most to be considered a successful match. A larger difference between propensity scores means there is no match between participants (Stuart, 2010).

Of the n = 211 children in the indicated condition, data from n = 155 children could be matched to children in the universal condition. For the n = 56 unmatched children in the indicted condition, the levels of psychosocial adjustment and school climate were too different from those in the universal trial (i.e., tolerance >0.1). The n = 45 children in de universal condition for whom no match was found, were also excluded. Next, the remaining n = 155 children in the indicated condition were matched to children in the control condition. A match was found for n = 148 children. The n = 191 children in the control condition for whom no match was found, were excluded. The n = 14 children (n = 7 in both the universal and indicated condition) for whom there was no match in the control condition were also excluded from the dataset. This matching procedure resulted in a final dataset of N = 444 children, with n = 148 in each of the three conditions.

In the final dataset, data was complete for n = 286 (64.4%) of the participants. For n = 27 (6%) participants, posttest data on the School Attitude Questionnaire was missing. For n = 26 (5.8%) of the participants, posttest data on the Topper Questionnaire was missing. For n = 105 (23.6%) of the participants, posttest data on both the School Attitude Questionnaire and the Topper Questionnaire was missing.

Outcome measures

School climate

The Dutch School Attitude Questionnaire (Schoolvragenlijst in Dutch; Smits & Vorst, 1990) was used to assess children’s attitudes toward and experience of their school life and classroom climate. We included the subscales for school enjoyment, social acceptance, and relationship with the teacher. Together these scales make up a composite of overall well-being at school. The three subscales combined include 16 items, which are answered on a three-point scale (1 = this is true to 3 = this is not true). Example items are “I get along well with my teacher,” “I often feel lonely in the classroom,” and “I enjoy going to school.” Higher scores reflect more favorable outcomes. The psychometric properties of the School Attitude Questionnaire are satisfactory (Smits & Vorst, 1990). The School Attitude Questionnaire Scale reliability was found to be good (αs > .80) in the original trials (Vliek, 2015).

Psychosocial adjustment

The Topper Questionnaire (Topper Training Foundation, 2007–2023) was used to assess children’s social and emotional adjustment. We included subscales for prosocial behavior, depressed mood, disruptive behavior, self-esteem, narcissism, and negative intentions. Each subscale included ten items answered on a four-point scale (1 = totally not true to 4 = totally true). An example item is “I am good at helping others”, “I dislike myself”, and “I want to be honest in school”. Higher scores on the subscales of prosocial behavior and self-esteem and lower scores on subscales of disruptive behavior, depressed mood, narcissism, and negative intentions reflect more favorable outcomes. The psychometric properties of the Topper Questionnaire are good, and scale reliability was found to be good (α's > .70) in the original trials (Vliek, 2015).

Factor analysis

To statistically confirm that the constructs of school climate and psychosocial adjustment indeed measure separate constructs, we ran a Principal Component Analysis in SPSS. This analysis produced a three-factor model with eigenvalues ranging from 1.436 to 3.723, together explaining 69.285% of the total variance. The first factor included all subscales of the TAQ, which had factor loadings of 0.765 to 0.984. The second factor included the TQ-subscales prosocial behavior, depressed mood, and self-esteem, and had factor loadings of 0.638 to 0.860. The third factor included the TQ-subscales narcissism, negative intentions, and disruptive behavior, which had factor loadings of 0.450 to 0.831. There were no strong correlations between the three factors. These results are a confirmation of the difference between the concepts of school climate and psychosocial adjustment.

Intervention

Topper Training is a SEL intervention based on cognitive behavioral theory (Topper Training Foundation, 2007–2023). The core aims of Topper Training are to teach children to behave authentically, be trustworthy, and socially competent, and with respect for others and themselves. Topper Training can be tailored to a specific implementation setting: a school-wide universal program aimed at promoting positive behavior, a universal intervention program aimed at classrooms in crisis, and an indicated program aimed at individual children presenting with moderate to severe psychosocial problems.

There are several similarities across the interventions implemented in the different settings. Topper Training includes multiple intervention components, such as emotional, cognitive, and skill-building components. In ten 90-minute sessions, children practice several skills: basic social skills, such as presenting themselves, how to address bullying situations, how to show interest, trust, and friendship, giving and receiving feedback, and reflecting on their worth. Topper Training reinforces children’s positive intentions and affirms their capacity to choose how to behave. This intervention goal is grounded in self-determination theory (Ryan & Deci, 2000). Through teaching these skills, Topper Training aims to improve class climate, self-esteem, and prosocial behavior and improve children’s psychosocial adjustment (Vliek, 2015). Parents and teachers are involved in the program: they are taught the Topper Training principles and how to support children’s positive behavior through feedback, setting and maintaining limits, and behaving authoritatively. Multiple studies have shown that Topper Training effectively improves children’s school climate perceptions and psychosocial adjustment (Vliek, 2015; Vliek et al., 2014, 2019).

This study assessed the effects of the universal and indicated implementation version of Topper Training. The universal crisis intervention version of Topper Training is implemented in classrooms that are socially disrupted to such a degree that teachers can no longer handle the situation. In these classes, children may have mild to severe psychosocial problems, a general classroom climate of distrust is extant, and social interactions are problematic (including but not limited to bullying and victimization). This program version is directed at the whole classroom (∼23 children) and is implemented by a licensed and trained psychologist.

The indicated version of Topper Training is implemented in a mental healthcare setting, which means that children and their parents visit a mental healthcare facility not affiliated with the school. The intervention is implemented in groups of 15 children. Children are eligible for this indicated version of the program when they experience internalizing, externalizing behavior problems, or both, and problematic social interactions in school. Similar to the universal implementation, a trained psychologist licensed to provide Topper Training implements this program version in a mental healthcare facility (i.e., outpatient care).

Statistical analyses

As not all participants had a complete posttest measurement, we assessed missingness at posttest using Little’s MCAR test. This analysis indicated that missing data at posttest were completely random (χ² [111] = 0.000, p = 1.000). Differences between conditions at pretest were assessed using analyses of variance (ANOVAs), and we assessed if within-condition changes were significant using paired t-tests. The difference between pretest-posttest changes in conditions was analyzed using analyses of covariance (ANCOVAs) with condition (i.e., universal condition, indicated condition, and control condition) as the between-group factor and time (i.e., change from pretest to posttest) as the within-group factor. Effect sizes (Cohen’s d) were calculated by dividing pretest-posttest changes of the experimental groups (i.e., universal condition or indicated condition) and the control group by the pooled standard deviation of the experimental group and control group at pretest (Morris, 2008). A d value of 0.20 is interpreted as a small effect, a value of .50 is interpreted as a medium effect, and a value of 0.80 is interpreted as a large effect (Cohen, 1988).

Results

Preliminary analyses

Table 1 presents the means and standard deviations by conditions at pretest and posttest. The ANOVAs did not reveal pretest differences on any outcome variables between the two intervention groups (universal and indicated condition), indicating that matching was successful. However, children in the universal condition were significantly younger (M = 9.39, SD = 1.56) compared to children in the indicated condition [M = 9.80, SD = 1.64, t (294) = −2.256, p < .05], and the control condition [M = 10.07, SD = 1.34, t (287.574) = 4.079, p < .001]. Also, the control group contained significantly fewer boys (n = 49, 33%) compared to the universal (n = 73, 49.3%) and indicated condition [n = 78, 52.7%; χ² (2) = 13.120, p < .01].

Table 1. Means and standard deviations at pretest and posttest.

	Universal implementation condition						Indicated implementation condition						Control condition
		Pretest		Posttest				Pretest		Posttest				Pretest		Posttest
	n	M	SD	M	SD	p	n	M	SD	M	SD	p	n	M	SD	M	SD	p
Well-being at school (SAQ)
School enjoyment	138	42.00	6.19	43.11	5.27	<.05	47	41.64	6.29	42.49	6.47	.330	127	44.15	4.38	43.72	5.35	.308
Social acceptance	138	41.67	6.47	43.47	5.21	<.001	47	41.70	5.86	42.49	6.14	.373	127	43.24	6.27	42.79	6.34	.287
Relationship with teacher	138	42.70	5.34	43.58	4.69	.060	47	42.43	6.04	43.06	5.32	.439	127	44.45	4.55	43.37	5.52	<.05
Overall well-being at school	138	126.38	15.18	130.16	12.79	<.001	47	125.77	14.71	128.04	13.98	.246	127	131.83	12.45	129.88	13.50	.051
Psychosocial adjustment (TQ)
Prosocial behavior	127	34.26	4.08	35.70	3.74	<.001	55	34.34	3.97	36.03	4.01	<.01	139	35.28	3.50	35.33	3.73	.886
Depressed mood	127	16.76	4.40	14.53	3.67	<.001	55	17.41	4.69	14.60	3.87	<.01	139	16.14	4.42	15.19	4.28	<.01
Disruptive behavior	127	13.29	3.38	12.92	3.08	.093	55	13.68	4.19	12.95	3.58	.246	139	12.84	2.55	13.06	2.83	.278
Self-esteem	127	33.79	4.29	35.88	3.06	<.001	54	34.24	4.04	37.19	2.66	<.001	139	34.14	3.98	34.93	4.28	<.05
Narcissism	127	20.39	5.33	20.38	5.10	.985	54	21.19	5.48	21.71	4.87	.462	139	19.90	5.89	19.16	5.22	.100
Negative intentions	122	11.24	1.98	11.10	2.33	.489	54	11.63	3.73	10.69	1.83	.092	137	10.09	0.52	10.99	1.91	<.001

SAQ: School Attitude Questionnaire; TQ: Topper Questionnaire.

In addition, analyses indicated pretest differences between the control condition and the intervention conditions on several variables. On school enjoyment (p < .05), children in the control condition scored significantly higher compared to children in the universal condition and the indicated condition. There was no significant difference between the universal and indicated condition. In relationship with the teacher (p < .01), children in the control condition scored significantly higher compared to children in the universal condition and the indicated condition. There was no significant difference between the universal and indicated condition. On well-being at school (p < .01), children in the control condition scored significantly higher compared to children in the universal condition and the indicated condition. There was no significant difference between the universal and indicated condition. There were no pretest differences between conditions on social acceptance.

On prosocial behavior (p < .05), children in the control condition scored significantly higher compared to children in the indicated condition, but not the universal condition. There was no significant difference between the universal and indicated condition. On disruptive behavior (p < .05), children in the control condition scored significantly lower compared to children in the indicated condition, but not the universal condition. There was no significant difference between the universal and indicated condition. Finally, on negative intentions (p < .001), children in the control condition scored significantly lower compared to children in the universal condition and the indicated condition. There was no significant difference between the universal and indicated condition. There were no pretest differences between conditions for depressed mood, self-esteem, and narcissism. Full statistics are presented in Supplementary Appendix A, Table A1. To handle differences between groups, we controlled for age, sex assigned at birth, and pretest scores on all outcome variables in all analyses.

Table 2 presents the bivariate correlations between outcome variables at pretest and posttest separately. All correlations were in the expected direction. Correlations between subscales measuring school climate ranged from moderate to strong (rs from 0.427 to 0.867 at the pretest and from 0.311 to .870 at the posttest). Correlations between subscales measuring psychosocial adjustment were weak to moderate (rs from 0.017 to 0.677 at the pretest and from 0.001 to 0.542 at the posttest). There was a weak, but significant correlation between children’s disruptive behavior and depressed mood at pretest (r = 0.282, p < .01) and posttest (r = 0.193, p < .01).

Table 2. Correlations at pretest (above diagonal) and posttest (below diagonal) between outcome variables.

	1.	2.	3.	4.	5.	6.	7.	8.	9.	10.
1. School enjoyment	–	.516**	.652**	.867**	.200**	−.261**	−.279**	.125**	.013	−.130**
2. Social acceptance	.470**	–	.427**	.804**	.151**	−.473**	−.095*	.259**	.010	−.033
3. Relationship with teacher	.640**	.311**	–	.815**	.158**	−.190**	−.235**	.133**	−.075	−.140**
4. Overall well-being at school	.870**	.756**	.790**	–	.205**	−.383**	−.240**	.213**	−.018	−.118*
5. Prosocial behavior	.180**	.251**	.176**	.252**	–	−.332**	−.379**	.410**	.192**	−.199**
6. Depressed mood	−.293**	−.584**	−.109	−.417**	−.317**	–	.282**	−.677**	−.017	.211*
7. Disruptive behavior	−.332**	−.147*	−.472**	−.367**	−323**	.193**	–	−.246**	.137**	.562**
8. Self-esteem	.129*	.263**	.080	.198**	.287**	−.542**	−.146**	–	.192**	−.118*
9. Narcissism	−.031	.053	−.111	−.032	.142*	.001	.070	.185**	–	.121*
10. Negative intentions	−.222**	.014	−.238**	−.179**	−.153**	.069	.349**	−.068	.132*	–

Note. Grayscale correlations are correlations between subscales from the same instrument. *p < .05, **p < .01.

Role of implementation setting: school climate and psychosocial adjustment

Table 1 shows whether within-group changes from pretest to posttest were significant. Results from the ANCOVA analyses are presented in Table 3. A visual presentation of the effect sizes for the changes in school climate and psychosocial adjustment in the universal and the indicated condition compared to the control condition is available in Figure 2.

Figure 2. Effect sizes (Cohen’s d) for pretest-posttest changes in the experimental groups. Note. Universal implementation condition indicated with striped bars; indicated implementation condition indicated with dotted bars. Asterisks indicate significant differences between the experimental group and the control group.

Table 3. Changes in the universal intervention condition and the indicated intervention condition compared to the control condition and each other.

	Universal vs. control^a			Indicated vs. control^a			Universal vs. indicated^a
	B	SE	d	B	SE	d	B	SE	F (df1, df2)	p
School climate (SAQ)
School enjoyment	.56	.76	.28	.25	.80	.26	−.31	.76	.43 (2, 312)	.65
Social acceptance	1.77**	.62	.35	.83	.82	.20	−.94	.78	4.10 (2, 312)	<.05
Relationship with teacher	1.31*	.62	.39	.83	.82	.34	−.48	.78	2.24 (2, 312)	.11
Overall well-being at school	3.64*	1.46	.41	1.91	1.92	.32	−1.73	1.83	3.13 (2, 312)	<.05
Psychosocial adjustment (TQ)
Prosocial behavior	.68	.47	.37	.92	.58	.45	.24	.57	1.61 (2, 321)	.20
Depressed mood	−1.08*	.46	.27	−1.33*	.58	.41	−.24	.67	3.72 (2, 321)	<.05
Disruptive behavior	−.32	.32	.20	−.38	.41	.31	−.06	.39	.63 (2, 231)	.53
Self-esteem	1.29**	.44	.31	2.19***	.55	.54	.90	.54	8.80 (2, 320)	<.001
Narcissism	.46	.58	.13	1.43^#	.73	.22	.97	.71	1.90 (2, 320)	.15
Negative intentions	−.38	.27	.74	−.77*	34	.91	−.39	.33	2.67 (2, 313)	.07

SAQ: School Attitude Questionnaire; TQ: Topper Questionnaire; B: mean difference between conditions; d: Cohen’s d effect size.

^#p < .08, *p < .05, **p < .01, ***p < .001.

^aSuperscripts indicate the reference group.

School climate

Children’s school enjoyment (d = 0.28), perceived social acceptance (d = 0.35), and overall well-being at school (d = 0.41) increased significantly from pretest to posttest in the universal condition. These increases were significantly larger compared to the reported changes in the control condition. These effect sizes are interpreted as small effects. There was a trend-significant improvement (p = .06) in children’s relationship with the teacher (d = 0.39) in the universal condition, and this improvement was bigger than in the control condition. This effect size is interpreted as a small effect.

In the indicated condition, school enjoyment, social acceptance, the relationship with the teacher, and overall well-being at school did not significantly change. There were no significant differences between the indicated and the control condition. In the control condition, children reported a significant deterioration in their relationship with the teacher. There were no significant changes in school enjoyment, social acceptance, and overall well-being at school in this condition.

Psychosocial adjustment

In the universal condition, children’s prosocial behavior, depressed mood, and self-esteem significantly improved from pretest to posttest. The improvements in depressed mood (d = 0.27) and self-esteem (d = 0.31)—but not prosocial behavior (d = 0.37)—were significantly larger than in the control condition. These effect sizes are interpreted as small effects. Children’s disruptive behavior, narcissism, and negative intentions did not significantly change over time.

In the indicated condition, children’s prosocial behavior, depressed mood, and self-esteem significantly improved from pretest to posttest. Children’s disruptive behavior, narcissism, and negative intentions did not significantly change over time in the indicated intervention condition. The improvements in depressed mood (d = 0.41) and self-esteem (d = 0.55) were significantly larger compared to changes in the control condition. However, the improvement in children’s prosocial behavior (d = 0.58) was not. There was also a significant difference between the indicated condition and the control group regarding negative intentions (d = 0.91). Children’s negative intentions increased in the control group, whereas these non-significantly decreased in the indicated condition. The effect size for depressed mood is interpreted as a small effect, for self-esteem as a medium effect, and negative intentions as a large effect.

In the control condition, children’s depressed mood and self-esteem significantly improved. Children’s negative intentions significantly deteriorated in this condition. There were no significant changes in children’s prosocial behavior, disruptive behavior, and narcissism.

Differences between the universal and indicated conditions

Despite the significant changes in the universal condition and the non-significant changes in the indicated condition, these conditions did not significantly differ in changes in school climate. There were also no significant differences between the changes in the universal condition and the indicated condition in children’s psychosocial adjustment.

Discussion

Previous studies on the effects of social-emotional learning (SEL) programs have often overlooked the role of implementation settings. To fill this gap, we examined to what extent either a universal or indicated implementation of SEL intervention Topper Training would yield differential effects on children’s school climate and psychosocial adjustment. In line with others (Vliek, 2015; Vliek et al., 2014, 2019), we found that SEL intervention Topper Training effectively improved children’s school climate perceptions and psychosocial adjustment.

Compared to children who did not receive the intervention, children improved in social acceptance, relationship with the teacher, and overall well-being at school after the universal intervention. Children in the indicated intervention did not significantly improve school climate outcomes. Children in both the universal intervention and the indicated intervention improved in depressed mood and self-esteem compared to children who did not receive the intervention, but effect sizes for psychosocial adjustment were larger in the indicated intervention. These findings suggest that a universal intervention approach yields effects on children’s perceptions of school climate, whereas an indicated intervention approach does not. Furthermore, based on our results we conclude, that both indicated interventions and universal interventions yield effects on psychosocial adjustment, as both improved depressed mood and self-esteem. However, since effect sizes for improvement in psychosocial adjustment were larger for the indicated intervention it may be that this setting is more effective in strengthening psychosocial variables, such as depressed mood and self-esteem. To our knowledge, this study is the first to provide preliminary evidence of differential effects of interventions based on their implementation setting.

Impact of implementation setting on SEL intervention effects

Our findings are partly in line with our hypotheses. Larger effect sizes were found for children’s psychosocial adjustment in the indicated intervention. The indicated intervention did not seem effective in improving children’s thoughts and feelings elicited by the school’s environment, whereas the universal intervention was. On the one hand, these findings could imply that universally implemented SEL interventions are most effective in improving children’s interpersonal behavior and perceptions of themselves in connection with others and indicated implementation of SEL interventions may be most effective in improving children’s intrapersonal psychosocial adjustment.

To substantiate this prudent conclusion, taking a close look at the instruments used in this study is helpful. The effects in the universal setting were largest on school climate perceptions, as measured by the Dutch School Attitude Questionnaire (Smits & Vorst, 1990). This measure includes items, such as “I often feel lonely in the classroom,” “I get along well with my teacher,” and “I enjoy going to school”. Together, these items reflect aspects of the classroom peer ecology and children’s perception of their school as an enjoyable place. Even though the universal intervention also had positive effects on children’s psychosocial adjustment, the largest effects were found on children’s psychosocial adjustment in the indicated setting. Psychosocial adjustment was measured using the Topper Questionnaire, including items, such as “I want to make fun of others”, “I dislike myself”, and “I want to be honest in school”, Thus, the School Attitude Questionnaire reflects experiences in interactions with others (interpersonal) while the Topper Questionnaire reflects behavioral tendencies that relate to a child’s inner world (intrapersonal).

A possible explanation for these findings may be that the universal intervention was implemented in classrooms with substantial problems regarding class climate. This means there may have been more to gain from the intervention in school climate in the universal condition, compared to the indicated condition. As children in the indicated condition were grouped with children who likely attended different schools, it may be more difficult to sort effects on school climate. Influencing children’s thoughts and feelings elicited by their school context may likely work best within that context. Teachers could play a role in this mechanism. In the universal condition, teachers are actively involved in the intervention, which may positively influence the generalization of more adaptive thoughts, emotions, and behaviors into the everyday school context in the universal condition. For example, positive feedback by teachers reinforces positive child behavior (e.g., Trussell et al., 2016). As teachers were not part of the indicated intervention and no data about teachers was collected, it is not possible to tease apart teachers’ influence on the current findings.

The finding that the indicated intervention yielded the strongest effects on children’s psychosocial adjustment may be explained by the dose of delivery of the intervention in an indicated setting. Previous research into the effective components of SEL interventions (de Mooij et al., 2020) found evidence to suggest that there is an optimal dose in which components should be implemented; when intervention dosage is too low, SEL interventions produce non-significant or small effects. Due to the generally smaller group size in indicated intervention settings, trainers do not have to divide their attention across many children. Consequently, children may have more opportunities to actively participate in exercises and receive feedback during the session. This could mean that the actual dosage of intervention components children receive in an indicated setting is higher than in a universal setting, generating more substantial changes in children’s psychosocial adjustment. Still, another possibility is that children in indicated implementation settings are more open to practicing alternative behavior because they are outside their regular norm group. It is important to note that children in both implementation settings were matched on their baseline adjustment—thus a stronger effect on children’s psychosocial adjustment in an indicated setting cannot be explained by higher pre-existing levels of adjustment problems in this group.

On the other hand, our findings also suggest that both universal and indicated interventions are effective in improving children’s psychosocial adjustment, as children’s depressed mood, self-esteem, and prosocial behavior improved in both interventions—even though the improvement was larger (i.e., larger effect sizes) in the indicated intervention. One of the core aims of the intervention is to instill in children a sense of self-value. Children are taught to be equally respectful to others as to themselves, as an imbalance between these two is expected to lead to low self-esteem (Salmivalli et al., 1999; Thomaes et al., 2009; Topper Training Foundation, 2007–2023). Learning to behave more authentically and valuing themselves likely increases children’s self-esteem, and decreases children’s depressed mood (Ryan & Deci, 2000).

The larger effect sizes for depressed mood and self-esteem in the indicated setting may be explained by the more personal and safer environment of the indicated setting. It may be that children in the indicated setting get more personal attention, as these groups are smaller, and consequently have more opportunities to improve on these outcome variables. It may also be that an indicated setting provides a safer environment, which is known to be an important condition for children to grow in self-esteem.

A noteworthy finding was that, in contrast to findings from previous work on Topper Training (Vliek, 2015), children’s disruptive behavior neither improved in the universal nor in the indicated intervention. This finding is likely explained by our dependence on self-report data, whereas the original trials also included parent- and teacher-report data. A possible explanation could be that children’s disruptive behavior only changes after more substantial and enduring changes in school climate and peer interactions have been established (Boutin et al., 2020). In our study, children’s perception of the school climate did not improve in the indicated intervention and only moderately (d = .28–.41) in the universal condition. Also, children’s disruptive behavior can be persistent and resistant to change. Perhaps, then, a combination of universal and indicated intervention elements is needed to have a substantial impact on children’s disruptive behavior. Indeed, previous research has provided support for the benefit of nesting an indicated intervention within a universal intervention in this regard (Lochman & Wells, 2002).

Another noteworthy finding was that children in the control group reported a deterioration in their relationship with the teacher from pretest to posttest. Considering the importance of a positive child-teacher relationship for children at risk of problem behavior, it is worrisome that our findings show that this relationship deteriorates without intervention. Studies have shown that the relationship between teacher and child predicts children’s behavioral adaptation (both in the internalizing and externalizing domain). A warm teacher-child relationship provides children with security and a scaffold for children’s positive development in the school environment (O’Connor et al., 2011). Our study thus shows that it is important to monitor the child-teacher relationship, especially for children presenting with mild to severe psychosocial problems, and—on the positive side—demonstrates that universally implemented SEL interventions can help to improve this relationship.

Limitations, strengths, and future directions

Some limitations warrant a cautious interpretation of our findings. First, our dataset did not include information on implementation fidelity, which is an important confounder for the assessment of intervention effects. We should therefore consider that, at least in part, differences in intervention effect might be due to differences in diligence to adhere to the intervention protocol.

Second, we could only use child-report data from original studies. This is important considering that most children are referred to SEL interventions by their teacher or parents, who may have viewed children’s behavior as more problematic. Previous meta-analytic research has shown that correspondence between parent- or teacher-reported and child-reported problem behavior is low (De Los Reyes et al., 2015). Therefore, had we conducted multi-informant instead of uni-informant analyses on children’s psychosocial adjustment, this may have yielded different results.

Third, and perhaps most importantly, differences in implementation setting across conditions were confounded with design. Data in the universal condition was collected using a quasi-experimental design and data from the indicated condition was collected using a randomized trial. Additionally, in the indicated intervention parents elected their children to participate in the intervention, whereas, in the universal condition, teachers registered for participation, which may also cause bias as parents and teachers may have different reasons for registering a child for an intervention. However, there were no differences between the universal and indicated intervention conditions on school climate or psychosocial adjustment at pretest, which suggests that children had comparable levels of problem behavior at the start of the intervention. Although propensity score matching removes selection bias in observational data (Pan & Bai, 2015), we cannot fully exclude the possibility that design was a confounder in this study.

Also, despite a stringent matching procedure, some pretest differences between the conditions remained. However, these differences were predominantly between the control group and the intervention groups, not between the intervention groups. Overall, children in the control group reported more positive thoughts and feelings about their school context, which in turn may explain the higher scores on prosocial behavior and lower scores on disruptive behavior and negative intentions. Possibly, a confounding factor here is the overall school climate, which was not measured in the original trials. To limit the influence of pre-existing differences between conditions, we controlled for pretest differences between the groups in our analyses. Thus, we are confident these did not significantly influence our findings. Even so, statistically controlling for differences between groups does not fully eliminate the impact of confounding variables, and as such the findings of this study should be considered in light of these limitations.

This study also holds several strengths. To account for the lack of (full) randomization of participants, we used a stringent matching procedure using the indicated intervention as the reference group. We selected children for all conditions with psychosocial adjustment levels that warranted participation in an indicated social-emotional learning intervention. This procedure, in which all outcome variables were used as covariates to determine propensity scores, increases the likelihood that children selected for the analyses reported comparable levels of (mal)adjustment at the pretest. Another strength of this study is that children were recruited from a community sample and the intervention was implemented in a real-world context, which increases the generalizability of our findings. Moreover, we assessed changes in the child’s experience across a broad array of measures, related to both their daily social school climate and psychosocial adjustment, as opposed to only assessing children’s changes in social-emotional skills. This is a strength as research often overlooks the importance of the social climate (Trach et al., 2018).

We advise future studies in this direction to use a multi-informant approach, to gain a more complete picture of children’s social functioning. Importantly, such a multi-informant approach should also include peer-reported data. It is vital to collect data from peers to learn how children with problematic social behavior are judged by others after a social-emotional learning intervention and if their peers also assess the school climate to be improved. To be able to conclude that an intervention is effective in improving the school climate, it seems important to collect data from all those that make up this climate. A way to do this may be by assessing children’s social-emotional behavior changes in the context of the wider social network using sociometric or SIENA analyses. This may provide more insight into the exact changes in interpersonal mechanisms driven by SEL interventions.

Concerning future research endeavors it is also important to emphasize that in our current study, we used data from Topper Training as a case SEL intervention. We did so because we had data for Topper Training from a single intervention paradigm across universal and indicated implementation settings. Thus, a big advantage of our approach was that we were certain that differences in intervention program content would not confound our findings. Still, it is necessary that this study—to our knowledge, the first to make an active comparison between the effects of universal and indicated implementation of SEL interventions—is repeated using data from other childhood SEL interventions collected in a single trial to substantiate our conclusions.

Main conclusions

An important takeaway from this study is that the setting in which an intervention is implemented may influence its effectiveness. when implementing a SEL intervention, one should be clear about what one hopes to gain. If the rationale for implementing an intervention is to improve children’s interpersonal behavior at school, such as school enjoyment, teacher-child relationship, and perceived acceptance by others, universal interventions that focus on interactions between children might be most suitable. Such intervention may include exercises like giving each other feedback, roleplays, and exercises that provide children insight into (maladaptive) group dynamics. However, if the goal is to improve individual children’s psychosocial behavior, indicated interventions might have larger effects. Such interventions may include exercises that focus on children’s inner world of belief. Even though we cannot draw strong conclusions based on the current study, our findings do suggest that practitioners may want to carefully align their implementation setting with the child behavior they aim to change when selecting a SEL intervention program. In light of cost-effectiveness, this may be an important insight. Universal interventions are generally cheaper than indicated interventions (e.g., Lee et al., 2017) and can serve more children at once. However, implementing interventions in the wrong setting may, in the end, be more expensive. Our findings may inform intervention developers and professionals implementing these interventions, thereby improving child mental health care and working toward all children’s positive development.

Disclosure statement

LV was employed as a psychologist and researcher by the Topper Training Foundation at the time of writing but was not involved in data analysis, and therefore there is no conflict of interest. None of the other authors declare a conflict of interest.

Data availability statement

The data that support the findings of this study are available from the corresponding author, BM, upon reasonable request.

Additional information

Funding

This work was supported by ZonMw under grant “Effectief werken in de Jeugdsector” [number 729300011].