Testing an intervention to stimulate early adolescents’ news literacy application in the Netherlands: A classroom experiment

ABSTRACT

In a time of blurring lines between online content, early adolescents’ (12–15 years old) lack of critical engagement with news is problematic. Therefore, we need more effective interventions to empower their news literacy. Interventions should go beyond only informing on the news production process and focus on individual’s news literacy application in everyday life. This study tests an intervention based on (news) media literacy and motivational theory, inspired by the needs and preferences of early adolescents themselves, to stimulate news literacy elements (news literacy self-efficacy, value for media literacy, motivation, and social norms) and news literacy application (news consumption, news analysis, and news evaluation). The intervention consists of three lessons in which early adolescents (N = 258) learn about the news production process, combined with a learning-by-doing approach: writing and checking news articles. Based on a between-subjects waitlist experiment, the intervention was effective in increasing participants’ news literacy self-efficacy, but not (yet) in stimulating other news literacy elements and news literacy application. Nevertheless, this experiment offers valuable insights on the development of news literacy interventions.

Impact Summary

Prior State of Knowledge: Early adolescents encounter lots of (online) news daily. However, they have great difficulty with deciding on its trustworthiness, accuracy, and relevance. There is a need for educational interventions that stimulate their everyday critical engagement with news: their news literacy application.

Novel Contributions: The study presents an intervention founded in (news literacy) theory and co-creation. Early adolescents gained self-efficacy through writing and checking news articles, combined with information, and incorporating their social context. However, they did not yet increase in news literacy application.

Practical Implications: Using learning-by-doing approaches to stimulate news literacy application can be complex. More attention to testing clear goals in media education, and to other factors that influence early adolescents’ learning, such as intervention length, feedback, and the social context, is needed.

KEYWORDS:

• News literacy
• news literacy application
• early adolescents
• intervention
• experiment

As eager internet users, early adolescents (12–15 years old) encounter an abundance of information daily (Smahel et al., 2020; Van Damme et al., 2022). Yet, one in three 13-year-olds in Europe lacks the ability to evaluate information. Moreover, only slightly more than half of the 15-year-old students in the European Union report receiving instruction on recognizing biased or subjective information (European Commission, Directorate-General for Education, Youth, Sport and Culture, [European Commission], 2022; Suarez-Alvarez, 2021). This is problematic particularly when information comes in the form of news, because news has democratic value (McCombs et al., 2011) and plays an integral part in individual’s every (social) lives (Swart, 2021). The fast expansion of news – nowadays produced and distributed not only by professionals but by everyone – makes it increasingly difficult to disentangle credible from less credible news and to find relevant and high-quality news (e.g., Vraga et al., 2021). It is thus necessary, possibly more than ever, to support early adolescents in engaging with (online) news. Or, more specifically, to stimulate their news literacy, that is, the “knowledge around personal and social processes by which news is produced, distributed and consumed, and skills that allow users some control over these processes” (Tully et al., 2022, p. 5).

News literacy is a type of media literacy, that broadly aims to stimulate individuals to better understand and assess mediated information, ranging from advertising to digital content (Potter, 2004). When early adolescents are news literate, they should understand the news production process and be able to access, analyze, and evaluate the content they encounter (Ashley et al., 2013). Based on their news literacy knowledge and skills, they are expected to apply their news literacy, such as by identifying misinformation and verifying news (Vraga et al., 2021). However, there is less evidence for this application in practice (Tamboer et al., 2022, 2023; Vraga & Tully, 2021). A major caveat in the literature and intervention practice is that little work discusses how or when news literacy is applied in youth’s daily lives (Swart, 2021). Elements that can inhibit the translation from knowledge and skills to news literacy application could be (perceived) unfavorable existing social norms and attitudes, perceived behavioral control (Swart, 2021; Vraga et al., 2021), little motivation, or value for media literacy (Tamboer et al., 2022, 2023). To stimulate news literacy application, there is a great need for initiatives that do not only target early adolescents’ knowledge and skills, but incorporate relevant elements in news literacy application – such as motivation and social norms – as well (Swart, 2021; Tamboer et al., 2023; Tully et al., 2022).

The current study aims to develop and assess the effectiveness of an intervention to empower early adolescents in their news literacy application. Intervention programs thus far are often a-theoretical or focus only on a specific implementation of news literacy, such as misinformation recognition (European, 2022; Vraga et al., 2021). Additionally, only a few interventions explicitly focused on early adolescents, and even fewer included the target groups’ preferences and views (e.g., Literat et al., 2020). Therefore, this study presents a theory-based and co-created intervention and tests its effects on essential elements of early adolescents’ news literacy elements (skills, value for media literacy, motivation, social norms) and their news literacy application (news consumption, news analysis, news evaluation).¹

Empowering early adolescents’ news literacy application

Over time, various interventions have been developed to stimulate news literacy. Existing approaches to news literacy range from experiments focused on inoculation in the form of a short news literacy statement before news consumption (e.g., Hameleers, 2022), to educational programs consisting of several news literacy lessons (Geers et al., 2020), and to (participatory) games (e.g., Literat et al., 2020; Roozenbeek & van der Linden, 2019). These interventions show that (brief) news literacy interventions can be successful in, for example, increasing the accuracy ratings of information (Hameleers, 2022). Nevertheless, these interventions do not explicitly consider or assess elements that play an essential role in everyday news literacy application, do not assess to what extent individuals engage in news literacy application outside of the intervention, and almost exclusively focus on older youth or adults.²

Early adolescents are a critical and specific target group for news literacy initiatives. For this group, most contact with news happens through their smartphones and online (Newman et al., 2019; Smahel et al., 2020; Van Damme et al., 2022). Consequently, early adolescents are often thought of as “digital natives” (Newman et al., 2019), a stereotype that is omnipresent, primarily in media research and public debate. However, the term “digital natives” is unwarranted, because it does not accurately reflect individuals’ interaction with technology (Smith et al., 2020). In fact, research shows that youth fail to classify different types of information, including real and fake news, and are confused about the quality of news and information (Notley et al., 2017; Van Damme et al., 2022). Early adolescents’ online presence does thus not warrant news literacy knowledge and skills and the application of their news literacy.³

Research has shown that early adolescents have relatively broad definitions of news, in line with the blurring of the lines within the media landscape (Tamboer et al., 2022). They only sometimes analyze news messages (e.g., reflect on the trustworthiness) and rarely evaluate news (e.g., cross-check information; Tamboer et al., 2023). This lack of news literacy application is problematic because early adolescence is a period of significant developmental changes in which various attitudes and habits are formed (Valkenburg & Piotrowski, 2017). Early adolescents develop lasting news consumption habits and political views and are highly susceptible in their political and societal interest (Russo & Stattin, 2017; York & Scholl, 2015). However, early adolescents’ self-control and planning still strongly depend on their motivation to engage in a particular behavior (Valkenburg & Piotrowski, 2017). Therefore, early adolescence is a significant period to empower individuals in their news literacy application, and initiatives should, first and foremost, succeed in motivating early adolescents.⁴

News literacy theory and early adolescents

News literacy elements

We build on theoretical insights regarding behavioral change, motivation, and news literacy to develop an intervention targeting early adolescents’ news literacy elements and application. The information-motivation-behavioral skills model (IMB-model; Fisher & Fisher, 2002), which has been proven effective in intervention development, posits that information, behavioral skills, and motivation are causally related to a certain type of behavior and should therefore be included in interventions to understand and promote this behavior (Fisher & Fisher, 2002). Translated to (news) media literacy, individuals should be knowledgeable of the (news) media system, have (news) media literacy skills, and be motivated to engage in (news) media literacy application (Potter, 2004; Rozendaal, 2017; Vraga et al., 2021). In addition, more recent research and theoretical models point to the role of the social context: this context should offer the ability to engage in (news) media literacy application (Rozendaal, 2017), or there should, at least, provide positive social norms to do so (Vraga et al., 2021).

These four theoretical cornerstones for designing an intervention – knowledge, skills, motivation, and the social context – can be translated into more concrete elements. Relevant elements in news literacy depend on the outcome and the target group (Vraga et al., 2015). Although knowledge is one of the core elements of media literacy and behavioral change theory and interventions (e.g., Fisher & Fisher, 2002; Rozendaal, 2017; Vraga et al., 2021), empirical research did not show a clear relation with news literacy application in early adolescents (Tamboer et al., 2023). The main focus in the current study is therefore on stimulating skills, motivation, and including the social context through social norms, to eventually stimulate the application of news literacy (Swart, 2021; Tamboer et al., 2023).

Early adolescents need news literacy skills, or at least feel that they can successfully apply their news literacy, i.e., have self-efficacy (Fisher & Fisher, 2002). Furthermore, early adolescents need to see the value of (news) media literacy, i.e., understand and believe in the importance of (news) media literacy (Vraga et al., 2015). Value for (news) media literacy correlates with media knowledge, current events knowledge, skepticism towards the press, self-perceived news literacy, and news literacy application (Tamboer et al., 2023; Van Damme & Van Leuven, 2019; Vraga & Tully, 2021; Vraga et al., 2015). Furthermore, early adolescents should have the motivation to apply their news literacy in their everyday news use because motivation to consume news and apply news literacy can lead to higher levels of news literacy application (Maksl et al., 2015; Tamboer et al., 2023).

Another critical element in early adolescents’ news literacy application is their social norms, especially those of peers (Tamboer et al., 2022, 2023). The application of news literacy is inherently social and connective (Swart, 2021). For youth, consuming and critically engaging with news is primarily done because of its social value. An important socializing sphere for early adolescents is the school, with their peers and teachers (Tamboer et al., 2022, 2023). Although parents are also known to influence early adolescents’ news literacy application (e.g., Edgerly et al., 2018), they are more difficult to reach through an in-class intervention. Therefore, the current study incorporates only peers and teachers as elements of social context.⁵

News literacy application

Together, several news literacy elements are expected to play a role in the extent to which individuals apply their news literacy (i.e., news literacy application; Tamboer et al., 2023). This application thus entails the extent to which individuals critically engage with news, or as Vraga et al. (2021, p. 8) write: “behaviors that occur when people engage with news content in a critical and mindful manner”. In the current study, we distinguish three forms of news literacy application: news consumption, analysis, and evaluation. Consuming news strongly correlates with news analysis and evaluation (Tamboer et al., 2023). This relationship is likely reciprocal: more news consumption can lead to more news analysis and evaluation (e.g., Swart, 2021), while more analysis and evaluation can boost (value for) news consumption (Ashley et al., 2013). In line with earlier research on early adolescents’ news literacy application (Tamboer et al., 2023), news consumption is the extent to which individuals are exposed to news. News analysis entails reflecting on news and analyzing important news criteria, such as trustworthiness and framing. News evaluation is a more active form of news literacy application, which considers the extent to which individuals evaluate sources and cross-check information (Tamboer et al., 2023).

The current study

We tested the effectiveness of an intervention that aims to not only inform, but also motivate early adolescents on both news literacy elements and news literacy application. As a first test of the effects of an intervention on these often overlooked concepts, our current hypotheses only included direct effects. We designed a between-subjects experiment, in which we tested the difference between the posttest scores (T1) of the experimental and control group, controlling for their scores on the pretest (T0). More precisely, we hypothesize: Participants in the experimental condition have a higher level of their news literacy elements; that they score higher on self-efficacy (H1), value for media literacy (H2), news literacy motivation (H3), (positive) peer social norms (H4), (positive) teacher social norms (H5) at T1 than participants in the control condition, controlling for T0. Furthermore, we expect participants in the experimental condition to show more application of news literacy; that they engage in more news consumption (H6), news analysis (H7), and news evaluation (H8) than participants in the control condition, controlling for T0.

Method

To test the hypotheses, an experiment was conducted in Dutch secondary schools from April to June 2022. Participating schools were assigned to the experimental condition (the news literacy intervention) or the passive waiting condition (receiving the intervention after a waiting period). The Netherlands is a country with historically high trust in news organizations (Newman et al., 2022). Yet, Dutch adolescents seem to consume only little news through these trusted organizations (Tamboer et al., 2023). The study was reviewed by the Ethics Committee of the Faculty of Social Sciences at Radboud University (ECSW-2021-041) and was preregistered at the Open Science Framework (https://osf.io/qt936).

The intervention

The “Bubble” intervention is an online environment that participants used during three lessons. The intervention is based on discussions with early adolescents (Tamboer et al., 2022) and was first tested to fit early adolescents’ preferences (Tamboer, 2023). In the intervention, participants (1) watch videos on the news production process and do a quiz to test their knowledge, and (2) get the assignment to make their news articles in pairs, based on a predetermined factsheet. They then (3) determine their goal (informing, entertaining, persuading, or misleading), outlet (such as a professional news outlet or social media post), their audience (such as children, teens, or adults), and other article characteristics (such as whether they mention sources or not) and write their news article based on the facts (or write a convincing fake story). Finally, (4) they read each other’s news stories and checked whether they could determine the article characteristics (like the ones they filled out for their article) based on the article itself. For all these steps, they could (5) gain points, resulting in a news literacy application ranking per class¹ (see Figure 1).

Figure 1. Overview of the bubble platform and intervention procedure.

The intervention is based on motivation theory and co-creation to optimally empower early adolescents in their news literacy elements and news literacy application. First, the intervention informs early adolescents on the news production process (Based on Vraga et al., 2021, Figure 1, Step 1) appealingly to make news literacy application more relevant to increase their motivation and confidence (e.g., Ryan & Deci, 2000). Second, participants train their news literacy skills by learning-by-doing (e.g., Dewey, 1916) – they write and check news articles – to increase their skills, confidence, and motivation (e.g., Ryan & Deci, 2000, Figure 1, Step 2–4). Third, the intervention provides a rationale for the need for news literacy application in early adolescents while staying close to the frame of reference of the target group to increase their value for (news) media literacy (e.g., Ryan & Deci, 2000; Tamboer et al., 2022; Figure 1, Step 1–2). News is defined based on the target group’s definition of news (following Edgerly & Vraga, 2020). This, for example, translates into examples of “news” content within the platform that align with the kind of content early adolescents generally consume and regard as “news” – such as stories about celebrities and other topics that are relevant to the target audience. Fourth, the intervention further incorporates motivation-enhancing elements by giving feedback on their activities within the intervention, by providing clear steps to analyze and evaluate news, and by adding competition (Kusurkar et al., 2011; Ryan & Deci, 2000; Tamboer et al., 2022; Figure 1, Step 5). Fifth, the intervention includes the social context by working in pairs to facilitate discussion (e.g., Bertucci et al., 2010), and with a role model in intervention videos (Tamboer et al., 2022; Figure 1, Step 1–3).

Participants and procedure

A priori sample size calculation was performed by using G*Power 3.1 (Faul et al., 2009). Expecting a small to medium effect size (Jeong et al., 2012, Cohen’s d =.37, f  = 0.18), we needed 245 participants for ANCOVA (two groups, alpha = .05, power 80%). A larger number of participants were recruited to account for the need for active parental consent and attrition (see Figure 2). In total, 27 schools were contacted, 16 schools did not reply after multiple reminders, 5 schools declined, and 6 schools agreed to participate. Eventually, 2 schools participated in the test phase (Tamboer, 2023) and 4 schools participated in the experiment. All participating schools gave active informed consent. Parents or legal guardians received an information letter and consent form. 33 early adolescents were excluded because they or their parents did not consent with participation.

Figure 2. CONSORT flow diagram of participants.

Schools were randomly assigned to the experimental/intervention (2 schools) or waitlist condition (2 schools). All classes within schools were in the same condition. All groups first filled out our pretest questionnaire at T0. After filling out the questionnaire, participants in the experimental group worked on the intervention platform for the remaining session time (total time ranged from 45 minutes to 1 hour, depending on the school lesson times). The second and third session lasted approximately 1 hour during which participants worked on the intervention platform and, in the final session, filled out the post-questionnaire (T1).

After filling out the pre-questionnaire at T0 supervised by their teacher(s), participants in the waitlist condition entered 2/3 weeks of passive control. After this waiting period, participants filled out the post-questionnaire (T1), and started their intervention lessons. After all lessons, these participants filled out an additional evaluation questionnaire (T2).

The initial sample consisted of 326 early adolescents. As preregistered, participants who did not complete all measures (answering only half of the questionnaire or participants who were absent at T0 or T1, n  = 51), participants with impossible answers or other indications of carelessness (e.g., answering “no education possible” instead of their educational level and giving the same answer in a long sequence/flatlining, n = 13), and participants who completed the questionnaire at T0 within 3 minutes (n = 2) were excluded. When administering the questionnaires, we realized that filling out the questionnaire for the second time (at T1) went faster than at T0. Therefore, we excluded participants who completed the questionnaire at T1 within 2 instead of 3 minutes (n = 2).

The final sample consisted of 258 participants (52.3% girls, 1.2% other) between 11 and 16 years old (M = 13.89, SD = 1.02, 253 participants between 12 and 15 years old). Most participants attended pre-university secondary education (VWO, 60.9%), followed by higher general secondary education (HAVO, 26.4%) and pre-vocational secondary education (VMBO, 12.8%)². Most participants were in their third year of secondary education (65.9%), followed by participants in their first (21.7%) and second year (12.4%).

Measures

Several questions were asked to measure the dependent variables, demographic characteristics, and intervention evaluation. The complete surveys can be found in the OSF project (osf.io/axz6f). All final scales were pretested with early adolescents (2 classes), had sufficient scores on the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy, a significant Bartlett’s test of sphericity, and a determinant score indicating the absence of multicollinearity.

News literacy elements

Self-efficacy

In line with other intervention models (e.g., John et al., 2017) and earlier research (Tamboer et al., 2023), we used self-efficacy as a proxy for behavioral skills. The self-efficacy measure was based on guidelines by Bandura (2006) and on the definition of news literacy by Craft et al. (2016), and tested before in Tamboer et al. (2023). An example item was “I am able to find news that is trustworthy.” Answers ranged from 1 = not at all true to 6 = completely true. Both for the pre- and posttest, PCA of the six items showed 1 factor with an eigenvalue larger than 1 (factor loadings ≥ .55) and Cronbach’s Alpha ≥ .78. For both timepoints, we calculated a mean score.

Value for media literacy

Value for media literacy was measured by an adapted version of a scale developed by Vraga et al. (2015), based on Tamboer et al. (2023). An example item was “Engaging with news in a critical manner is important in a democracy.” The scale ranged from 1 = not at all true to 6 = completely true. For the pretest, a PCA of the five items showed 1 factor with an eigenvalue larger than 1 (factor loadings ≥ .41), with a Cronbach’s Alpha of .69. For the posttest, a PCA of the five items showed 2 factors with an eigenvalue larger than 1. Inspection of the factor loadings did not indicate an informative 2 factor solution. When forced on 1 factor, the value for media literacy scale has factor loadings ≥ .43 and an acceptable Alpha (.68). We calculated mean scores for the pre- and posttest

Motivation

News literacy application motivation was measured with three items based on Maksl et al. (2015), Van Damme and Van Leuven (2019), and Tully et al. (2022). An example items was “I critically look at news because it is beneficial for me.” The scale ranged from 1 = not at all true to 6 = completely true. For the pre- and posttest, a PCA indicated one factor with an eigenvalue larger than 1 (factor loadings ≥ .81), with a good Alpha (≥.81). We calculated mean scores for the pre- and posttest.

Social norms: Peers

The measure for peer norms was based on the measure by Tamboer et al. (2023), which was roughly based on earlier work by Edgerly et al. (2018). An example item was “My peers believe it is important to know what is going on in the world.” The four statements were answered on a scale from 1 = not at all true to 6 = completely true. For both the pre- and posttest, the PCA yielded one factor with an eigenvalue larger than one (factor loadings ≥ .73) and good Alpha’s (≥.83). We calculated mean scores for the pre- and posttest.

Social norms: Teachers

Teacher norms were measured in line with peer norms using the same 6-point scale. For both the pre- and posttest, the PCA yielded one factor with an eigenvalue larger than one (factor loadings ≥ .76) and good Alpha’s (α ≥.82). We calculated mean scores for the pre- and posttest.

News literacy application

News consumption

For news consumption, participants indicated how often they consume news per medium (print, TV, radio, news via the mobile application/website of news organizations, social media news, other people) ranging from 0 = zero days per week to 7 = seven days per week. We calculated a total score, counting all different types of news consumption into one measure of news consumption.

News analysis

We used the measure developed in Tamboer et al. (2023) to measure news analysis. An example question was “How often do you reflect on whether a news message is trustworthy?.” Participants answered on a 6-point Likert scale (1 = never to 6 = always). For both the pre- and posttest, the PCA of four items yielded one factor with an eigenvalue larger than one (factor loadings ≥ .76) and good Alpha’s (≥.82). We calculated mean scores for the pre- and posttest.

News evaluation

In line with news analysis, we used the measure developed in Tamboer et al. (2023) to measure news evaluation. An example question was “How often do you check whether you can also find that news somewhere else?.” Participants answered the four questions on a 6-point Likert scale (1 = never to 6 = always). For both the pre- and posttest, the PCA yielded one factor with an eigenvalue larger than one (factor loadings ≥ .72) and good Alpha’s (≥.79). We calculated mean scores for the pre- and posttest.

Demographics

On the pretest, participants indicated their gender by choosing 0 = male, 1 = female, or 2 = other/would rather not say, filled out their age, and indicated their educational level² and school year.

Intervention evaluation

In the control condition, participants completed an intervention evaluation questionnaire after participating in the lessons (T2). In this questionnaire they were asked about how much they liked the lessons, how interesting they were, how important they believe the lessons were, and whether they think they will change something in how they engage with news – i.e., expected news literacy application. There was also space to elaborate on what they believe could be improved.

Strategy of analysis

The data were analyzed in R (RStudio version 2022.07.2+576; RStudio Team, 2022). First, we conducted preparatory analyses. As preregistered, we assessed the clustering of the data by examining the intraclass correlation coefficient (ICC), and adjustments were made where necessary to account for the hierarchical structure of the data. We then conducted a Fisher’s exact test (gender), a one-way independent t-test (age), and a chi-square test (educational level) to check whether randomization across conditions was successful. Second, we generated descriptive statistics to better understand and describe our sample. Third, we conducted our main analyses. Based on news literacy theory and previous research, gender, age, and educational level were included as covariates (Kleemans & Eggink, 2016; Maksl et al., 2015; Tamboer et al., 2023). We performed mixed-effects models using the lme4 package (Bates et al., 2015) and the lmerTest package (Kuznetsova et al., 2017). Based on the ICC, we added a random intercept for class. The models on value for media literacy, teacher norms, and news consumption led to warnings, after which the same model was fitted using a Bayesian approach with brms (Bürkner, 2017). As robustness checks, all models were also tested using a Bayesian approach and tested without influential classes and outliers. All checks led to the same patterns of results. Finally, descriptive statistics were generated for the intervention evaluations.

Results

Preparatory analyses

Hierarchical structure of the data

Because participants were nested in classes and in schools, we first tested for the most appropriate random effects structure for the data. Because there were only four schools and because there is a strong correlation between school and condition³, the difference in baseline scores between schools was checked. For all schools, the interpretation of these scores was similar. Therefore, we proceeded with testing the variance in outcome variables explained by class to an intercept-only model. Looking at the intraclass correlations (ICC) and chi-square likelihood tests, we found that for news literacy motivation (χ²_change (1) = 4.26, p = .039, ICC = .06) and news literacy analysis (χ²_change (1) = 6.91, p = .009, ICC = .09) adding a random intercept per class significantly increased the variance explained in the dependent variables. Because there was evidence of variation depending on class, we proceeded with testing mixed-effects models for all hypotheses, including a random intercept per class.

Randomization check

The two conditions were comparable in terms of gender (Fisher’s exact test, p = .692), but differed in terms of age (t(256) = −19.14, p  < .001) and educational level (χ²(2) = 21.41, p < .001). Participants in the experimental condition were somewhat older (M = 14.45, SD = 0.59) than participants in the control condition (M = 12.81, SD = 0.77). Furthermore, participants in the experimental condition more often attended pre-vocational secondary education or pre-university secondary education than participants in the control condition¹. The differences between the conditions were accounted for in the subsequent analyses, because age and educational level were already included as covariates.

Descriptive statistics

At baseline (T0), participants’ news literacy factors (self-efficacy, value for media literacy, news literacy motivation, and peer and teacher norms) range between 3.50 and 4.81 on a 6-point scale (see Table 1). Participants often agree with statements on the importance of news media literacy, showing value for media literacy. On average, they feel that they can engage in news literacy application, scoring “agree a little” to “agree” on self-efficacy. Their motivation to engage in news literacy application is somewhat lower; participants generally score neutral on motivation. Regarding social norms, participants mostly feel like their teachers believe news literacy application is important, while they believe their peers are neutral.

Table 1. Descriptive statistics for news literacy elements and news literacy application at baseline (T0).

Measurement	Mean (Standard Deviation)	Minimum-Maximum
Self-Efficacy	4.43 (0.65)	2.33–6.00
Value for Media Literacy	4.56 (0.68)	1.60–6.00
Motivation	3.52 (1.10)	1.00–6.00
Social Norms: Peers	3.50 (0.92)	1.00–6.00
Social Norms: Teachers	4.81 (0.71)	1.75–6.00
News Consumption	14.80 (7.32)	0.00–36.00
Newspaper	0.57 (1.23)	0.00–7.00
Television News	2.58 (2.15)	0.00–7.00
Radio News	2.29 (2.37)	0.00–7.00
App/Website News Organization	1.79 (2.26)	0.00–7.00
Social Media News Organization	3.47 (2.55)	0.00–7.00
Other People	4.07 (1.99)	0.00–7.00
News Literacy Analysis	2.73 (0.97)	1.00–6.00
News Literacy Evaluation	2.30 (0.91)	1.00–5.25

Note. N = 258, all scales are 1 (never) to 6 (always), except for total news consumption, which ranges from 0 (no news consumption) to 36 (most news consumption, through various channels), and news consumption per medium, which ranges from 0 to 7 (days a week).

Regarding participants’ news literacy application, participants rarely read (offline) newspapers. They get most news, around 4 days a week, through other people, such as their family, friends, and school, followed by news via news organizations through social media. Participants indicate to engage in news literacy application rarely, answering that they “almost never” to “sometimes” analyze news, and evaluate news even less.

Main analyses

In line with H1, participants in the experimental condition (covariate-adjusted M = 4.64, SE = 0.10) reported more self-efficacy at T1 than participants in the control condition (covariate-adjusted M = 4.27, SE = 0.08, all results see Table 2, see online supplemental materials Appendix A, B, C). There was, however, no effect of condition (experimental vs. control) on value for media literacy (H2, b  = 0.17, SE = 0.11, p = .132), news literacy motivation (H3, b  = 0.33, SE = 0.25, p = .191), positive peer social norms (H4, b  = 0.20, SE = 0.20, p = .311), and positive teacher social norms (H5, b = −0.08, SE = 0.15, p = .604), controlling for baseline scores. Thus, we did not find support for H2 through H5. There was a significant effect of condition on news literacy self-efficacy post-intervention (b  = 0.38, SE = 0.12, p = .004, also when controlling for multiple testing⁴). Furthermore, our results show no support for an effect of condition (experimental vs. control) on news consumption (H6, b  = 1.28, SE = 1.04, p = .220), news analysis (H7, b  = 0.23, SE = 0.17, p = .191), and news evaluation (H8, b = −0.00, SE = 0.21, p = .991), thus H6, H7, and H8 also have to be rejected.

Table 2. Linear mixed-effects model and Bayesian mixed-effects model results for condition, for all hypotheses.

	Linear Mixed-Effects Model					Bayesian Mixed-Effects Model
	Estimate	95%CI	SE	df	p	Estimate	95%CI	SE
H1: Self-efficacy	0.38	0.14–0.62	0.12	34.20	.004	0.38	0.13–0.64	0.13
H2: Value for media literacy	0.17	−0.05–0.39	0.11	248	.132	0.18	−0.07–0.42	0.12
H3: News literacy motivation	0.33	−0.15–0.82	0.25	23.17	.191	0.33	−0.19–0.88	0.27
H4: Peer social norms	0.20	−0.19–0.59	0.20	30.33	.311	0.22	−0.20–0.65	0.22
H5: Teacher social norms	−0.08	−0.37–0.22	0.15	248	.604	−0.07	−0.38–0.24	0.16
H6: News consumption	1.28	−0.77–3.33	1.04	248	.220	1.32	−0.81–3.45	1.09
H7: News literacy: analysis	0.23	−0.11–0.57	0.17	29.41	.191	0.22	−0.15–0.60	0.17
H8: News literacy: evaluation	−0.00	−0.41–0.41	0.21	28.70	.991	−0.00	−0.46–0.43	0.22

Note. N = 255. Condition: 0 = Control condition, 1 = Experimental condition. CI = Confidence interval.

Intervention evaluation

Participants (n  = 76⁵) were fairly neutral in their overall evaluations of the intervention lessons (mean scores between 3–4, see online supplemental materials, Appendix D). Lessons were “a little not fun” to “a little fun” (M = 3.64, SD = 1.35) and “a little uninteresting” to “a little interesting” (M = 3.59, SD = 1.25). The platform clarity (M = 4.11, SD = 1.29), assignment understanding (M = 4.59, SD = 1.07), and explanation clarity (M  = 4.47, SD = 1.01) were more positively rated. When looking at the different activities within the intervention platform, participants liked doing the quizzes best (M = 4.04, SD = 1.05), which they also rated the least difficult (M = 2.59, SD = 1.33). Next to the quizzes, participants liked writing their articles (M = 3.50, SD = 1.57). However, they also believed writing these articles was the most difficult (M = 3.22, SD = 1.28). The assignment explanations (M = 3.36, SD = 1.22) and checking articles (M = 3.36, SD = 1.45) were liked least, but still scored between “a little fun” and “fun.” Finally, participants indicated that they would “not so much” (M = 3.00, SD = 1.32) look differently at news articles after these lessons.

With regard to improving the intervention, participants included clarifying the website, giving feedback on whether the news check answers were correct, shortening the information videos, and discussing more in class.

Discussion

In this study, we tested an intervention to stimulate early adolescents’ news literacy elements and application. Contrary to our expectations, we only found higher news literacy self-efficacy in the experimental compared to the control group after the intervention but no differences on the other news literacy elements and applications. Participants positively evaluated the interventions’ clarity and were fairly neutral in their overall intervention evaluations. Although the intervention was not (yet) effective in empowering most news literacy elements and applications, this study does provide valuable insights for news literacy theory and practice.

There is not yet a wide range of evidence-based news literacy interventions (European, 2022; Vraga et al., 2021), nor have there been empirical tests of interventions based on the most recent news media literacy models. Therefore, gaining more insight into what works and does not work is of great value. The lack of effectiveness of this intervention questions the effectiveness of learning-by-doing as a means to increase news literacy application. Engaging in news production is often mentioned as a way to increase the critical consumption of news (e.g., Hobbs, 2010). Yet, the current intervention remained mostly ineffective in increasing news literacy application. Using learning-by-doing for news literacy might thus be more complex than expected. These results underline the need for defining clear goals in media education and testing their effectiveness in reaching these.

One complexity for news literacy interventions might be that empowering news literacy elements in early adolescents and, especially, their news literacy application is a more long-term goal (e.g., Potter, 2022). Overall, media literacy interventions affect knowledge more than actual media literate behaviors (Jeong et al., 2012). Therefore, an intervention of three lessons might have been too short to go beyond (potentially) increasing knowledge and self-efficacy. Although research is still mixed on the effects of intervention length (Jeong et al., 2012;Vahedi et al., 2018), future research could test the effects of longer interventions with more repetition of the intervention elements.

A second insight relates to the lack of feedback that participants received. Although participants received feedback on their knowledge quiz answers, it was impossible to show their scores on their writing and checking of news articles due to technical issues. Furthermore, the intervention did not offer direct feedback after writing or checking a news article. This could have inhibited participants’ growth in news literacy because positive and constructive feedback can lead to higher self-confidence and increase intrinsic motivation (Kusurkar et al., 2011; Ryan & Deci, 2000) and because the feedback would have been a gamifying element to increase competition, engagement with the intervention, news literacy application, and bolster their liking of the intervention (following Tamboer et al., 2022). Thus, future interventions should include better feedback.

Third, future research could go further in customizing interventions, both for early adolescents in general, as well as for individuals and their learning curves. The format used to create news within the platform – written text – could be further amended to the kind of content early adolescents often consume – such as audiovisual content. Furthermore, our descriptive statistics show that early adolescents get most news through other people – in line with earlier research that showed the importance of other people in youth’s news use (e.g., Swart, 2021; Tamboer et al., 2022). In future interventions this could be more central in the intervention, for example, by using more examples of social news use, by being peer-led (Vahedi et al., 2018), or by using peer news literacy ambassadors (Lim & Tan, 2020). These adaptations could increase the relevance of news literacy application and ameliorate social norms.

Regarding customization on the individual level, recent research indicates the existence of different news literacy profiles (Van Damme et al., 2022), which could be the starting point for more personalized news literacy initiatives. For example, a different strategy might be necessary for the news “fans” – who score high on news consumption and news trust, but low to average on news literacy and knowledge – and news “avoiders” – who consume news the least, but score high on news knowledge, and low on news trust. While for news fans, more information and news literacy training might be the optimal strategy, news avoiders could benefit most from focusing on news relevance.

When discussing how the findings of our study can inspire theory and practice, some limitations should be acknowledged. Lower-educated early adolescents were underrepresented in our sample, which makes the results less generalizable to these groups. Future research could include more participants from these educational backgrounds or compare the effectiveness over various educational backgrounds.

Regarding our measures, we rely on self-reports for all variables, also as a proxy for their news literacy application. Participants indicated how often they, in a typical week, use, analyze, and evaluate news. Future research could develop or use more objective measures, such as news diaries, analyzing digital traces, and evaluation tests to measure skills (instead of self-efficacy only). As such, future research could also test whether participants’ increase in self-efficacy was warranted by an increase in their skills, or not. Besides, the wording of our motivation measure could be interpreted in two ways. On the one hand, it could be correctly interpreted as an indication of their motivation. On the other hand, they could have interpreted it as an indication of whether they apply their news literacy. Although we believe that participants interpreted these items correctly, future research should use more straightforward items. In addition, the time an individual spends on doing something does not equal interest, attention, or engagement (Groot Kormelink & Costera Meijer, 2020). Future research should consider not (only) measuring how often someone applies their news literacy, but (also) other measures focused on individuals’ experience, such as the think-aloud protocol, watching and discussing news, and two-sided video ethnography (Groot Kormelink, 2020).

In this first formal test of an intervention to increase news literacy elements and application, we focused on direct effects only and did not include interactions and mediations. We thus did not explicitly test the relationship between news literacy elements and news literacy application, nor interactions or possible sequencing between the news literacy elements. In the theoretical models by Rozendaal (2017) and Vraga et al. (2021), the effects on news literacy elements and news literacy application are stepwise and imply mediation. For example, one might argue that more positive social norms lead to positive intentions for news literacy application, and in turn to an increase in news literacy application (Vraga et al., 2021). Since this experiment was a first formal test that focuses on elements that are generally overlooked in news literacy research, our aim was to test whether the intervention had any direct effects on the news literacy elements and applications. Future research should go into the relations between the news literacy elements, possible sequencing, and mediation.

This study is the first to develop and test a co-created news literacy intervention based on the most recent insights on behavioral change, news literacy, and motivation. Although the current intervention was ineffective in increasing most news literacy elements and news literacy application, it does lay the groundwork for future interventions. To effectively aid early adolescents in finding their way in the abundance of information, more attention should be paid to whether interventions have the expected effects. In particular, researchers could build on the current results and continue by examining the optimal intervention length, integrating feedback, and exploring strategies to further incorporate early adolescents’ social context in interventions. This hopefully brings us closer to answers on how we can empower early adolescents in their engagement with news.

Disclosure statement

No potential conflict of interest was reported by the authors.

Supplementary material

Supplemental data for this article can be accessed online at https://doi.org/10.1080/17482798.2023.2271078

Data availability statement

The data that support the findings of this study and the materials are openly available in the Open Science Framework at osf.io/axz6f.

Additional information

Notes on contributors

Sanne L. Tamboer

Sanne L. Tamboer (Ph.D.) is a postdoctoral researcher at the Behavioural Science Institute, Radboud University. In her dissertation research, she studied how to empower early adolescents in their news literacy application. She is broadly interested in youths’ relationship with the news, from finding ways to increase our understanding of the relationship between youth and news, to co-creating fitting and empowering interventions with youth.

Inge Molenaar

Inge Molenaar (Ph.D.) is a professor of Education and Artificial Intelligence at the Behavioural Science Institute, Radboud University. In her research, she designs, develops and investigates the effects of technology-empowered innovations in education that facilitate students’ learning and teachers’ teaching.

Tibor Bosse

Tibor Bosse (Ph.D.) is a professor of Communication Science and Artificial Intelligence at the Behavioural Science Institute, Radboud University. In his research, he combines insights from these disciplines in order to design, implement and evaluate ‘Social AI’ systems, i.e. intelligent media technologies that have the ability to engage in natural social interactions with human beings. Such systems include, among others, social robots, virtual agents and chatbots.

Mariska Kleemans

Mariska Kleemans (Ph.D.) is a professor at the Behavioural Science Institute, Radboud University. In her research, she investigates how the bond between children, adolescents and the news can be improved, in order to prepare them for their (future) role in society. She also investigates how young news consumers can become more news literate, with a focus on fake news.

Notes

1. Due to technical issues, the scores per participant and per class, and thus the general feedback on their achievements, could not be shown to participants.

2. In the Dutch school system learning processes are adapted to the needs of individuals by dividing pupils into different streams, based on their achievement levels at the end of primary education (around the age of 12). Pupils go to pre-vocational secondary education (called VMBO), higher general secondary education (HAVO), or pre-university level (VWO). This experiment was conducted in classes in which most adolescents are between 12 and 15 years old, but some are slightly older.

3. To include a variable as a clustering factor, it is recommended for this variable to have at least 5 to 6 levels (Bolker, 2022).

4. Using a Bonferroni correction for multiple testing, alpha would be .006 (.05 (alpha)/8 (the amount of tests) = .00625).

5. Of the control group, 76 participants participated in the evaluation.