The effectiveness of automated writing evaluation in EFL/ESL writing: a three-level meta-analysis

ABSTRACT

The present study performs a three-level meta-analysis to investigate the overall effectiveness of automated writing evaluation (AWE) on EFL/ESL student writing performance. 24 primary studies representing 85 between-group effect sizes and 34 studies representing 178 within-group effect sizes found from 1993 to 2021 were separately meta-analyzed. The results indicated a medium overall between-group effect size (g = 0.59) and a large overall within-group effect size (g = 0.98) of AWE on student writing performance. Analyses of moderators show that: (1)- AWE is more effective in improving vocabulary usage but less effective in improving grammar in students’ writing; (2)- Grammarly shows potential in being a highly effective tool, though Pigai did not demonstrate such effectiveness; (3)- Medium to long duration of AWE usage leads to a higher effect, but short duration leads to a lower effect in writing outcome compared to non-AWE treatment; (4)- Studying with peers in AWE condition potentially produces a large effect; (5)- AWE is beneficial to students at the undergraduate level, students in the EFL context, and students with intermediate English proficiency. Directions for future research are also discussed in the present study. Overall, AWE is a beneficial application and is recommended for integration in the writing classroom.

KEYWORDS:

• Automated writing evaluation
• AWE
• writing
• meta-analysis
• effectiveness

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the Ministry of Science and Technology in Taiwan [grant number MOST110-2923-H-003 -002 -MY2].

Notes on contributors

Thuy Thi-Nhu Ngo

Thuy Thi-Nhu Ngo is a doctoral student of the English Department at National Taiwan Normal University, Taipei, Taiwan. Her research interests include computer-assisted language learning and meta-analysis.

Howard Hao-Jan Chen

Howard Hao-Jan Chen is distinguished Professor of the English Department at National Taiwan Normal University, Taipei, Taiwan. Professor Chen has published several papers in CALL Journal, ReCALL Journal, and several related language learning journals. His research interests include computer-assisted language learning, corpus research, and second language acquisition.

Kyle Kuo-Wei Lai

Kyle Kuo-Wei Lai is a doctoral student of the English Department at National Taiwan Normal University, Taipei, Taiwan. His research interests include computer-assisted language learning and digital game-based language learning.