The effectiveness of automated writing evaluation in EFL/ESL writing: a three-level meta-analysis

References

• References marked with an asterisk indicate studies included in the meta-analysis.
   Google Scholar
• *Al-Mofti, K. (2020). The effect of using online automated feedback on Iraqi EFL learners’ writings at university level. Journal of College of Education for Women, 31(3), 1–14. https://doi.org/10.36231/coeduw/vol31no3.12.
   Google Scholar
• *Aluthman, E. (2016). The effect of using automated essay evaluation on ESL undergraduate students’ writing skill. International Journal of English Linguistics, 6(5), 54–67. https://doi.org/10.5539/ijel.v6n5p54.
   Google Scholar
• Assink, M., & Wibbelink, C. (2016). Fitting three-level meta-analytic models in R: A step-by-step tutorial. The Quantitative Methods for Psychology, 12(3), 154–174. https://doi.org/10.20982/tqmp.12.3.p154.
   Web of Science ®Google Scholar
• Balduzzi, S., Rücker, G., & Schwarzer, G. (2019). How to perform a meta-analysis with R: A practical tutorial. Evidence Based Mental Health, 22(4), 153–160. https://doi.org/10.1136/ebmental-2019-300117.
   PubMed Web of Science ®Google Scholar
• *Barrot, J. (2021). Using automated written corrective feedback in the writing classrooms: Effects on L2 writing accuracy. Computer Assisted Language Learning, 1–24. https://doi.org/10.1080/09588221.2021.1936071.
   Web of Science ®Google Scholar
• Boulton, A., & Cobb, T. (2017). Corpus use in language learning: A meta-analysis. Language Learning, 67(2), 348–393. https://doi.org/10.1111/lang.12224.
   Web of Science ®Google Scholar
• *Chang, T.-S., Huang, H.-W., Li, Y., & Whitfield, B. (2021). Exploring EFL students’ writing performance and their acceptance of AI-based automated writing feedback. 2021 2nd International Conference on Education Development and Studies (ICEDS 2021), March 9–11, Hilo HI, USA. ACM, New York, NY, USA, 5 pages. https://doi.org/10.1145/3459043.3459065
   Google Scholar
• Chen, C.-F. E., & Cheng, W.-Y. E. (2008). Beyond the design of automated writing evaluation: Pedagogical practices and perceived learning effectiveness in EFL writing classes. Language Learning & Technology, 12(2), 94–112. http://hdl.handle.net/10125/44145.
   Web of Science ®Google Scholar
• Chen, H.-J., Cheng, H.-W., & Yang, T.-Y. (2017). Comparing grammar feedback provided by teachers with an automated writing evaluation system. English Teaching & Learning, 41(4), 99–131. https://doi.org/10.6330/ETL.2017.41.4.04.
   Google Scholar
• *Cheng, G. (2017). The impact of online automated feedback on students’ reflective journal writing in an EFL course. The Internet and Higher Education, 34, 18–27. https://doi.org/10.1016/j.iheduc.2017.04.002.
   Web of Science ®Google Scholar
• *Choi, J. (2010). The impact of automated essay scoring (AES) for improving English language learner’s essay writing (Publication No. 3437732) [Doctoral dissertation]. University of Virginia. ProQuest Dissertations Publishing. http://pqdd.sinica.edu.tw/doc/3437732
   Google Scholar
• *Choi, J. (2011). Integration of computerized feedback to improve interactive use of written feedback in English writing class. Educational Technology International, 12(2), 71–94.
   Google Scholar
• Coady, J., & Huckin, T. (1997). Second language vocabulary acquisition: A rationale for pedagogy. Cambridge University Press.
   Google Scholar
• Cotos, E. (2014). Automated writing evaluation. In E. Cotos (Ed.), Genre-based automated writing evaluation for L2 research writing (pp. 1–64). Palgrave Macmillan.
   Google Scholar
• *Cotos, E. (2011). Potential of automated writing evaluation feedback. CALICO Journal, 28(2), 420–459. https://doi.org/10.11139/cj.28.2.420-459.
   Google Scholar
• Cuijpers, P., Weitz, E., Cristea, I., & Twisk, J. (2017). Pre-post effect sizes should be avoided in meta-analyses. Epidemiology and Psychiatric Sciences, 26(4), 364–368. https://doi.org/10.1017/S2045796016000809.
   PubMed Web of Science ®Google Scholar
• DeKeyser, R. (2007). Skill acquisition theory. In B. VanPatten, & J. Williams (Eds.), Theories in second language acquisition: An introduction (pp. 97–113). Lawrence Erlbaum Associates.
   Google Scholar
• Gao, J. (2021). Exploring the feedback quality of an automated writing evaluation system pigai. International Journal of Emerging Technologies in Learning (IJET), 16(11), 322–330. https://doi.org/10.3991/ijet.v16i11.19657.
   Web of Science ®Google Scholar
• *Gao, J., & Ma, S. (2019). The effect of two forms of computer-automated metalinguistic corrective feedback. Language Learning & Technology, 23(2), 65–83. https://10125/44683.
   Web of Science ®Google Scholar
• *Ghufron, M., & Rosyida, F. (2018). The role of grammarly in assessing English as a foreign language (EFL) writing. Lingua Cultura, 12(4), 395–403. https://doi.org/10.21512/lc.v12i4.4582.
   Google Scholar
• Graham, S. (2019). Changing how writing is taught. Review of Research in Education, 43(1), 277–303. https://doi.org/10.3102/0091732X18821125.
   Web of Science ®Google Scholar
• Grimes, D., & Warschauer, M. (2010). Utility in a fallible tool: A multi-site case study of automated writing evaluation. Journal of Technology, Learning, and Assessment, 8(6), 1–44. https://ejournals.bc.edu/index.php/jtla/article/view/1625
   Google Scholar
• Harrer, M., Cuijpers, P., Furukawa, T., & Ebert, D. (2021). Doing meta-analysis with R: A hands-on guide (1st ed.). Chapman & Hall/CRC Press (Taylor & Francis). https://doi.org/10.1201/9781003107347.
   Google Scholar
• *Hassanzadeh, M., & Fotoohnejad, S. (2021). Implementing an automated feedback program for a foreign language writing course: A learner-centric study. Journal of Computer Assisted Learning, 37(5), 1494–1507. https://doi.org/10.1111/jcal.12587.
   Web of Science ®Google Scholar
• Hedges, L., & Olkin, I. (1985). Statistical methods for meta-analysis. Acedamic Press, INC.
   Google Scholar
• *Hoang, T. L. G. (2019). Examining automated corrective feedback in EFL writing classrooms: A case study of criterion [Doctoral dissertation]. The University of Melbourne. UM Campus Repository. http://hdl.handle.net/11343/234481
   Google Scholar
• Hocky, N. (2019). Automated writing evaluation. ELT Journal, 73(1), 82–88. https://doi.org/10.1093/elt/ccy044.
   Web of Science ®Google Scholar
• *Hou, Y. (2020). Implications of aes system of pigai for selfregulated learning. Theory and Practice in Language Studies, 10(3), 261–268. https://doi.org/10.17507/tpls.1003.01.
   Google Scholar
• *Hu, Y., & Zhao, D. (2015). A comparative study of teacher feedback and automated essay scoring in college English writing. International Journal of Linguistics and Communication, 3(2), 82–97. https://doi.org/10.15640/ijlc.v3n2a9.
   Google Scholar
• *Huang, S., & Renandya, W. (2020). Exploring the integration of automated feedback among lower-proficiency EFL learners. Innovation in Language Learning and Teaching, 14(1), 15–26. https://doi.org/10.1080/17501229.2018.1471083.
   Web of Science ®Google Scholar
• *Jayavalan, K., & Razali, A. (2018). Effectiveness of online grammar checker to improve secondary students’ English narrative essay writing. International Research Journal of Education and Sciences (IRJES), 2(1), 1–6. ISSN 2550-2158.
   Google Scholar
• *Khoii, R., & Doroudian, A. (2013). Automated scoring of EFL learners’ written performance: A torture or a blessing? Proceedings of ICT for Language Learning, 5146–5155.
   Google Scholar
• *Lai, Y.-H. (2010). Which do students prefer to evaluate their essays: Peers or computer program. British Journal of Educational Technology, 41(3), 432–454. https://doi.org/10.1111/j.1467-8535.2009.00959.x.
   Web of Science ®Google Scholar
• Lee, H., Warschauer, M., & Lee, J. (2019). The effects of corpus use on second language vocabulary learning: A multilevel meta-analysis. Applied Linguistics, 40(5), 721–753. https://doi.org/10.1093/applin/amy012.
   Web of Science ®Google Scholar
• *Li, J., Link, S., & Hegelheimer, V. (2015). Rethinking the role of automated writing evaluation (AWE) feedback in ESL writing instruction. Journal of Second Language Writing, 27, 1–18. https://doi.org/10.1016/j.jslw.2014.10.004.
   Web of Science ®Google Scholar
• *Li, Z., Feng, H.-H., & Saricaoglu, A. (2017). The short-term and long-term effects of AWE feedback on ESL students’ development of grammatical accuracy. CALICO Journal, 34(3), 355–375. https://doi.org/10.1558/cj.26382.
   Google Scholar
• *Li, Z., & Yan, D. (2020). Effect of pigai.org on English majors’ writing self-efficacy and writing performance. Journal of Physics: Conference Series, 1533, 1–6. https://doi.org/10.1088/1742-6596/1533/4/042086.
   Google Scholar
• *Liao, H.-C. (2016a). Using automated writing evaluation to reduce grammar errors in writing. ELT Journal, 70(3), 308–319. https://doi.org/10.1093/elt/ccv058.
   Web of Science ®Google Scholar
• *Liao, H.-C. (2016b). Enhancing the grammatical accuracy of EFL writing by using an AWE-assisted process approach. System, 62, 77–92. https://doi.org/10.1016/j.system.2016.02.007.
   Web of Science ®Google Scholar
• *Liou, H.-C. (1993). Investigation of using text-critiquing programs in a process-oriented writing class. CALICO Journal, 10(4), 17–38. https://doi.org/10.1558/cj.v10i4.17-38.
   Google Scholar
• Lipsey, M., & Wilson, D. (2001). Practical meta-analysis. Sage.
   Google Scholar
• *Liu, M., Li, Y., Xu, W., & Liu, L. (2017). Automated essay feedback generation and its impact on revision. IEEE Transactions on Learning Technologies, 10(4), 502–513. https://doi.org/10.1109/TLT.2016.2612659.
   Web of Science ®Google Scholar
• *Lu, Z., Li, X., & Li, Z. (2015). AWE-based corrective feedback on developing EFL learners’ writing skill. In F. Helm, L. Bradley, M. Guarda, & S. Thouësny (Eds.), Critical CALL –proceedings of the 2015 EUROCALL conference, Padova, Italy (pp. 375–380). Research-publishing.net. https://doi.org/10.14705/rpnet.2015.000361.
   Google Scholar
• *Lu, Z., & Li, Z. (2016). Exploring EFL learners’ lexical application in AWE-based writing. In S. Papadima-Sophocleous, L. Bradley, & S. Thouësny (Eds.), Call communities and culture – short papers from EUROCALL 2016 (pp. 295–301). Research-publishing.net. https://doi.org/10.14705/rpnet.2016.eurocall2016.578.
   Google Scholar
• *Ma, K. (2013). Improving EFL graduate students’ proficiency in writing through an online automated essay assessing system. English Language Teaching, 6(7), 158–167. https://doi.org/10.5539/elt.v6n7p158.
   Google Scholar
• *Mohsen, M., & Alshahrani, A. (2019). The effectiveness of using a hybrid mode of automated writing evaluation system on efl students’ writing. Teaching English with Technology, 19(1), 118–131. ISSN: 1642-1027.
   Google Scholar
• *Mørch, A. I., Engeness, I., Cheng, V. C., Cheung, W. K., & Wong, K. C. (2017). Essaycritic: Writing to learn with a knowledge-based design critiquing system. Educational Technology & Society, 20(2), 213–223.
   Web of Science ®Google Scholar
• Nunan, D. (1992). Research methods in language learning. Cambridge University Press.
   Google Scholar
• Nunes, A., Cordeiro, C., Limpo, T., & Castro, S. (2021). Effectiveness of automated writing evaluation systems in school settings: A systematic review of studies from 2000 to 2020. Journal of Computer Assisted Learning, 38, 599–620. https://doi.org/10.1111/jcal.12635.
   Web of Science ®Google Scholar
• *Ohta, R. (2008). The impact of an automated evaluation system on student-writing performance. KATE Journal, 22, 23–33. https://doi.org/10.20806/katejo.22.0_23.
   Google Scholar
• O’Neill, R., & Russell, A. (2019). Stop! grammar time: University students’ perceptions of the automated feedback program grammarly. Australasian Journal of Educational Technology, 35(1), 42–56. https://doi.org/10.14742/ajet.3795.
   Web of Science ®Google Scholar
• *Park, J. (2020). Implications of AI-based grammar checker in EFL learning and testing: Korean high school students’ writing (Publication No. I804:11009-000000127977) [Master’s thesis]. Korea University. https://library.korea.ac.kr/detail/?cid=CAT000046022553&ctype=t&lang=en.
   Google Scholar
• *Parra, G. L., & Calero, S. X. (2019). Automated writing evaluation tools in the improvement of the writing skill. International Journal of Instruction, 12(2), 209–226. https://doi.org/10.29333/iji.2019.12214a.
   Web of Science ®Google Scholar
• Pawson, R., & Tilley, N. (2004). Realist evaluation. In H.-U. Otto, A. Polutta, & H. Ziegler (Eds.), Evidence-based practice: Modernising the knowledge base of social work? (pp. 151–182). Barbara Budrich.
   Google Scholar
• *Rich, C. (2012). The impact of online automated writing evaluation: A case study from Dalian. Chinese Journal of Applied Linguistics, 35(1), 63–79. https://doi.org/10.1515/cjal-2012-0006.
   Google Scholar
• *Salavatizadeh, M., & Tahriri, A. (2020). The effect of blended online automated feedback and teacher feedback on EFL learners’ essay writing ability and perception. Journal of Teaching Language Skills, 39(3.2), 181–225. https://doi.org/10.22099/jtls.2021.38753.2899.
   Google Scholar
• *Saricaoglu, A., & Bilki, Z. (2021). Voluntary use of automated writing evaluation by content course students. ReCALL, 33, 265–277. https://doi.org/10.1017/S0958344021000021.
   Web of Science ®Google Scholar
• *Shang, H.-F. (2019). Exploring online peer feedback and automated corrective feedback on EFL writing performance. Interactive Learning Environments, 30, 4–16. https://doi.org/10.1080/10494820.2019.1629601.
   Web of Science ®Google Scholar
• *Toranj, S., & Ansari, D. (2012). Automated versus human essay scoring: A comparative study. Theory and Practice in Language Studies, 2(4), 719–725. https://doi.org/10.4304/tpls.2.4.719-725.
   Google Scholar
• Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor Package. Journal of Statistical Software, 36(3), 1–48. https://doi.org/10.18637/jss.v036.i03.
   Web of Science ®Google Scholar
• Wang, Y. (2012). Differences in L1 and L2 academic writing. Theory and Practice in Language Studies, 2(3), 637–641. https://doi.org/10.4304/tpls.2.3.637-641.
   Google Scholar
• *Wang, P.-L. (2013). Can automated writing evaluation programs help students improve their English writing? International Journal of Applied Linguistics & English Literature, 2(1), 6–12. https://doi.org/10.7575/ijalel.v.2n.1p.6.
   Google Scholar
• *Wang, S., & Xian, Y. (2011). A case study on the efficacy of error correction practice by using the automated writing evaluation system WRM 2.0 on Chinese college students’ English writing. 2011 International Conference on Computational and Information Sciences (pp. 988–991). Chengdu, China, 2011, October 21-23. https://doi.org/10.1109/ICIS.2011.21.
   Google Scholar
• *Wang, Y.-J., Shang, H.-F., & Briody, P. (2013). Exploring the impact of using automated writing evaluation in English as a foreign language university students’ writing. Computer Assisted Language Learning, 26(3), 234–257. https://doi.org/10.1080/09588221.2012.655300.
   Web of Science ®Google Scholar
• Warschauer, M., & Ware, P. (2006). Automated writing evaluation: Defining the classroom research agenda. Language Teaching Research, 10(2), 157–180. https://doi.org/10.1191/1362168806lr190oa.
   Web of Science ®Google Scholar
• Wickham, H., Averick, M., Bryan, J., Chang, W., McGowan, L. D., François, R., Grolemund, G., Hayes, A., Henry, L., Hester, J., Kuhn, M., Pedersen, T. L., Miller, E., Bache, S. M., Müller, K., Ooms, J., Robinson, D., Seidel, D. P., Spinu, V., … Yutani, H. (2019). Welcome to the tidyverse. Journal of Open Source Software, 4(43), 1686. https://doi.org/10.21105/joss.01686.
   Google Scholar
• Willson, V. L., & Putnam, R. R. (1982). A meta-analysis of pretest sensitization effects in experimental design. American Educational Research Journal, 19(2), 249–258. https://doi.org/10.3102/00028312019002249.
   Web of Science ®Google Scholar
• *Xie, Y., Huang, L., & Wang, Y. (2020). The impact of AWE and peer feedback on Chinese EFL learners’ English writing performance. In L.-K. Lee, et al., L. U, F. Wang, S. Cheung, O. Au, & K. Li (Eds.), ICTE 2020, CCIS 1302 (pp. 258–270). https://doi.org/10.1007/978-981-33-4594-2_22.
   Google Scholar
• *Yang, H. (2018). Efficiency of online grammar checker in English writing performance and students’ perceptions. Korean Journal of English Language and Linguistics, 18(3), 328–348. https://doi.org/10.15738/kjell.18.3.201809.328.
   Google Scholar
• *Zhang, L., Huang, Z. (2020). Effect of an automated writing evaluation system on students’ EFL writing performance. In H.-J. So, M. Rodrigo, J. Mason, & A. Mitrovic (Eds.), Proceedings of the 28th International Conference on computers in education (pp. 567–569).
   Google Scholar