488
Views
1
CrossRef citations to date
0
Altmetric
Review Article

The applications of machine learning in computational thinking assessments: a scoping review

, & ORCID Icon
Pages 193-221 | Received 15 Jul 2022, Accepted 04 Aug 2023, Published online: 12 Aug 2023

References

  • Akram, B., Azizolsoltani, H., Min, W., Wiebe, E. N., Navied, A., Mott, B. W. … Lester, J. C. (2020). A data-driven approach to automatically assessing concept-level cs competencies based on student programs. Proceedings of CSEDM@ EDM. http://ceur-ws.org/Vol-2734/
  • Ala-Mutka, K., Uimonen, T., & Jarvinen, H. M. (2004). Supporting students in C++ programming courses with automatic program style assessment. Journal of Information Technology Education Research, 3(1), 245–262. Retrieved fromhttps://www.learntechlib.org/p/111452/
  • Alpaydin, E. (2014). Introduction to machine learning. MIT Press.
  • Arksey, H., & O’Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19–32. https://doi.org/10.1080/1364557032000119616
  • Arlot, S., & Celisse, A. (2010). A survey of cross-validation procedures for model selection. Statistics Surveys, 4(none), 40–79. https://doi.org/10.1214/09-SS054
  • Asgharheidari, F., & Tahriri, A. (2015). A survey of EFL teachers’ attitudes towards critical thinking instruction. Journal of Language Teaching and Research, 6(2), 388–396. https://doi.org/10.17507/jltr.0602.20
  • Barr, V., & Stephenson, C. (2011). Bringing computational thinking to K-12: What is involved and what is the role of the computer science education community? ACM Inroads, 2(1), 48–54. https://doi.org/10.1145/1929887.1929905
  • Bleidorn, W., & Hopwood, C. J. (2019). Using machine learning to advance personality assessment and theory. Personality and Social Psychology Review, 23(2), 190–203. https://doi.org/10.1177/1088868318772990
  • Brasiel, S., Close, K., Jeong, S., Lawanto, K., Janisiewicz, P., & Martin, T. (2017). Measuring computational thinking development with the FUN! Tool. In Emerging research, practice, and policy on computational thinking (pp. 327–347). Springer International Publishing. https://doi.org/10.1007/978-3-319-52691-1_20
  • Brennan, K., & Resnick, M. (2012). New frameworks for studying and assessing the development of computational thinking. Annual Meeting of the American Educational Research Association 2012, Vancouver, BC.
  • Brien, S. E., Lorenzetti, D. L., Lewis, S., Kennedy, J., & Ghali, W. A. (2010). Overview of a formal scoping review on health system report cards. Implementation Science, 5(1), 1–12. https://doi.org/10.1186/1748-5908-5-2
  • Cabitza, F., & Banfi, G. (2018). Machine learning in laboratory medicine: Waiting for the flood? Clinical Chemistry and Laboratory Medicine (CCLM), 56(4), 516–524. https://doi.org/10.1515/cclm-2017-0287
  • Chai, X., Sun, Y., Luo, H., & Guizani, M. (2021). DWES: A dynamic weighted evaluation system for Scratch based on computational thinking. IEEE Transactions on Emerging Topics in Computing,10(2), 917–932. https://doi.org/10.1109/TETC.2020.3044588
  • Chang, C. K., & Tsai, Y. T. (2018). Pair-programming curriculum development of motion-based game for enhancing computational thinking skills. 2018 7th International Congress on Advanced Applied Informatics (pp. 284–287). IEEE. https://doi.org/10.1109/IIAI-AAI.2018.00061
  • Chen, X., Zou, D., Xie, H., Cheng, G., & Liu, C. (2022). Two decades of artificial intelligence in Education. Educational Technology & Society, 25(1), 28–47. https://www.jstor.org/stable/48647028
  • Costa, E. J., Campelo, C. E., & Campos, L. M. S. (2019). Automatic classification of computational thinking skills in elementary school math questions. 2019 IEEE Frontiers in Education Conference (pp. 1–9). https://doi.org/10.1109/FIE43999.2019.9028499
  • Curzon, P., McOwan, P. W., Plant, N., & Meagher, L. R. (2014). Introducing teachers to computational thinking using unplugged storytelling. Proceedings of the 9th workshop in primary and secondary computing education (pp. 89–92). https://doi.org/10.1145/2670757.2670767
  • Cutumisu, M., Adams, C., & Lu, C. (2019). A scoping review of empirical research on recent computational thinking assessments. Journal of Science Education and Technology, 28(6), 651–676. https://doi.org/10.1007/s10956-019-09799-3
  • Cutumisu, M., & Guo, Q. (2019). Using topic modeling to extract pre-service teachers’ understandings of computational thinking from their coding reflections. IEEE Transactions on Education, 62(4), 325–332. https://doi.org/10.1109/TE.2019.2925253
  • Dayan, P., Sahani, M., & Deback, G. (1999). Unsupervised learning. The MIT encyclopedia of the cognitive sciences.
  • De Souza, A. A., Barcelos, T. S., Munoz, R., Villarroel, R., & Silva, L. A. (2019). Data mining framework to analyze the evolution of computational thinking skills in game building workshops. IEEE Access, 7, 82848–82866. https://doi.org/10.1109/ACCESS.2019.2924343
  • De Souza, R. L., Ferreira, F. Z., & da Silva Botelho, S. (2020). A proposal for source code assessment through static analysis. 2020 IEEE Frontiers in Education Conference (pp. 1–5). IEEE. https://doi.org/10.1109/FIE44824.2020.9274050
  • Diana, N., Eagle, M., Stamper, J., Grover, S., Bienkowski, M., & Basu, S. (2017). An instructor dashboard for real-time analytics in interactive programming assignments. Proceedings of the Seventh International Learning Analytics & Knowledge Conference (pp. 272–279). https://doi.org/10.1145/3027385.3027441
  • Diana, N., Eagle, M., Stamper, J., Grover, S., Bienkowski, M., & Basu, S. (2018). Measuring transfer of data-driven code features across tasks in Alice. Proceedings of SPLICE 2018 workshop Computing Science Education Infrastructure, New York, NY (pp. 272–279). ACM.
  • El Naqa, I., & Murphy, M. J. (2015). What is machine learning? In I. El Naqa, R. Li, & M. Murphy (Eds.), Machine learning in radiation oncology (pp. 3–11). Springer. https://doi.org/10.1007/978-3-319-18305-3_1
  • Emerson, A., Geden, M., Smith, A., Wiebe, E., Mott, B., Boyer, K. E., & Lester, J. (2020). Predictive student modeling in block-based programming environments with Bayesian hierarchical models. Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization (pp. 62–70). https://doi.org/10.1145/3340631.3394853
  • Feehan, L. M., Beck, C. A., Harris, S. R., MacIntyre, D. L., & Li, L. C. (2011). Exercise prescription after fragility fracture in older adults: A scoping review. Osteoporosis International, 22(5), 1289–1322. https://doi.org/10.1007/s00198-010-1408-x
  • Figueiredo, J., Lopes, N., & García-Peñalvo, F. J. (2019). Predicting student failure in an introductory programming course with multiple back-propagation. Proceedings of the Seventh International Conference on Technological Ecosystems for Enhancing Multiculturality (pp. 44–49). https://doi.org/10.1145/3362789.3362925
  • Filvà, D. A., Forment, M. A., García-Peñalvo, F. J., Escudero, D. F., & Casañ, M. J. (2019). Clickstream for learning analytics to assess students’ behavior with Scratch. Future Generation Computer Systems, 93, 673–686. https://doi.org/10.1016/j.future.2018.10.057
  • Garneli, V., & Chorianopoulos, K. (2018). Programming video games and simulations in science education: Exploring computational thinking through code analysis. Interactive Learning Environments, 26(3), 386–401. https://doi.org/10.1080/10494820.2017.1337036
  • Grover, S., Pea, R., & Cooper, S. (2015). “Systems of Assessments” for deeper learning of computational thinking in K-12. Proceedings of the 2015 Annual Meeting of the American Educational Research Association, Chicago, IL (pp. 15–20).
  • Guenaga, M., Eguíluz, A., Garaizar, P., & Gibaja, J. (2021). How do students develop computational thinking? Assessing early programmers in a maze-based online game. Computer Science Education, 31(2), 259–289. https://doi.org/10.1080/08993408.2021.1903248
  • Haseski, H. İ., Ilic, U., & Tugtekin, U. (2018). Defining a new 21st century skill-computational thinking: Concepts and trends. International Education Studies, 11(4), 29–42. https://doi.org/10.5539/ies.v11n4p29
  • Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102/003465430298487
  • Hooshyar, D., Malva, L., Yang, Y., Pedaste, M., Wang, M., & Lim, H. (2021). An adaptive educational computer game: Effects on students’ knowledge and learning attitude in computational thinking. Computers in Human Behavior, 114, 106575. https://doi.org/10.1016/j.chb.2020.106575
  • Irgens, G. A., Dabholkar, S., Bain, C., Woods, P., Hall, K., Swanson, H., Horn, M., & Wilensky, U. (2020). Modeling and measuring high school students’ computational thinking practices in science. Journal of Science Education and Technology, 29(1), 137–161. https://doi.org/10.1007/s10956-020-09811-1
  • James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning. Springer New York. https://doi.org/10.1007/978-1-4614-7138-7
  • Jong, M. S. Y., Geng, J., Chai, C. S., & Lin, P. Y. (2020). Development and predictive validity of the computational thinking disposition questionnaire. Sustainability, 12(11), 4459. https://doi.org/10.3390/su12114459
  • Jordan, M. I., & Mitchell, T. M. (2015). Machine learning: Trends, perspectives, and prospects. Science: Advanced Materials and Devices, 349(6245), 255–260. https://doi.org/10.1126/science.aaa8415
  • Ke, F., Moon, J., & Sokolikj, Z. (2020). Tracking representational flexibility development through speech data mining. 2020 IEEE Frontiers in Education Conference (pp. 1–4). https://doi.org/10.1109/FIE44824.2020.9273818
  • Koh, K. H., Basawapatna, A., Bennett, V., & Repenning, A. (2010). Towards the automatic recognition of computational thinking for adaptive visual language learning. 2010 IEEE symposium on visual languages and human-centric computing (pp. 59–66). Leganes, Spain. https://doi.org/10.1109/VLHCC.2010.17
  • Koh, K. H., Basawapatna, A., Nickerson, H., & Repenning, A. (2014). Real time assessment of computational thinking. 2014 IEEE Symposium on Visual Languages and Human-Centric Computing. Melbourne, VIC, Australia. https://doi.org/10.1109/VLHCC.2014.6883021
  • Kong, S. C., & Liu, B. (2020). A performance-based assessment platform for developing computational thinking concepts and practices: EasyCode. Bulletin of the Technical Committee on Learning Technology, 20(2), 3–10. Retrived from https://tc.computer.org/tclt/10-1109-2020-0201001/
  • Korkmaz, Ö., Çakir, R., & Özden, M. Y. (2017). A validity and reliability study of the computational thinking scales (CTS). Computers in Human Behavior, 72, 558–569. https://doi.org/10.1016/j.chb.2017.01.005
  • Korkmaz, C., & Correia, A. P. (2019). A review of research on machine learning in educational technology. Educational Media International, 56(3), 250–267. https://doi.org/10.1080/09523987.2019.1669875
  • Kraleva, R., Kralev, V., & Kostadinova, D. (2019). A methodology for the analysis of block-based programming languages appropriate for children. Journal of Computing Science and Engineering, 13(1), 1–10. https://doi.org/10.5626/JCSE.2019.13.1.1
  • Leonard, J., Buss, A., Gamboa, R., Mitchell, M., Fashola, O. S., Hubert, T., & Almughyirah, S. (2016). Using robotics and game design to enhance children’s self-efficacy, STEM attitudes, and computational thinking skills. Journal of Science Education and Technology, 25(6), 860–876. https://doi.org/10.1007/s10956-016-9628-2
  • Li, Y., Pan, Y., Liu, W., & Zhang, X. (2018). An automated evaluation system for App Inventor Apps. 2018 IEEE 16th international conference on Dependable, Autonomic and Secure Computing (pp. 230–235). https://doi.org/10.1109/DASC/PiCom/DataCom/CyberSciTec.2018.00048
  • Looi, C. K., How, M. L., Longkai, W., Seow, P., & Liu, L. (2018). Analysis of linkages between an unplugged activity and the development of computational thinking. Computer Science Education, 28(3), 255–279. https://doi.org/10.1080/08993408.2018.1533297
  • Min, W., Frankosky, M. H., Mott, B. W., Rowe, J. P., Smith, A., Wiebe, E., Boyer, K. E., & Lester, J. C. (2019). DeepStealth: Game-based learning stealth assessment with deep neural networks. IEEE Transactions on Learning Technologies, 13(2), 312–325. https://doi.org/10.1109/TLT.2019.2922356
  • Moors, L., Luxton-Reilly, A., & Denny, P. (2018). Transitioning from block-based to text-based programming languages. Proceedings of the IEEE 2018 International Conference on Learning and Teaching in Computing and Engineering (pp. 57–64). https://doi.org/10.1109/LaTICE.2018.000-5
  • Moreno-León, J., & Robles, G. (2015, November). Dr. Scratch: A web tool to automatically evaluate Scratch projects. Proceedings of the workshop in primary and secondary computing education (pp. 132–133). https://doi.org/10.1145/2818314.2818338
  • Mouza, C., Marzocchi, A., Pan, Y. C., & Pollock, L. (2016). Development, implementation, and outcomes of an equitable computer science after-school program: Findings from middle-school students. Journal of Research on Technology in Education, 48(2), 84–104. https://doi.org/10.1080/15391523.2016.1146561
  • Mwendia, S. N. (2018). Evaluation model for improving ambient learning systems towards achieving sustainable development goal four. IEEE 2018 5th International Conference on Soft Computing & Machine Intelligence. Nairobi, Kenya. https://doi.org/10.1109/ISCMI.2018.8703251
  • Nurakhmetov, D. (2019). Reinforcement learning applied to adaptive classification testing. Proceeding of Theoretical and Practical Advances in Computer-Based Educational Measurement, 325–336. https://doi.org/10.1007/978-3-030-18480-3_17
  • OECD. (2021). PISA 2022 mathematics framework. https://pisa2022-maths.oecd.org/
  • Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., & Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. International Journal of Surgery, 88, 105906. https://doi.org/10.1016/j.ijsu.2021.105906
  • Pal, M., & Mather, P. M. (2003). An assessment of the effectiveness of decision tree methods for land cover classification. Remote Sensing of Environment, 86(4), 554–565. https://doi.org/10.1016/S0034-4257(03)00132-9
  • Papatga, E., & Ersoy, A. (2016). Improving reading comprehension skills through the SCRATCH Program. International Electronic Journal of Elementary Education, 9(1), 124–150. Retrieved from https://www.iejee.com/index.php/IEJEE/article/view/148
  • Pham, M. T., Rajić, A., Greig, J. D., Sargeant, J. M., Papadopoulos, A., & McEwen, S. A. (2014). A scoping review of scoping reviews: Advancing the approach and enhancing the consistency. Research Synthesis Methods, 5(4), 371–385. https://doi.org/10.1002/jrsm.1123
  • Qi, P., Sun, Y., Luo, H., & Guizani, M. (2020). Scratch-dkg: A framework for constructing scratch domain knowledge graph. IEEE Transactions on Emerging Topics in Computing, 10(1), 170–185. https://doi.org/10.1109/TETC.2020.2996710
  • Raudys, S. J., & Jain, A. K. (1991). Small sample size effects in statistical pattern recognition: Recommendations for practitioners. IEEE Transactions on Pattern Analysis & Machine Intelligence, 13(3), 252–264. https://doi.org/10.1109/34.75512
  • Rich, P. J., & Hodges, C. B. (Eds.). (2017). Emerging research, practice, and policy on computational thinking. Springer International Publishing.
  • Román-González, M., Moreno-León, J., & Robles, G. (2019). Combining assessment tools for a comprehensive evaluation of computational thinking interventions. In Computational thinking education (pp. 79–98). Springer. https://doi.org/10.1007/978-981-13-6528-7_6
  • Román-González, M., Pérez-González, J. C., & Jiménez-Fernández, C. (2017). Which cognitive abilities underlie computational thinking? Criterion validity of the computational thinking test. Computers in Human Behavior, 72, 678–691. https://doi.org/10.1016/j.chb.2016.08.047
  • Romli, R., Sulaiman, S., & Zamli, K. Z. (2010). Automatic programming assessment and test data generation a review on its approaches. IEEE 2010 International Symposium on Information Technology. Kuala Lumpur, Malaysia. https://doi.org/10.1109/ITSIM.2010.5561488
  • Rowe, E., Almeda, M. V., Asbell-Clarke, J., Scruggs, R., Baker, R., Bardar, E., & Gasca, S. (2021). Assessing implicit computational thinking in Zoombinis puzzle gameplay. Computers in Human Behavior, 120, 106707. https://doi.org/10.1016/j.chb.2021.106707
  • Sáez-López, J. M., Román-González, M., & Vázquez-Cano, E. (2016). Visual programming languages integrated across the curriculum in elementary school: A two year case study using “Scratch” in five schools. Computers & Education, 97, 129–141. https://doi.org/10.1016/j.compedu.2016.03.003
  • Samuel, A. L. (1959). Some Studies in Machine Learning Using the Game of Checkers. IBM Journal of Research and Development, 3(3), 210–229. https://doi.org/10.1147/rd.33.0210
  • Sarıtepeci, M., & Durak, H. (2017). Analyzing the effect of block and robotic coding activities on computational thinking in programming education. In I. Koleva & G. Duman (Eds.), Educational research and practice, Chapter 49 (pp. 490–501). St. Kliment Ohridski University Press.
  • Selby, C., & Woollard, J. (2013). Computational thinking: The developing definition. Special Interest Group on Computer Science Education Conference. Canterbury, United Kingdom. https://eprints.soton.ac.uk/id/eprint/356481
  • Shute, V. J. Tobias, S., & Fletcher, J. D. (2011). Stealth assessment in computer-based games to support learning. Computer Games and Instruction, 55(2), 503–524.
  • Shute, V. J., Sun, C., & Asbell-Clarke, J. (2017). Demystifying computational thinking. Educational Research Review, 22, 142–158. https://doi.org/10.1016/j.edurev.2017.09.003
  • Statistics Canada. (2022November 30). Table 98-10-0402-01: Occupation (STEM and non-STEM) by major field of study (STEM and BHASE, detailed) and highest level of education: Canada, provinces and territories. https://doi.org/10.25318/9810040201-eng
  • Steven, L. M., Mortazavi, B. J., Deo, R. C., Curtis, L., & Kao, D. P. (2020). Recommendations for reporting machine learning analyses in clinical research. Circulation: Cardiovascular Quality and Outcomes, 13(10), e006556. https://doi.org/10.1161/CIRCOUTCOMES.120.006556
  • Sun, L., Hu, L., & Zhou, D. (2021). Improving 7th-graders’ computational thinking skills through unplugged programming activities: A study on the influence of multiple factors. Thinking Skills and Creativity, 42, 100926. https://doi.org/10.1016/j.tsc.2021.100926
  • Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction. MIT press.
  • Tang, X., Yin, Y., Lin, Q., Hadad, R., & Zhai, X. (2020). Assessing computational thinking: A systematic review of empirical studies. Computers & Education, 148, 103798. https://doi.org/10.1016/j.compedu.2019.103798
  • Vabalas, A., Gowen, E., Poliakoff, E., Casson, A. J., & Hernandez-Lemus, E. (2019). Machine learning algorithm validation with a limited sample size. PloS One, 14(11), e0224365. https://doi.org/10.1371/journal.pone.0224365
  • Vittorini, P., Menini, S., & Tonelli, S. (2021). An AI-based system for formative and summative assessment in data science courses. International Journal of Artificial Intelligence in Education, 31(2), 159–185. https://doi.org/10.1007/s40593-020-00230-2
  • Von Wangenheim, C. G., Hauck, J. C., Demetrio, M. F., Pelle, R., da Cruz Alves, N., Barbosa, H., & Azevedo, L. F. (2018). CodeMaster–Automatic assessment and grading of app inventor and snap! programs. Informatics in Education, 17(1), 117–150. https://doi.org/10.15388/infedu.2018.08
  • Weintrop, D., & Wilensky, U. (2017). Comparing block-based and text-based programming in high school computer science classrooms. ACM Transactions on Computing Education (TOCE), 18(1), 1–25. https://doi.org/10.1145/3089799
  • Wing, J. M. (2006). Computational thinking. Communications of the ACM, 49(3), 33–35. Retrieved from. https://www.cs.cmu.edu/~15110-s13/Wing06-ct.pdf
  • Wing, J. M. (2008). Computational thinking and thinking about computing. Philosophical Transactions of the Royal Society A: Mathematical, Physical & Engineering Sciences, 366(1881), 3717–3725. https://doi.org/10.1098/rsta.2008.0118
  • Wing, J. M. (2014). Computational Thinking Benefits Society. Social Issues in Computing. http://socialissues.cs.toronto.edu/2014/01/computational-thinking/
  • Wohlin, C. (2014). Guidelines for snowballing in systematic literature studies and a replication in software engineering. 18th International Conference on Evaluation and Assessment in Software Engineering, EASE’14, ACM. New York, NY, USA. https://doi.org/10.1145/2601248.2601268
  • Yadav, A., Burkhart, D., Moix, D., Snow, E., Bandaru, P., & Clayborn, L. (2015). Sowing the seeds: A landscape study on assessment in secondary computer science education. Computer Science Teachers Association. http://www.csteachers.org/page/reports
  • Yadav, A., Mayfield, C., Zhou, N., Hambrusch, S., & Korb, J. T. (2014). Computational thinking in elementary and secondary teacher education. ACM Transactions on Computing Education, 14(1), 1–16. https://doi.org/10.1145/2576872
  • Zhang, L., & Nouri, J. (2019). A systematic review of learning computational thinking through scratch in K-9. Computers & Education, 141, 103607. https://doi.org/10.1016/j.compedu.2019.103607

Reprints and Corporate Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

To request a reprint or corporate permissions for this article, please click on the relevant link below:

Academic Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

Obtain permissions instantly via Rightslink by clicking on the button below:

If you are unable to obtain permissions via Rightslink, please complete and submit this Permissions form. For more information, please visit our Permissions help page.