Automated speech scoring of dialogue response by Japanese learners of English as a foreign language

ABSTRACT

Purpose: This study builds a new system for automatically assessing learners’ speech elicited from an oral discourse completion task (DCT), and evaluates the prediction capability of the system with a view to better understanding factors deemed influential in predicting speaking proficiency scores and the pedagogical implications of the system. Methodology: We developed a system with a tripartite structure using an automatic speech recogniser, a set of modules that compute a number of speech features, and a scoring model. In total, 210 participants with intermediate English language proficiency level were administered a multi-turn oral DCT, a task which closely resembles discourse in real-life situations. The collected speech and transcribed files were eyeballed and rated by human raters first. Eighty percent of the original dataset was then used to train and examine our prediction model against the remainder of the dataset. Findings: The exact agreement between human and machine scores was 72%, moderately high, and comparable to the literature on automated speech scoring. It could provide a basis for deploying the system in a low-stakes practice environment. Originality/value: This study makes a unique contribution to the wider scholarship, where the use of single-turn DCTs remains prevalent, by presenting a new reliable scoring system for learner speech using an automated DCT with multiple turns. It offers useful insight into ways in which the system could be used in a low-stakes environment, including foreign language classroom settings.

KEYWORDS:

• Automated scoring system
• automatic speech recognition
• discourse completion task
• speaking proficiency
• intelligibility

Acknowledgements

The authors would like to express their sincere gratitude to Akira Murakami, Peter W. Roux, Jonathan Moxon, Yuki Koga, Jo Wakashiba, Chinatsu Ando, Mikikazu Uchida and anonymous referees for their advice and feedback on the earlier drafts of the article. Special thanks also extend to all participants for their hard work throughout the study. Last but not least, we would like to thank the Research and Development Center of Saga Densan Center (https://www.sdcns.co.jp/) for their expertise in system engineering, and Editage (www.editage.com) for English language editing.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 The test-taker’s responses are sent to ETS to be scored by a combination of AI scoring and certified human raters to ensure fairness and quality (ETS 2022a).

2 Information on the latest versions of the TOEFL Junior^® tests is available at https://www.ets.org/toefl/junior.html for.

3 Word error rate (WER) is one of the indices to evaluate the accuracy of Automated speech recognition. It is calculated by the formula WER = (S+D+I)/N, where S is the number of substitution; D, that of deletion; I, that of Insertion; and N, the total number of words.

Additional information

Funding

The current study was funded by JSPS KAKENHI [Grant Number JP18K0074].

Notes on contributors

Yuko Hayashi

Yuko Hayashi is an associate professor in applied linguistics at the Faculty of Education, Saga University in Japan. Since completing her doctoral studies at the University of Oxford in 2012, she has worked on projects on working memory training and automated scoring of speech. Her teaching responsibilities lie in the areas of foreign language teaching methods and research methods.

Yusuke Kondo

Yusuke Kondo works as Associate Professor at Waseda University. He received the degrees of BA, MEd, and PhD from School of Education, Waseda University. His research interests are in language testing, automated scoring of language learners' performance, and learner corpus.

Yutaka Ishii

Yutaka Ishii is an Associate Professor at the Faculty of Education, Chiba University in Japan. His research interests lie in English Language Education and Educational Technology.