Exploring the Potential of a Spoken Dialog System-Delivered Paired Discussion Task for Assessing Interactional Competence

References

• Artunc, E., & Hart, D. (2020). Interactional competence in paired speaking tests: A study on proficiency-based pairings. System, 89, 102–194. https://doi.org/10.1016/j.system.2019.102194
   Web of Science ®Google Scholar
• Bell, G., Gustafson, J., & Heldner, M. (2003). Prosodic adaptation in human-computer interaction. Proceedings of the International Congress of Phonetic Sciences, 15, 2453–2456.
   Google Scholar
• Branigan, H., Pickering, M., Pearson, J., & McLean, J. (2010). Linguistic alignment between humans and computers. Journal of Pragmatics, 42(9), 2355–2368. https://doi.org/10.1016/j.pragma.2009.12.012
   Web of Science ®Google Scholar
• Brooks, L. (2009). Interacting in pairs in a test of oral proficiency: Co-constructing a better performance. Language Testing, 26(3), 341–366. https://doi.org/10.1177/0265532209104666
   Web of Science ®Google Scholar
• Campbell, J., Quincy, C., Osserman, J., & Pedersen, O. (2013). Coding in-depth semi-structured interviews: Problems of unitization and intercoder reliability and agreement. Sociological Methods & Research, 42(3), 294–320. https://doi.org/10.1177/0049124113500475
   Web of Science ®Google Scholar
• Chukharev-Hudilainen, E., & Ockey, G. (2021). The development and evaluation of interactional competence elicitor for oral language assessments. (TOEFL Research Report No. RR-92). ETS. https://doi.org/10.1002/ets2.12319
   Google Scholar
• Crookes, G. (1990). The utterance and other basic units for second language discourse analysis. Applied Linguistics, 11(2), 183–199. https://doi.org/10.1093/applin/11.2.183
   Web of Science ®Google Scholar
• Ducasse, A., & Brown, A. (2009). Assessing paired orals: Rater’s orientation to interaction. Language Testing, 26(3), 423–443. https://doi.org/10.1177/0265532209104669
   Web of Science ®Google Scholar
• Evanini, K., So, Y., Tao, J., Zapata-Rivera, D., Luce, C., Battistini, L., & Wang, X. (2014). Performance of a trialogue-based prototype system for English language assessment for young learners. Proceedings of the Interspeech Workshop on Child Computer Interaction (pp. 79–84). Singapore.
   Google Scholar
• Fischer, K. (2016). Designing speech for a recipient: The roles of partner modelling, alignment, and feedback in so-called ‘simplified registers’. John Benjamins. https://doi.org/10.1075/pbns.270
   Google Scholar
• Fischer, K., & Saunders, J. (2012). Getting acquainted with a developing robot. In A. Salah, J. Ruiz-Del-Solar, C. Mericli, & P. Oudeyer (Eds.), Human behavior understanding (pp. 125–133). Springer. https://doi.org/10.1007/978-3-642-34014-7
   Google Scholar
• Forsyth, C., Luce, C., Zapata-Rivera, D., Tanner Jackson, G., Evanini, K., & So, Y. (2018). Evaluating english language learners’ conversations: Man vs. machine. Computer Assisted Language Learning, 32(4), 398–417. https://doi.org/10.1080/09588221.2018.1517126
   Web of Science ®Google Scholar
• Fulcher, G. (2003). Testing second language speaking. Longman/Pearson Education.
   Google Scholar
• Fulcher, G. (2010). Practical language testing. Hodder Education.
   Google Scholar
• Galaczi, E. (2004). Peer-peer interaction in a paired speaking test: The case of the First Certificate in English [ Doctoral dissertation], Columbia University.
   Google Scholar
• Galaczi, E. (2008). Peer–peer interaction in a speaking test: The case of the first certificate in English examination. Language Assessment Quarterly, 2(2), 89–119. https://doi.org/10.1080/15434300801934702
   Google Scholar
• Galaczi, E. (2014). Interactional competence across proficiency levels: How do learners manage interaction in paired speaking tests? Applied Linguistics, 35(5), 553–574. https://doi.org/10.1093/applin/amt017
   Web of Science ®Google Scholar
• Galaczi, E., & Taylor, L. (2018). Interactional competence: Conceptualizations, operationalizations, and outstanding questions. Language Assessment Quarterly, 15(3), 219–236. https://doi.org/10.1080/15434303.2018.1453816
   Web of Science ®Google Scholar
• Gan, Z. (2010). Interaction in group oral assessment: A case study of higher- and lower-scoring students. Language Testing, 27(4), 585–602. https://doi.org/10.1177/0265532210364049
   Web of Science ®Google Scholar
• Gokturk, N. (2020). Development and evaluation of a spoken dialog system-mediated paired oral task for measuring second language oral communication ability in English [ Doctoral dissertation, Iowa State University]. ProQuest Dissertations Publishing. https://doi.org/10.31274/etd-20210114-53
   Google Scholar
• Gokturk, N., & Chukharev-Hudilainen, E. (2023). Strategy use in a spoken dialog system–delivered paired discussion task: A stimulated recall study. Language Testing, 40(3), 630–657. https://doi.org/10.1177/02655322231152620
   Web of Science ®Google Scholar
• Hayes, A., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1), 77–89. https://doi.org/10.1080/19312450709336664
   Google Scholar
• Hutchby, I., & Wooffitt, R. (2008). Conversation Analysis (2nd ed.). Polity Press.
   Google Scholar
• Jefferson, G. (1989). Preliminary notes on a possible metric which provides for a “standard maximum” silence of approximately one second in conversation. In D., Roger & P. Bull (Eds.), Conversation: An interdisciplinary perspective (pp. 156–197). Multilingual Matters.
   Google Scholar
• Jurafsky, D., & Martin, J. (2023, June 21). Speech and language processing: An introduction to natural language processing, speech recognition, and computational linguistics (3rd). https://web.stanford.edu/~jurafsky/slp3/
   Google Scholar
• Karatay, Y. (2022). Development and validation of spoken dialog system-based oral communication tasks in an ESP context (Publication No. 29165842) [Doctoral dissertation, Iowa State University]. ProQuest Dissertations Publishing.
   Google Scholar
• Kley, K. (2015). Interactional competence in paired speaking tests: Role of paired task and test- taker speaking ability in co-constructed discourse (Publication No. 3711677) [Doctoral dissertation, University of Iowa]. ProQuest Dissertations Publishing.
   Google Scholar
• Kramsch, C. (1986). From language proficiency to IC. The Modern Language Journal, 70(4), 366–372.
   Web of Science ®Google Scholar
• Lam, D. (2018). What counts as “responding”? Contingency on previous speaker contribution as a feature of interactional competence. Language Testing, 35(3), 377–401. https://doi.org/10.1177/0265532218758126
   Web of Science ®Google Scholar
• Lam, D. (2019). Enhancing learning-oriented feedback for cambridge English: First paired interactions. Cambridge Assessment English-Research Notes, 75, 1–25.
   Google Scholar
• Lam, D. (2021). Don’t turn a deaf ear: A case for assessing interactive listening. Applied Linguistics, 42(4), 740–764. https://doi.org/10.1093/applin/amaa064
   Web of Science ®Google Scholar
• Lam, D., Galaczi, E., Nakatsuhara, F., & May, L. (2023). Assessing interactional competence: Exploring ratability challenges. Applied Pragmatics, 5(2), 208–233. https://doi.org/10.1075/ap.00014.lam
   Google Scholar
• Lazaraton, A. (2002). A qualitative approach to the validation of oral language tests. Cambridge University Press.
   Google Scholar
• Leaper, D., & Brawn, J. (2019). Detecting development of speaking proficiency with a group oral test: A quantitative analysis. Language Testing, 36(2), 181–206. https://doi.org/10.1177/0265532218779626
   Web of Science ®Google Scholar
• Luk, J. (2010). Talking to score: Impression management in L2 oral assessment and the co-construction of a test discourse genre. Language Assessment Quarterly, 7(1), 25–53. https://doi.org/10.1080/15434300903473997
   Web of Science ®Google Scholar
• Lumivero. (2020). NVivo (Version 13, 2020 R1). Retrieved from https://lumivero.com/products/nvivo/
   Google Scholar
• Marge, M., Wilson, C., Ward, N., Alwan, A., Artzi, Y., Bansal, M., Blankenship, G., Chai, J., Daumé III, H., Dey, D., Harper, M., Howard, T., Kennington, C., Kruijff-Korbayová, I., Manocha, D., Matuszek, C., Mead, R., Mooney, R., Moore, R. … Yu, Z. (2022). Spoken language interaction with robots: Recommendations for future research. Computer Speech and Language, 71, 101255. https://doi.org/10.1016/j.csl.2021.101255
   Web of Science ®Google Scholar
• Marian, K., & Balaman, U. (2018). Second language interactional competence and its development: An overview of conversation analytic research on interactional change over time. Language and Linguistics Compass, 12(8), 1–16. https://doi.org/10.1111/lnc3.12285
   Google Scholar
• May, L. (2011). Interactional competence in a paired speaking test: Features salient to raters. Language Assessment Quarterly, 8(2), 127–145. https://doi.org/10.1080/15434303.2011.565845
   Web of Science ®Google Scholar
• McCarthy, M. (2010). Spoken fluency revisited. English Profile Journal, 1(1), 1–15. https://doi.org/10.1017/S2041536210000012
   Google Scholar
• McGraw, K., & Wong, S. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30–46. https://doi.org/10.1037/1082-989X.1.1.30
   Web of Science ®Google Scholar
• McKenney, S., & Reeves, T. (2012). Conducting educational design research (2nd ed.). Routledge. https://doi.org/10.4324/9780203818183
   Google Scholar
• Mehan, H. (1979). Learning lessons: Social organization in the classroom. Harvard University Press. https://doi.org/10.4159/harvard.9780674420106
   Google Scholar
• Messick, S. (1989). Validity. In R. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). Macmillan Publishing.
   Google Scholar
• Nakatsuhara, F. (2013). The co-construction of conversation in group oral tests. Peter Lang. https://doi.org/10.3726/978-3-653-03584-1
   Google Scholar
• Nakatsuhara, F., May, L., Lam, D., & Galaczi, E. (2018). Learning oriented feedback in the development and assessment of interactional competence. Research Notes, 70, 1–68. https://www.cambridgeenglish.org/Images/517543-research-notes-70.pdf
   Google Scholar
• Ockey, G. (2009). The effects of a test taker’s group members’ personalities on the test taker’s second language group oral discussion test scores. Language Testing, 26(2), 161–186. https://doi.org/10.1177/0265532208101005
   Web of Science ®Google Scholar
• Ockey, G., & Chukharev-Hudilainen, E. (2021). Human versus computer partner in the paired oral discussion task. Applied Linguistics, 42(5), 924–944. https://doi.org/10.1093/applin/amaa067
   Web of Science ®Google Scholar
• Ockey, G., Chukharev-Hudilainen, E. & Hirch, R.(2023). Assessing interactional competence: ICE versus a human partner.Language assessment quarterly. Early View, 1–22. https://doi.org/10.1080/15434303.2023.2237486
   Google Scholar
• Ockey, G., & Li, Z. (2015). New and not so new methods for assessing oral communication. Language Value, 7(1), 1–21. https://doi.org/10.6035/LanguageV.2015.7.2
   Google Scholar
• O’Sullivan, B. (2002). Learner acquaintanceship and oral proficiency test pair-task performance. Language Testing, 19(3), 277–295. https://doi.org/10.1191/0265532202lt205oa
   Google Scholar
• Pelikan, H., & Broth, M. (2016). Why that Nao? How humans adapt to a conventional humanoid robot in taking turns-at-talk. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, (pp. 4921–4932). California, USA. https://doi.org/10.1145/2858036.2858478
   Google Scholar
• Peltonen, P. (2020). Individual and interactional speech fluency in L2 English from a problem-solving perspective: A mixed-methods approach [ Doctoral dissertation, University of Turku]. UT Open Institutional Repository. https://urn.fi/URN:ISBN:978-951-29-8137-3
   Google Scholar
• Peltonen, P. (2022). Connections between measured and assessed fluency in L2 peer interaction: A problem-solving perspective. International Review of Applied Linguistics in Language Teaching, 60(4), 983–1011. https://doi.org/10.1515/iral-2020-0030
   Web of Science ®Google Scholar
• Pica, T., Kanagy, R., & Falodun, J. (1993). Choosing and using communication tasks for second language instruction. In G. Crookes & S. Gass (Eds.), Tasks and language learning (pp. 9–34). Multilingual Matters.
   Google Scholar
• Plough, I., MacMillan, F., & O’Connell, S. (2011). Changing tasks … changing evidence: A comparative study of two speaking proficiency tests. Selected Proceedings of the 2010 Second Language Research Forum (pp. 91–104). Massachusetts, USA.
   Google Scholar
• Ramanarayanan, V., Lange, P., Evanini, K., Molloy, H., & Suendermann-Oeft, D. (2017). Human and automated scoring of fluency, pronunciation, and intonation during human-machine spoken dialog interaction. Proceedings of the Interspeech Conference (pp. 1711–1715). Stockholm, Sweden. https://doi.org/10.21437/Interspeech.2017-1213
   Google Scholar
• Ramanarayanan, V., Suendermann-Oeft, D., Lange, P., Ivanov, A., Evanini, K., Yu, Z., & Qian, Y. (2016). Bootstrapping development of a cloud-based spoken dialog system in the educational domain from scratch using crowdsourced data (Report No. RR-16-16). ETS. https://doi.org/10.1002/ets2.12105
   Google Scholar
• Sadlund, E., & Greer, T. (2020). How do raters understand rubrics for assessing L2 interactional engagement? A comparative study of CA- and non-CA-formulated performance descriptors. Papers in Language Testing and Assessment, 9(1), 128–163. https://doi.org/10.58379/JCIW3943
   Web of Science ®Google Scholar
• Saldaña, J. (2013). The coding manual for qualitative researchers (2nd ed.). Sage.
   Google Scholar
• Sert, O. (2019). The interplay between collaborative turn sequences and active listenership: Implications for the development of L2 interactional competence. In M. R. Salaberry & S. Kunitz (Eds.), Teaching and testing L2 interactional competence: Bridging theory and practice (pp. 142–166). Routledge. https://doi.org/10.4324/9781315177021-6
   Google Scholar
• Taylor, L. (2011). Examining speaking: Research and practice in assessing second language speaking. Cambridge University Press.
   Google Scholar
• Timpe-Laughlin, V., Evanini, K., Green, A., Blood, I., Dombi, J., & Ramanarayanan, V. (2017). Designing interactive, automated dialogues for L2 pragmatics learning. Proceedings of the SEMDIAL 2017 Workshop on the Semantics and Pragmatics of Dialogue, 21, 116–125. https://doi.org/10.21437/SemDial.2017-13
   Google Scholar
• Timpe-Laughlin, V., Sydorenko, T., & Dombi, J. (2022). Human versus machine: Investigating L2 learner output in face-to-face versus fully automated role-plays. Computer Assisted Language Learning, 1–30. https://doi.org/10.1080/09588221.2022.2032184
   Web of Science ®Google Scholar
• Van Moere, A. (2007). Group oral tests: How does task affect candidate performance and test scores? (Publication No. 507288) [Doctoral dissertation, Lancaster University]. British Library E-Thesis Online Service.
   Google Scholar
• Vo, S. (2021). Evaluating interactional competence in interview and paired discussion tasks: A rater cognition study. TESOL Journal, 12(2). https://doi.org/10.1002/tesj.563
   Web of Science ®Google Scholar
• Wang, F., & Hannafin, M. (2005). Design-based research and technology-enhanced learning environments. Educational Technology Research & Development, 53(4), 5–23. https://doi.org/10.1007/BF02504682
   Web of Science ®Google Scholar
• Young, R. (2011). IC in language learning, teaching, and testing. In E. Hinkel (Ed.), Handbook of research in second language teaching and learning (pp. 426–443). Routledge.
   Google Scholar