Feasibility of administering chatbot-based speaking assessment for low-level EFL students

Abstract

This study examines the feasibility of using a chatbot-based English speaking test (C-EST) in assessing low-level students’ speaking skills. The study included three phases. First, the C-EST was compared with a face-to-face interview test (F-EST) using the ACTFL OPI, focusing on interrater reliability, holistic and analytic test scores, and participants’ perceived number of questions, topic familiarity, and difficulty level. Second, the study investigated participants’ perceived effectiveness of chatbot-based tasks and their interest, motivation, and comfort in taking the C-EST. Last, the study constructed arguments for the C-EST based on test design considerations and the results from the above analyses. Thirty-nine low-level university students completed both tests and questionnaires. Follow-up interviews were conducted with seven participants. Three raters rated both tests. The findings revealed that two tests were comparable regarding interrater reliability, holistic and analytical scores, and participants’ perceptions. Furthermore, participants’ perceptions of the effectiveness of chatbot tasks were uniformly favorable, and their levels of interest, motivation, and comfort were similarly high. The study supported the positive potential of chatbots in speaking assessment for low-level learners and revealed limitations to be resolved. Future suggestions for advancing and implementing chatbots in EFL classrooms are discussed.