The potential advantages of using an LLM-based chatbot for automated writing evaluation for English teaching practitioners
Date
2025-05-05
Authors
Contributor
Advisor
Department
Instructor
Depositor
Speaker
Researcher
Consultant
Interviewer
Narrator
Transcriber
Annotator
Journal Title
Journal ISSN
Volume Title
Publisher
University of Hawaii National Foreign Language Resource Center
Center for Language & Technology
Center for Language & Technology
Volume
29
Number/Issue
1
Starting Page
1
Ending Page
12
Alternative Title
Abstract
With the ever-increasing demand for assessing large amounts of student writing, Automated Writing Evaluation (AWE) has emerged as an efficient system to satisfy this demand. However, it has also been suggested that applying it in teachers’ writing classes may offer limited results. Given this, the present study developed an AWE chatbot based on ChatGPT 4.0-turbo, designed as an automated rater. A total of 465 narrative essays written by Korean high school EFL students were scored according to three criteria by the developed tool; these were then compared with the scores administered by two professional raters using various analytic measures. The results showed that the AWE chatbot’s scoring was strongly correlated with that of the human raters. Meanwhile, the many-facet Rasch model’s result indicated that the two human raters’ statistics demonstrated an excellent fit, whereas those of the developed AWE chatbot were slightly lower. The Coh-Metrix analysis suggested that the human raters’ scoring tendencies and GPT are largely aligned, indicating that both raters scored the essays similarly. Based on our findings, we suggest that the Large Language Model (LLM)-based AWE chatbot has great potential to assist teachers in EFL writing classrooms.
Description
Keywords
Automated Writing Evaluation, ChatGPT, L2 Writing, Many-Facet Rasch Model
Citation
Kim, K., Lee, J. H., & Shin, D. (2025). The potential advantages of using an LLM-based chatbot for automated writing evaluation for English teaching practitioners. Language Learning & Technology, 29(1), 1–12. https://hdl.handle.net/10125/73628
Extent
12
Format
Geographic Location
Time Period
Related To
Related To (URI)
Table of Contents
Rights
Rights Holder
Local Contexts
Collections
Email libraryada-l@lists.hawaii.edu if you need this content in ADA-compliant format.