A protocol for evaluating AI chatbots’ capabilities for low-resource language teachers

Abstract

One constraint on the potential of generative AI (genAI) in language teaching is the variability in performance across languages. Large language models (LLMs) typically perform better in higher-resource languages in which they have received more training data (Godwin-Jones, 2025; Kern, 2024), but it is difficult to know how well an LLM performs in a given language. Since most users interact with an LLM through a chatbot, this paper presents a protocol that language instructors can use to evaluate a genAI chatbot’s ability to generate and analyze their target language in ways related to language teaching, along with results from piloting this protocol with 46 instructors across 26 languages. Within this protocol, instructors submitted a series of language teaching-related prompts to Microsoft Copilot and evaluated output in terms of linguistic quality, task completion, and usefulness for their teaching using a four-point Likert scale. Analyses of responses to Likert-scale items and open-ended questions indicated that while performance was perceived to be better in higher-resource languages, instructors of low- and extremely low-resource languages generally found the output to be potentially useful. The paper concludes with implications for evaluating genAI chatbots in language teaching and recommendations for categorizing languages based on availability of resources.