Solving Coding Challenges Jointly with a Large Language Model: Understanding Student Journeys Through Bloom’s Taxonomy

Abstract

With the advancement of large language models (LLMs) such as ChatGPT, students increasingly employ LLMs during university courses to improve their coding skills. However, it is still unclear what using these systems means for student learning. In our study, we explore how students employ a LLM when solving a realistic coding challenge over the course of a semester in small groups. We analyze students’ LLM chat log data through the framework of Bloom’s taxonomy, a well-established education framework, to understand their problem-solving and learning behavior. We enrich our analysis with students’ responses to weekly survey questions, contextualizing our findings with subjective experiences. We find that students primarily use LLMs for lower-order thinking skills like remembering, understanding, and applying procedural knowledge. With our study, we contribute to the growing literature on understanding the effects of working and learning with LLMs and offer practical suggestions for teachers and students.