Generative AI versus Faculty-Facilitated Scenario-Based Simulation Design by Medical Students

Abstract

Introduction: Interest in generative AI and its application to various disciplines, including medical education, has been exponentially growing. ChatGPT was released in 2022 and has garnered much attention due to its free public access. However, research exploring its use to design scenario-based simulations (SBSs) is limited. Rodgers’ Simulation in Healthcare article (2023) describes ChatGPT’s potential as a useful tool for simulationists to streamline instructional design. Yet, they underscore the crucial role of human intervention in addressing shortcomings related to errors, complexity, and formatting. A background in simulation educational design may be a prerequisite. Often when SBS design is undertaken by novice simulationists, the process can be overwhelming and the instructional design may be incomplete, especially without the guidance of experienced simulationists. The applicability of ChatGPT in aiding non-simulationists with SBS design in healthcare education has not been explored.

Objective: To describe the instructional design process and outcomes of SBS created by medical students using ChatGPT and compare them to SBS created by medical students with simulation-expert faculty guidance.

Methods: Five existing SBSs designed by medical student interest groups (SIG) with simulation faculty guidance were collected from simulation center archives, and scenario goals and patient synopsis were extracted. Medical students unfamiliar with the complete scenario details used the extracted goals and synopsis to create new scenarios using ChatGPT. A blank scenario design template outlining essential elements was used for reference. The ChatGPT conversation tool facilitated iterative refinement of missing elements, errors, or desired modifications. Five scenarios were produced in one session, with elapsed time recorded. The number of design elements and objectives were quantified and compared to the scenarios crafted by SIGs; analysis employed a two-tailed T-test.

Results: On average (n=5), the ChatGPT scenarios design time was 37±11.8 minutes and 5.8±1.3 prompts were needed to produce the final scenario. In contrast, SBSs designed by SIGs with faculty input were created over months, and required multiple faculty-student meetings. ChatGPT produced an average of 4.0±0.7 learning objectives, compared to 3.2±1.6 when developed with faculty. ChatGPT’s objectives were often repetitions of the initial input goals. ChatGPT fulfilled an average of 11.8±0.8 out of 18 template elements, compared to 12.8±3.8 in faculty-guided scenarios.

Discussion: The most notable difference between ChatGPT and faculty guided scenarios is substantial reduction in creation time. AI-assisted scenarios were created in mere minutes, while faculty guided scenarios took months to complete. Time efficiency could allow students to jumpstart the design process and time saved could support further simulation refinement under faculty guidance. There were no statistical differences between groups in the number of fulfilled elements (p=0.53) or objectives (p=0.35). However, the quality and accuracy of the ChatGPT scenarios have yet to be examined by simulation experts. Challenges experienced while using ChatGPT include the omission of requested scenario components, inadvertent removal of desired elements during the iterative process, and inconsistencies in formatting between scenarios.

Target Audience: Novice and expert simulationists, medical students, faculty