Undergraduate Pacific Studies Exam Generation and Answering Using Retrieval Augmented Generation and Large Language Models

Simple item page
dc.contributor.authorTyndall, Erick
dc.contributor.authorGayheart, Colleen
dc.contributor.authorSome, Alexandre
dc.contributor.authorGenz, Joseph
dc.contributor.authorLanghals, Brent
dc.contributor.authorWagner, Torrey
dc.date.accessioned2024-12-26T21:05:45Z
dc.date.available2024-12-26T21:05:45Z
dc.date.issued2025-01-07
dc.description.abstractThe capabilities of large language models have increased to the point where entire textbooks can be queried using retrieval-augmented generation (RAG). The study evaluates the ability of OpenAI’s ChatGPT-3.5-Turbo and ChatGPT-4-Turbo models to create and answer exam questions based on an undergraduate textbook. 14 exams were created with true-false, multiple-choice, and short-answer questions from a textbook available online. The accuracy of the models in answering these questions is assessed both with and without access to the source material. Performance was evaluated using text-similarity metrics including ROUGE-1, cosine similarity, and word embeddings. 56 exam scores were analyzed to find that RAG-assisted models outperformed those without access to the textbook, and that ChatGPT-4-Turbo was more accurate than ChatGPT-3.5-Turbo on nearly all exams. The findings demonstrate the potential of generative artificial intelligence tools in academic assessments and provide insights into comparative performance of these models.
dc.format.extent10
dc.identifier.doi10.24251/HICSS.2025.193
dc.identifier.isbn978-0-9981331-8-8
dc.identifier.otherfb3ba1ac-d053-4855-a821-1e76fed84d51
dc.identifier.urihttps://hdl.handle.net/10125/109033
dc.relation.ispartofProceedings of the 58th Hawaii International Conference on System Sciences
dc.rightsAttribution-NonCommercial-NoDerivatives 4.0 International
dc.rights.urihttps://creativecommons.org/licenses/by-nc-nd/4.0/
dc.subjectNatural Language Processing and Large Language Models Supporting Data Analytics for System Sciences
dc.subjectacademic examinations, generative artificial intelligence, large language models, retrieval augmented generation
dc.titleUndergraduate Pacific Studies Exam Generation and Answering Using Retrieval Augmented Generation and Large Language Models
dc.typeConference Paper
dc.type.dcmiText
prism.startingpage1600

Files

Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
0157.pdf
Size:
578.66 KB
Format:
Adobe Portable Document Format
Download

Collections

Natural Language Processing and Large Language Models Supporting Data Analytics for System Sciences