ChatGPT for Classification: Evaluation of an Automated Course Mapping Method in Academic Libraries

Abstract

The introduction of ChatGPT has been the focus of much attention, but few specific applications for academic libraries have been shown. This study shows a proof of concept for the use of prompt engineering using reference data in GPT-4 to provide automated classifications of undergraduate course descriptions using the Library of Congress classification system. The method reduces the rate of false hallucinations from 48% to 4%, and achieves a precision of 73%. The method was tested with multiple subjects in the natural sciences, applied to 930 classifications from 181 courses, and used for collection development. This method can be implemented by librarians without extensive experience in machine learning techniques.