Generative AI for Systems Thinking: Can a GPT Question-Answering System Turn Text into the Causal Maps Produced by Human Readers?
Generative AI for Systems Thinking: Can a GPT Question-Answering System Turn Text into the Causal Maps Produced by Human Readers?
Files
Date
2024-01-03
Authors
Giabbanelli, Philippe
Witkowicz, Nathan
Contributor
Advisor
Department
Instructor
Depositor
Speaker
Researcher
Consultant
Interviewer
Annotator
Journal Title
Journal ISSN
Volume Title
Publisher
Volume
Number/Issue
Starting Page
7540
Ending Page
Alternative Title
Abstract
Representing a system as a network is critical to support systems thinking, hence several tools have been developed to derive networks from text in educational technology, modeling and simulation, or forecasting. Large-Scale Pre-Trained Language Models (PLMs) have recently come to the forefront to create question-answering systems (Q&A) that can extract networks from text. In this paper, we design and implement a Q&A system that uses GPT-3.5 together with 12 filters to extract causal maps text. Our evaluation on two topics via several policy documents finds that GPT accurately extracts relevant concept nodes but occasionally reverses causal directions and struggles with the type of causality as it lacks an understanding of event sequence. We also show that automatically extracted maps can only partially resemble human-made maps collected on the same topics. By making our Q&A system open-source on a permanent repository, researchers can evaluate it with newer PLMs as technology improves.
Description
Keywords
Generative and Conversational AI in Information Systems Research and Education: Opportunities and Challenges,
causal map,
generative ai,
gpt,
network analysis,
systems science
Citation
Extent
10 pages
Format
Geographic Location
Time Period
Related To
Proceedings of the 57th Hawaii International Conference on System Sciences
Table of Contents
Rights
Attribution-NonCommercial-NoDerivatives 4.0 International
Rights Holder
Local Contexts
Email libraryada-l@lists.hawaii.edu if you need this content in ADA-compliant format.