AI Assistants and Generative AI for Knowledge Creation, Retention, and Use
Permanent URI for this collection
Browse
Recent Submissions
Item The Design and Evaluation of the Collaboration between Researchers and Generative AI for Systematic Literature Reviews(2025-01-07) Pham, Vy; Lin, Fu-RenLiterature reviews are crucial in academic research to identify prior coverage. Conventionally, researchers navigate through repositories to select relevant papers and conduct analysis and synthesis, which is time-consuming. With the emergence of generative AI (GAI) technologies, researchers anticipate that GAI will reduce the cognitive load and time spent completing a literature review. However, a researcher's exact trajectory while cooperating with GAI is not fully exposed. This research aims to make the interactive process between a researcher and GAI technology transparent and specify the marginal benefits and limitations for a researcher by cooperating with GAI to complete the literature review tasks. The findings demonstrate GAI's effectiveness in analysis and synthesis while revealing researcher challenges in content validation and nuanced discourse with GAI. The insights obtained from the GAI collaborative literature review could contribute to the interaction design for GAI apps for collaborative literature review.Item Using Large Language Models for the Assessment of Sustainable Forest Investment Projects(2025-01-07) Jannibelli, Maria Letizia; Luo, Jiayu; Sprenkamp, Kilian; Zavolokina, LiudmilaThe integration of Large Language Models (LLMs) into the assessment processes of sustainable forest investment projects is a compelling prospect, given the limitations present in manual assessment. This paper examines how such an LLM-based assessment tool can be designed and whether such a tool can serve as a viable alternative to human experts in this task through the development and subsequent evaluation of a prototype. The analysis shows that the use of retrieval augmented generation to extract and summarize relevant information from project documents is promising but reveals challenges in the use of LLMs for more complex analysis and grading tasks. Design principles and possible steps for further development of the tool are proposed.Item Construct Relation Extraction from Scientific Papers: Is It Automatable Yet?(2025-01-07) Scharfenberger, Jonas; Funk, BurkhardtThe process of identifying relevant prior research articles is crucial for theoretical advancements, but often requires significant human effort. This study examines the feasibility of using large language models (LLMs) to support this task by extracting tested hypotheses, which consist of related constructs, moderators or mediators, path coefficients, and p-values, from empirical studies using structural equation modeling (SEM). We combine state-of-the-art LLMs with a variety of post-processing measures to improve the relation extraction quality. An extensive evaluation yields recall scores of up to 79.2% in construct entity extraction, 58.4% in construct-mediator/moderator-construct extraction, and 39.3% in extracting the full tested hypotheses. We provide a manually annotated dataset of 72 SEM articles and 749 construct relations to facilitate future research. Our findings offer critical insights and suggest promising directions for advancing the field of automated construct relation extraction from scholarly documents.Item Introduction to the Minitrack on AI Assistants and Generative AI for Knowledge Creation, Retention, and Use(2025-01-07) Hadaya, Pierre; Smolnik, Stefan; Holford, W. David; Nissen, Anika