Follow-Up Questions Improve Generative AI Output and User Experience: Working Towards a Collaborative Model of Human-AI Interaction

Abstract

This research investigates the impact of Large Language Models (LLMs) generating follow-up questions in response to user requests for short (1-page) text documents. This dissertation will argue that there are clear benefits to LLMs asking follow-up questions, and engaging users in thought-provoking and context-clarifying dialog before producing documents and other outputs. Two experiments support this research, including a pilot study and a larger full study with an improved design based on insights from the pilot study. In both experiments, users interacted with a novel web-based AI system designed to ask follow-up questions. Users requested documents they would like the AI to produce. The AI then generated follow-up questions to clarify the user’s needs or offer additional insights before generating the requested documents. After answering the questions, users were shown a document generated using both the initial request and the questions and answers, and a document generated using only the initial request. Users indicated which document they preferred and gave feedback about their experience with the question-answering process. The findings of these experiments show clear benefits to question-asking both in document preference and in the qualitative user experience, and further show that users found more value in questions which were thought-provoking, open-ended, or offered unique insights into the user’s request as opposed to simple information-gathering questions. These results point to the need to incorporate follow-up questions and collaborative dialog into LLMs as a part of the human / AI interaction experience.