Integrating Generative AI into Information Systems Research: A Framework for Synthetic Data Evaluation

Abstract

Generative AI is paving its way into the research process. Among the plethora of available generative AI solutions, the generation of synthetic data is one of the most controversial. The current division of opinion and the lack of formal approach to AI use in research create a situation of conflicting bad practices and under-used potential. This work aims to add nuance and structure to this research practice by providing a general framework to evaluate the use of synthetic data in different stages of the research process, based on the objective and methods of generation. Relying on a breakout literature review, we explore the fields of Data quality management and Control theory to transfer method theories from these fields to help us build the framework. The resulting conceptual framework provides an iterative scheme where, based on the desired properties of the data and its comparison to the synthetic result, the researcher can improve the outcome of the generation process and, equivalently, formally present the properties that make this data suitable for research.