Using Knowledge Graphs to Test for the Presence of Hallucinations in Closed RAG-based and Custom GPT-based LLM Systems

Abstract

Many companies and government agencies are building large language model (LLM) systems for proprietary or specialized uses with sensitive data. These closed systems are not permitted to access the internet as there are concerns about data leakage. At the same time, the absence of internet access limits the data to data contained in the vector database which is used to implement retrieval augmented generation (RAG) or the ingested data in a Custom Generative Pre-trained Transformer (GPT). These systems wish to present authoritative responses and avoid hallucinations, factually inaccurate responses to various prompts. In sensitive government and enterprise environments, hallucinated responses can lead to serious consequences, including misinformation, policy errors, or breaches of trust, making their detection and prevention a critical priority. Using knowledge graphs, we can graph the data or parts of the data and use patterns to predict the areas to test where prompts may create these hallucinations in closed systems. Given the very large number of possible prompts for such a system, being able to target the testing of these systems is extremely important. Once issues are discovered, these graphs can also help developers fix issues where data is missing, erroneous, or must be enhanced using data curation.