Transforming Generative Large Language Models' Limitations into Strengths using Gestalt: A Synergetic Approach to Mathematical Problem-Solving with Computational Engines

Abstract

This paper presents an innovative approach, known as Gestalt, to enhance the mathematical problem-solving capabilities of Generative Large Language Models (GLLMs) while addressing their inherent limitations. Recognizing the inherent structure and discerning strength of GLLMs, the core of our approach strategically offloads computations, deterministic questions, and knowledge retrieval to external tools such as Wolfram Alpha and Python REPL. This critical augmentation not only mitigates GLLMs' variable reliability in these areas but also fortifies their innate strength - understanding the underlying structure of the problems at hand. With this novel implementation, GLLMs can harness the potential of external systems through well-structured queries, enabling them to make significant strides in problem-solving. In a preliminary evaluation, the Gestalt system demonstrates exceptional performance on a portion of the MATH benchmark dataset, achieving a state-of-the-art accuracy of 59.00%. In comparison, GPT-4 achieves an accuracy of 53.9% on the identical dataset. Through our augmentation approach, we aim to transform the limitations of GLLMs into their strengths, opening up exciting new possibilities not only in advanced mathematical problem-solving but also in various deterministic tasks such as medical diagnosis.