Counting Species of Ideas: A Bayesian Capture–Recapture Ecology Framework for Estimating LLM Novelty

Abstract

Large Language Models (LLMs) are increasingly used for ideation tasks across domains, ranging from product development to marketing and creative writing. Yet, we lack principled methods to quantify their genuine capacity for novelty and ideation. LLMs derive their generative potential from extensive training data, model architectures, and optimization objectives. These factors collectively define a large yet bounded ideation space. However, standard evaluation methods—typically centered around output-level novelty or diversity—only capture a limited view of this broader ideation landscape. To overcome this limitation, we propose shifting the evaluation focus from isolated outputs to this underlying ideation space. Effectively exploring this space involves answering foundational questions: How expansive is this space? How many unique ideas can the model potentially generate? By adapting capture-recapture (CR) theory from ecology, we introduce an estimation framework tailored to the generative behavior of LLMs and infer the unseen idea space beyond observed samples. Validation through asymptotic extrapolation confirms the reliability of our framework, which offers a principled approach to understanding and comparing the innovation capacities of LLMs.