Prompting for the Unknown: Leveraging In-Context-Learning for Few-Shot Open Set Classification

Abstract

Recognising customer intent is crucial for applications such as chatbots and virtual assistants, requiring accurate interpretation of user inputs. While traditional intent recognition systems depend on large datasets and complex machine learning pipelines, large language models (LLMs) offer competitive performance with significantly less training data through in-context learning (ICL). In this work, we assess the effectiveness of ICL for intent recognition, with a particular focus on detecting out-of-distribution (OOD) inputs. We explore prompting strategies to improve OOD detection and systematically evaluate few-shot classifiers under varying OOD proportions. Our results show that implicit prompting strategies yield better precision for OOD detection, while explicit strategies excel at recall. Moreover, we confirm that LLMs perform comparably to conventional classifiers on in-distribution data. However, a significant fraction of OOD errors are non-overlapping between LLMs and traditional models, highlighting limitations in LLM robustness and suggesting new directions for enhancing generalisation in intent recognition systems.