Beyond Zero-Shot: Enhancing LLM Financial Complaint Classification with Relevancy-Driven RAG-Based Few-Shot Prompting

Abstract

Large language models (LLMs) have shown significant promise in natural language processing (NLP) tasks, yet their efficacy in real-world consumer complaint classification without fine-tuning remains a challenge. Zero-shot classification offers a valuable solution for categorizing consumer complaints, particularly for handling new and dynamic financial issues, as it allows models to classify data without prior labeled training. However, the nuanced and often overlapping nature of financial complaint categories makes this task particularly difficult. This study explores both zero-shot and a novel few-shot prompting approach for classifying consumer complaints submitted to the Consumer Financial Protection Bureau (CFPB). We compared traditional zero-shot prompting with two few-shot methods: one using randomly selected classified examples and another leveraging the top 5 most relevant classified examples with semantic similarity for in-context learning. Our results consistently demonstrated superior performance with our relevancy-driven, retrieval-augmented generation (RAG) prompting. To validate these findings and ensure they weren't due to chance, we replicated our experiments across several leading LLM models, including GPT-4o, QWEN, Deepseek V3, and Anthropic Claude Sonnet 4.0. Across all tested models, the relevancy-based few-shot approach yielded consistently better results, which we rigorously validated using accuracy, precision, recall, and F1-score. Furthermore, when benchmarked against traditional machine learning models including a fine-tuned RoBERTa, SVM, and logistic regression, our relevancy-driven few-shot approach demonstrated markedly superior performance, validating its effectiveness for this complex classification task. This research highlights the significant potential of carefully curated, relevant examples in enhancing LLM performance for complex text classification tasks in the financial domain.