Evaluating Summarization Quality of Locally Hosted 3-Billion Parameter Large Language Models

Abstract

This study evaluates summarization performance of 3-billion parameter large language models that can run locally on consumer-grade hardware. Using a corpus of 1,000 articles from the XSum dataset, models from the LLaMA, Phi, and Qwen families generated single-sentence summaries with a unified zero-shot prompt. A total of 4,000 summaries, consisting of model-generated outputs and human-authored references, were analyzed using 74 extracted features capturing linguistic abstractiveness, extractiveness, and informativeness. From these, 41 metrics were selected for nonparametric statistical comparison to evaluate model performance relative to human-written summaries. Results show that Phi and LLaMA frequently outperformed the human baseline in informativeness and extractiveness, while struggling with abstraction. Qwen performed well in content retention but was less consistent overall. These findings suggest small-scale models can achieve near-human summarization quality across several metrics but still struggle with abstraction. The study underscores their promise for privacy-centric, resource-limited environments and the importance of transparent, multidimensional evaluation.