From PogChamps to Insights: Detecting Original Content in Twitch Chat

Abstract

The vast volume of chat messages generated during esports events on Twitch represents a valuable source of data for understanding audience behavior. However, the sheer quantity and dynamic nature of this data make manual analysis impractical. This study addresses this challenge by introducing FinTwitchBERT, a model fine-tuned to classify Twitch chat messages into four categories based on their uniqueness. Our model demonstrates the ability to distinguish between original content, repetitive messages such as emote spamming, formulaic messages, and interactive commands chat participants use to interact with channel bots. Pre-trained on over 18 million Finnish Twitch chat messages and utilizing a combination of semi-supervised learning and iterative pseudo-labeling with human-in-the-loop validation, FinTwitchBERT achieves 97.42% accuracy on a test set of unseen chat messages with a limited initial dataset of only 7,529 manually annotated messages.