Enhancing User Behavior Modeling via Machine Learning with Combined Text and Image Data

Abstract

Existing works typically operate on either image or text data from social media, but rarely work with both content types simultaneously. We propose and validate a technique for combining image and text data for predicting user engagement metrics based on social media data. We collected image and text data from 366,415 Facebook posts and a respective 1,305,375 million comments. The combined model achieves a 3.5x improvement in mean squared error when predicting share count and a 14% improvement for comment sentiment over single data type models. Finally, the study demonstrates the ability to pick more performant advertisement out of 16.7 billion pairs; the resulting machine learning models successfully predicts for a greater comment sentiment, comment count, and share count 93%, 65%, and 63% of the time.