Comparison of Latent Dirichlet Modeling and Factor Analysis for Topic Extraction: A Lesson of History

Peladeau, Normand
Davoodi, Elnaz
Journal Title
Journal ISSN
Volume Title
Topic modeling is often perceived as a relatively new development in information retrieval sciences, and new methods such as Probabilistic Latent Semantic Analysis and Latent Dirichlet Allocation have generated a lot of research. However, attempts to extract topics from unstructured text using Factor Analysis techniques can be found as early as the 1960s. This paper compares the perceived coherence of topics extracted on three different datasets using Factor Analysis and Latent Dirichlet Allocation. To perform such a comparison a new extrinsic evaluation method is proposed. Results suggest that Factor Analysis can produce topics perceived by human coders as more coherent than Latent Dirichlet Allocation and warrant a revisit of a topic extraction method developed more than fifty-five years ago, yet forgotten.
Text Mining in Big Data Analytics, Factor Analysis, Latent Dirichlet Allocation, Text Mining, Topic Modeling
Access Rights
Email if you need this content in ADA-compliant format.