A Quantitative Machine Learning Approach to Evaluating Letters of Recommendation

Abstract

Letters of Recommendation (LOR) are key components of the undergraduate and graduate admissions process. A fair and objective evaluation of these LORs is difficult due to diverse applicant-recommender relationships, a lack of standardized criteria, and limited resources for reviewing the LORs. In this paper, we describe three criteria, relevance, specificity, and positivity, for characterizing the quality of an LOR. Approximately 4,000 LORs written in support of students applying to either a Master's in Computer Science or a Master's in Data Science degree are manually rated using these criteria along with rating guidelines developed for this study. Predictive models utilizing natural language processing and machine learning are trained to predict these ratings directly from the LOR text. The work described in this paper can aid in objective and automatic assessment of LORs, or help the admissions committee selectively review the LORs when resources are limited. This work can be extended to support the admissions process for other graduate and undergraduate programs.