Exploring User Evaluations of Machine Learning Models: A Qualitative Study on the Impact of Confidence Intervals

Abstract

Research on artificial intelligence and machine learning models has burgeoned in the last decade. However, research has seldom utilized qualitative methods for assessing user-based experiences and system evaluations of AI/ML models. This study aims to provide an example of how thematic text analysis can be used to provide greater insight into user experiences with these systems and examine how varying levels of model transparency affects evaluations. Participants (N = 130) completed an image binning monitoring task with either an uncalibrated classification model (UCM), which displayed high confidence regardless of classification accuracy or a calibrated classification model (CCM), which had greater calibration between accuracy and confidence. Results revealed detailed information on user evaluations for both models including various performance perceptions, impressions, and strategy behaviors. Furthermore, we identified key differences in user evaluations between these models and our confidence manipulation, such as greater trust and confidence display use. Qualitative analysis has been shown to be an effective approach for detailed investigation of user experiences and model evaluation.