Investigating the Effects of Classification Model Error Type on Trust-relevant Criteria in a Human-Machine Learning Interaction Task

Abstract

Machine learning models have been critiqued for their opaqueness, so recent work has created models to accurately convey model confidence. High performance is the most important aspect of trust in the model. However, when performance drops, accurate decision confidence leads to higher trust outcomes. The current research expands upon this work investigating how incorrect, low confidence decisions differentially impact the trust process. Incorrect decisions were either made on stimuli the model was trained to classify or stimuli outside those classification categories. In a between-subjects design, participants monitored low performing models of varying low confidence error type in an online image classification task. Results demonstrated when the model flagged incorrect stimuli it was not trained to classify with low confidence, process perceptions increased, while decision time and task performance decreased. Our results extend the current framework regarding how model confidence influences the trust process. Implications, limitations, and future research are discussed.