An Exploratory Study on Students' Perceptions and Experiences of Assertion Messages in Test Cases

Abstract

Unit testing, or the process of evaluating code at its smallest functional level, is an essential step in ensuring the overall quality of software projects. Core to unit testing are assertion statements developers use to verify that an expected condition has been met (i.e., check a unit of code for correctness.) JUnit, one of the most popular unit test and assertion libraries for Java projects, offers multiple assertion method types to easily and repeatedly check the code. The reason for the failure of a test can be found in the error log report within the integrated development environment (IDE) or from the command line. Furthermore, developers can add their own custom error messages to explain a more detailed cause for the failure of the test. While there is prior research on unit testing and assertions, there is a lack of research on custom assertion error messages.This study investigates the impact of assertion messages on university students' abilities to troubleshoot bugs in common Java string utility methods. Eighty-seven students participated in the study and were randomly assigned to one of three study groups: Group A (no messages), Group B (detailed messages of failure), and Group C (messages that were intended to be less descriptive than those of Group B). The study found that the presence of messages had little impact on the students' abilities to fix the code. Most students could fix all code errors, regardless of academic standing. Students with at least a moderate familiarity in unit testing were significantly more likely to fix more than 4 tests. These findings indicate that familiarity with unit testing had some impact on students' ability to solve the errors in the code. Students were able to assess the quality of messages and students noticed a difference in the quality of the messages between Groups B and C. This study reports features and potential patterns in messages perceived by students to be more or less informative. This study also highlights a discrepancy between understandability and helpfulness, where students could find a message understandable but not necessarily useful in understanding the purpose of the test case.