Diagnosability, Adequacy & Size: How Test Suites Impact Autograding

dc.contributor.authorClegg, Benjamin
dc.contributor.authorFraser, Gordon
dc.contributor.authorMcminn, Phil
dc.date.accessioned2021-12-24T17:23:59Z
dc.date.available2021-12-24T17:23:59Z
dc.date.issued2022-01-04
dc.description.abstractAutomated grading is now prevalent in software engineering courses, typically assessing the correctness of students' programs using automated test suites. However, deficiencies in test suites could result in inconsistent grading. As such, we investigate how different test suites impact grades, and the extent to which their observable properties influence these grades. We build upon existing work, using students' solution programs, and test suites that we constructed using a sampling approach. We find that there is a high variation in grades from different test suites, with a standard deviation of ~10.1%. We further investigate how several properties of test suites influence these grades, including the number of tests, coverage, ability to detect other faults, and uniqueness. We use our findings to provide tutors with strategies for building test suites that evaluate students' software with consistency. These strategies include constructing test suites with high coverage, writing unique and diverse tests which evaluate solutions' correctness in different ways, and to run the tests against artificial faults to determine their quality.
dc.format.extent10 pages
dc.identifier.doi10.24251/HICSS.2022.107
dc.identifier.isbn978-0-9981331-5-7
dc.identifier.urihttp://hdl.handle.net/10125/79438
dc.language.isoeng
dc.relation.ispartofProceedings of the 55th Hawaii International Conference on System Sciences
dc.rightsAttribution-NonCommercial-NoDerivatives 4.0 International
dc.rights.urihttps://creativecommons.org/licenses/by-nc-nd/4.0/
dc.subjectAssessment, Evaluation and Measurements (AEM)
dc.subjectautograding
dc.subjectcoverage
dc.subjectdiagnosability
dc.subjectmutation
dc.subjecttesting
dc.titleDiagnosability, Adequacy & Size: How Test Suites Impact Autograding
dc.type.dcmitext

Files

Original bundle
Now showing 1 - 1 of 1
Loading...
Thumbnail Image
Name:
0085.pdf
Size:
389.16 KB
Format:
Adobe Portable Document Format