Success Factors of Automated Programming Assessment Systems in Different Learning Environments

Abstract

In recent years, the demand for programming skills has surged due to the digitization of the economy and society. Consequently, a limited number of lecturers face an increasing number of students from diverse disciplines in programming education. To address this challenge, there is a need for technical solutions that assist teachers in managing programming tasks, automating student solution evaluation, and facilitating objective and standardized feedback. Automated Programming Assessment Systems (APASs) have emerged as a suitable solution to meet these requirements and handle the growing demand for programming education. However, there is a limited understanding of the success factors specific to APASs, as they have primarily been examined in terms of a single system. To address this gap, we conducted a replication study combining data from two distinct learning environments employing different APASs. Our findings are consistent with the results of previous studies, revealing some common success factors of APASs, thereby facilitating their generalization. Nevertheless, we also identified differences that warrant further investigation and analysis.