An Empirical Study of Automated Adversary Emulators

Abstract

Automated adversary emulators are increasingly researched and applied, but their effectiveness remains unclear. This paper empirically evaluates four emulators using data collected from 1700 hours of tests within a cyber range: Caldera, Deep Exploit, Infection Monkey, and Lore. The outcome from each test was scored according to fulfillment of different tactics in MITRE ATT&CK. The results show that Lore consistently discovered all machines and compromised 97% on average. Caldera discovered 27% and compromised 7%, Infection Monkey discovered 23% but compromised none, and Deep Exploit neither discovered nor compromised any machine. All emulators generated similar intrusion alerts except Caldera, which triggered significantly more when starting with elevated privileges.