Topological Data Analysis for Enhancing Embedded Analytics for Enterprise Cyber Log Analysis and Forensics

Abstract

Forensic analysis of logs is one responsibility of an enterprise cyber defense team; inherently, this is a big data task with thousands of events possibly logged in minutes of activity. Logged events range from authorized users typing incorrect passwords to malignant threats. Log analysis is necessary to understand current threats, be proactive against emerging threats, and develop new firewall rules. This paper describes embedded analytics for log analysis, which incorporates five mechanisms: numerical, similarity, graph-based, graphical analysis, and interactive feedback. Topological Data Analysis (TDA) is introduced for log analysis with TDA providing novel graph-based similarity understanding of threats which additionally enables a feedback mechanism to further analyze log files. Using real-world firewall log data from an enterprise-level organization, our end-to-end evaluation shows the effective detection and interpretation of log anomalies via the proposed process, many of which would have otherwise been missed by traditional means.