Achieving Lean Data Science Agility Via Data Driven Scrum

Abstract

This paper first explores the concept of a lean project and defines four principles team should follow to achieve lean data science. It then describes a new team process framework, which we call Data Driven Scrum (DDS), which enables lean data science project agility. DDS is similar to Scrum but key differences include that DDS defines capability-based iterations (as compared to Scrum time-based sprints), DDS increases the focus in observing and analyzing the output of each iteration (experiment), and that DDS defines process improvement meetings (e.g. retrospectives iteration reviews) to be held on a frequency the team deems appropriate (as compared to Scrum which defines these meetings to be at the end of each iteration). The paper also reports on a pilot study of an organization that adopted the DDS framework.