Evaluating Data Science Project Agility by Exploring Process Frameworks Used by Data Science Teams

Abstract

The lack of effective team process is often noted as one of the key drivers of data science project inefficiencies and failures. To help address this challenge, this research reports on semi-structured interviews, across 16 organizations, which explored data science agile framework usage. While 62% of the organizations reported using an agile framework, none actually followed the Scrum Guide (or any other published framework), but rather, each organization had defined their own process that incorporated one or more aspects of Scrum. The other organizations used a proprietary / ad-hoc approach, often based on a proprietary data science life cycle. In short, while many data science teams are trying to be agile, they are adapting existing frameworks to work within a data science context. Future research could explore how data science teams can best achieve agility, perhaps via new agile frameworks that address the unique data science project management challenges.