AI Readiness: A Reusability Study of Popular AI Algorithms

Abstract

The FAIR Data Principles of findability, accessibility, interoperability, and reusability provide a roadmap to reusing data analysis findings and reproducibility of AI-based data analysis. However, the work done during this research project has identified an issue that impacts AI reproducibility before code and data interoperability can be considered. Namely, code reusability when attempting to recreate the hardware and system-level software or the “runtime environment.” While attempting to determine the metadata needed to FAIRly couple datasets with AI algorithms, the research team determined that the problem of recreating the runtime environment of published state-of-the-art algorithms from the website Papers with Code provided a hurdle that must be overcome before automated data-algorithm coupling can be considered. While containerization solutions such as Docker or Singularity are created to address the issue of inconsistent runtime environments, few AI algorithm developers have embraced publishing containers alongside their AI codes, opting for documenting software dependencies, which only tell part of the runtime story. Additionally, containers are software, and many issues affecting the recreation of runtime environments can also affect orchestrated container solutions. This work describes the process employed to survey 75 openly available AI algorithms, as recorded by Papers with Code, spanning the machine learning areas of computer vision, audio analysis, and natural language processing. It also makes a case that merely publishing the algorithm software repository and datasets used to benchmark the accuracy of the analysis is not enough to enable the reproducibility of results or reuse of AI algorithms. Finally, it identifies the gap in runtime environment reusability between code repositories like GitHub and commercial services like Hugging Face to focus future work. It proposes providing solutions like a container library, enhanced documentation, and other methods to allow reproducible and reusable research and a roadmap for continuing toward a review of enhancing AI-ready data.