Rapid Selection of Machine Learning Models Using Greedy Cross Validation

Abstract

This paper introduces a greedy method of performing k-fold cross validation and shows how the proposed greedy method can be used to rapidly identify optimal or near-optimal machine learning (ML) models. Although many methods have been proposed that apply metaheuristic and other search methods to the hyperparameter space as a means of accelerating ML model selection, the cross-validation process itself has been overlooked as a means of rapidly identifying optimal ML models. The current study remedies this oversight by describing a simple, greedy cross validation algorithm and demonstrating that even in its simplest form, the greedy cross validation method can vastly reduce the average time required to identify an optimal or near-optimal ML model within a large set of candidate models. This substantially reduced search time is shown to hold across a variety of different ML algorithms and real-world datasets.