On the feasibility of dynamic fractional resource scheduling for clusters

Abstract

Clusters are used in areas beyond traditional high performance computing, including internet service hosting and large scale data processing. Clusters provide a large amount of resources by connecting multiple nodes. In many instances, user jobs run concurrently and use the same cluster requiring the sharing of cluster resources in a fair manner. Clusters also represent a large monetary investment (hardware, cooling, power, staff), which is justified by maintaining a high level of utilization. In order to address both of these requirements, different resource allocation approaches have been utilized. One promising approach is Dynamic Fractional Resource Scheduling (DFRS). DFRS accounts for multiple types of resources provided by cluster nodes. It calculates resource allocations by optimizing a clearly defined objective function defined based on job resource needs and current resource usage. In this thesis we design, implement, and validate a prototype DFRS system as a plugin to an existing virtual cluster management system. A key feature of our DFRS system is that it leverages existing Virtual Machine (VM) technology to share resources in an efficient manner and to discover the resource needs ofVMinstances at runtime. Resource need discovery is fundamental for DFRS to be effective in practice. We verify that state-of-the-art VM technology is both sufficiently accurate and responsive for a sound implementation of DFRS. We describe the design of our DFRS system and our resource need discovery algorithms. Finally, we perform experiments in simulation and on a real-world cluster using our prototype DFRS system. Our results show that DFRS can be implemented in practice and outperforms the default scheduler of the Xen Virtual machine monitor.