The HCl Scheduler: Going all-in on Heterogeneity

Authors: 

Michael Kaufmann, IBM Research Zurich, Karlsruhe Institute of Technology; Kornilios Kourtis, IBM Research Zurich

Abstract: 

Heterogeneity is a growing concern for scheduling on the cloud. Hardware is increasingly heterogeneous (e.g., GPUs, FPGAs, diverse I/O devices), emphasizing the need to build schedulers that identify the internal structure of applications and utilize available hardware resources to their full potential. In this paper we present our initial efforts to build a scheduler that tackles heterogeneity (in hardware and in software) as a primary concern. Our scheduler, HCl (Heterogeneous Cluster), models applications as annotated directed acyclic graphs (DAGs), where each node represents a task. It maps tasks onto hardware nodes, also organized in DAGs. Initial results using application models based on TPC-DS queries running on Apache Spark show that HCl can improve significantly upon approaches that do not consider heterogeneity and generate schedules that approach the critical path in length.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

BibTeX
@inproceedings {203310,
author = {Michael Kaufmann and Kornilios Kourtis},
title = {The HCl Scheduler: Going all-in on Heterogeneity},
booktitle = {9th {USENIX} Workshop on Hot Topics in Cloud Computing (HotCloud 17)},
year = {2017},
address = {Santa Clara, CA},
url = {https://www.usenix.org/conference/hotcloud17/program/presentation/kaufmann},
publisher = {{USENIX} Association},
}