Skip to main content
USENIX
  • Conferences
  • Students
Sign in
  • Home
  • Attend
    • Registration Information
    • Registration Discounts
    • Venue, Hotel, and Travel
    • Students and Grants
  • Activities
    • Birds-of-a-Feather Sessions
    • Poster Session and Happy Hour
  • Program
    • At a Glance
    • Technical Sessions
  • Sponsorship
  • Participate
    • Instructions for Participants
    • Call for Papers
    • Call for Posters
  • About
    • Organizers
    • Help Promote
    • Questions
    • Past Symposia
  • Home
  • Attend
  • Activities
  • Program
  • Sponsorship
  • Participate
  • About

sponsors

Silver Sponsor
Silver Sponsor
Silver Sponsor
Silver Sponsor
Bronze Sponsor
Media Sponsor
Media Sponsor
Media Sponsor
Media Sponsor
Media Sponsor
Media Sponsor
Media Sponsor
Media Sponsor
Industry Partner

help promote

NSDI '16 button

Get more
Help Promote graphics!

connect with us


  •  Twitter
  •  Facebook
  •  LinkedIn
  •  Google+
  •  YouTube

twitter

Tweets by @usenix

usenix conference policies

  • Event Code of Conduct
  • Conference Network Policy
  • Statement on Environmental Responsibility Policy

You are here

Home ยป Ernest: Efficient Performance Prediction for Large-Scale Advanced Analytics
Tweet

connect with us

Ernest: Efficient Performance Prediction for Large-Scale Advanced Analytics

Authors: 

Shivaram Venkataraman, Zongheng Yang, Michael Franklin, Benjamin Recht, and Ion Stoica, University of California, Berkeley

Abstract: 

Recent workload trends indicate rapid growth in the deployment of machine learning, genomics and scientific workloads on cloud computing infrastructure. However, efficiently running these applications on shared infrastructure is challenging and we find that choosing the right hardware configuration can significantly improve performance and cost. The key to address the above challenge is having the ability to predict performance of applications under various resource configurations so that we can automatically choose the optimal configuration.

Our insight is that a number of jobs have predictable structure in terms of computation and communication. Thus we can build performance models based on the behavior of the job on small samples of data and then predict its performance on larger datasets and cluster sizes. To minimize the time and resources spent in building a model, we use optimal experiment design, a statistical technique that allows us to collect as few training points as required. We have built Ernest, a performance prediction framework for large scale analytics and our evaluation on Amazon EC2 using several workloads shows that our prediction error is low while having a training overhead of less than 5% for long-running jobs.

Shivaram Venkataraman, University of California, Berkeley

Zongheng Yang, University of California, Berkeley

Michael Franklin, University of California, Berkeley

Benjamin Recht, University of California, Berkeley

Ion Stoica, University of California, Berkeley

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

BibTeX
@inproceedings {194946,
author = {Shivaram Venkataraman and Zongheng Yang and Michael Franklin and Benjamin Recht and Ion Stoica},
title = {Ernest: Efficient Performance Prediction for {Large-Scale} Advanced Analytics},
booktitle = {13th USENIX Symposium on Networked Systems Design and Implementation (NSDI 16)},
year = {2016},
isbn = {978-1-931971-29-4},
address = {Santa Clara, CA},
pages = {363--378},
url = {https://www.usenix.org/conference/nsdi16/technical-sessions/presentation/venkataraman},
publisher = {USENIX Association},
month = mar,
}
Download
Venkataraman PDF
View the slides

Presentation Audio

MP3 Download

Download Audio

  • Log in or    Register to post comments

Silver Sponsors

Bronze Sponsors

Media Sponsors & Industry Partners

Open Access Publishing Partner

© USENIX

  • Privacy Policy
  • Contact Us