Skip to main content
USENIX
  • Conferences
  • Students
Sign in
  • NSDI '12 Home
  • Registration Information
  • Discounts
  • Organizers
  • At a Glance
  • Technical Sessions
  • Poster and Demo Session
  • Birds-of-a-Feather Sessions
  • Workshops
  • Sponsors
  • Activities
  • Calendar
  • Hotel and Travel Information
  • Students
  • Questions?
  • Help Promote
  • For Participants
  • Call for Papers
  • Past Proceedings

sponsors

Gold Sponsor
Silver Sponsor
Silver Sponsor
Silver Sponsor
Microsoft Research
Silver Sponsor
Bronze Sponsor
Bronze Sponsor
Bronze Sponsor
Media Sponsor
LXer

twitter

Tweets by @usenix

usenix conference policies

  • Event Code of Conduct
  • Conference Network Policy
  • Statement on Environmental Responsibility Policy

You are here

Home » Reoptimizing Data Parallel Computing
Tweet

connect with us

http://twitter.com/usenix
http://www.facebook.com/events/307418625975555/

Reoptimizing Data Parallel Computing

Authors: 

Sameer Agarwal, University of California, Berkeley; Srikanth Kandula, Microsoft Research; Nico Bruno and Ming-Chuan Wu, Microsoft Bing; Ion Stoica, University of California, Berkeley; Jingren Zhou, Microsoft Bing

Abstract: 

Performant execution of data-parallel jobs needs good execution plans. Certain properties of the code, the data, and the interaction between them are crucial to generate these plans. Yet, these properties are difficult to estimate due to the highly distributed nature of these frameworks, the freedom that allows users to specify arbitrary code as operations on the data, and since jobs in modern clusters have evolved beyond single map and reduce phases to logical graphs of operations. Using fixed apriori estimates of these properties to choose execution plans, as modern systems do, leads to poor performance in several instances. We present RoPE, a first step towards re-optimizing data-parallel jobs. RoPE collects certain code and data properties by piggybacking on job execution. It adapts execution plans by feeding these properties to a query optimizer. We show how this improves the future invocations of the same (and similar) jobs and characterize the scenarios of benefit. Experiments on Bing’s production clusters show up to 2× improvement across response time for production jobs at the 75th percentile while using 1.5× fewer resources.

Sameer Agarwal, University of California, Berkeley

Srikanth Kandula, Microsoft Research

Nico Bruno, Microsoft Bing

Ming-Chuan Wu, Microsoft Bing

Ion Stoica, University of California, Berkeley

Jingren Zhou, Microsoft Bing

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

BibTeX
@inproceedings {180662,
author = {Sameer Agarwal and Srikanth Kandula and Nico Bruno and Ming-Chuan Wu and Ion Stoica and Jingren Zhou},
title = {Reoptimizing Data Parallel Computing},
booktitle = {9th USENIX Symposium on Networked Systems Design and Implementation (NSDI 12)},
year = {2012},
isbn = {978-931971-92-8},
address = {San Jose, CA},
pages = {281--294},
url = {https://www.usenix.org/conference/nsdi12/technical-sessions/presentation/agarwal},
publisher = {USENIX Association},
month = apr,
}
Download
Agarwal PDF
View the slides

Presentation Video

Presentation Audio

MP3 Download OGG Download

Download Audio

  • Log in or    Register to post comments

Gold Sponsors

Silver Sponsors

Microsoft Research

Bronze Sponsors

Media Sponsors & Industry Partners

LXer

© USENIX

  • Privacy Policy
  • Contact Us