Skip to main content
USENIX
  • Conferences
  • Students
Sign in
  • Overview
  • Conference Organizers
  • Registration Information
    • Registration Discounts
    • Hotel and Travel Information
    • Live Streaming
  • Purchase the Box Set
  • Why Attend LISA '13?
    • Watch the Video
  • Convince Your Boss
  • Program
    • At a Glance
    • Calendar
    • Training Program
    • Technical Sessions
    • Invited Speakers
    • Workshops
    • Conference Themes
  • Co-located Events
    • SESA '13
    • Gluster Community Day
    • Puppet Camp DC
    • Data Storage Day
    • Build a Cloud Day
  • Students and Grants
  • Sponsorship and Exhibition
    • Sponsors and Exhibitors
    • Vendor Exhibition
    • Exhibitor Services
    • Download Prospectus
  • Call for Participation
  • For Participants
    • Speaker Resources
  • Help Promote!
    • Flyer PDF
    • Brochure PDF
  • Activities
    • Birds-of-a-Feather Sessions
    • Poster Session
    • Lightning Talks Sign Up Form
    • LISA Lab Hack Space
  • Services
  • Questions
  • Past Conferences

sponsors

Gold Sponsor
Gold Sponsor
Silver Sponsor
Silver Sponsor
Silver Sponsor
Silver Sponsor
Bronze Sponsor
Bronze Sponsor
Bronze Sponsor
Bronze Sponsor
Bronze Sponsor
Bronze Sponsor
Bronze Sponsor
Bronze Sponsor
Bronze Sponsor
General Sponsor
General Sponsor
Media Sponsor
Media Sponsor
Media Sponsor
Media Sponsor
Media Sponsor
Media Sponsor
Media Sponsor
Media Sponsor
Media Sponsor
Media Sponsor
Media Sponsor
Media Sponsor
Media Sponsor
Media Sponsor
Media Sponsor
Media Sponsor
Media Sponsor
Media Sponsor
Media Sponsor
Media Sponsor
Media Sponsor
Media Sponsor
Media Sponsor
Industry Partner
Industry Partner
Industry Partner
Industry Partner

twitter

Tweets by @LISAConference

usenix conference policies

  • Event Code of Conduct
  • Conference Network Policy
  • Statement on Environmental Responsibility Policy

You are here

Home ยป Challenges to Error Diagnosis in Hadoop Ecosystems
Tweet

connect with us

http://www.facebook.com/LISAConf
http://twitter.com/LISAConference
http://www.linkedin.com/groups?home=&gid=49559
https://plus.google.com/108588319090208187909/posts
http://www.youtube.com/user/USENIXAssociation

Challenges to Error Diagnosis in Hadoop Ecosystems

Cloud System Administration
Authors: 

Jim (Zhanwen) Li, NICTA; Siyuan He, Citibank; Liming Zhu, NICTA and University of New South Wales; Xiwei Xu, NICTA; Min Fu, University of New South Wales; Len Bass and Anna Liu, NICTA and University of New South Wales; An Binh Tran, University of New South Wales

Abstract: 

Deploying a large-scale distributed ecosystem such as HBase/Hadoop in the cloud is complicated and error-prone. Multiple layers of largely independently evolving software are deployed across distributed nodes on third party infrastructures. In addition to software incompatibility and typical misconfiguration within each layer, many subtle and hard to diagnose errors happen due to misconfigurations across layers and nodes. These errors are difficult to diagnose because of scattered log management and lack of ecosystem-awareness in many diagnosis tools and processes.

We report on some failure experiences in a real world deployment of HBase/Hadoop and propose some initial ideas for better trouble-shooting during deployment. We identify the following types of subtle errors and the corresponding challenges in trouble-shooting: 1) dealing with inconsistency among distributed logs, 2) distinguishing useful information from noisy logging, and 3) probabilistic determination of root causes.

Jim Zhanwen Li, NICTA

Siyuan He, Citibank

Liming Zhu, NICTA

Xiwei Xu, NICTA

Min Fu, NICTA

Len Bass, NICTA & UNSW

Anna Liu, NICTA & UNSW

An Binh Tran, UNSW

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

Li PDF
View the slides
  • Log in or    Register to post comments

Gold Sponsors

Silver Sponsors

Bronze Sponsors

General Sponsors

Media Sponsors & Industry Partners

© USENIX

LISA is a registered trademark of the USENIX Association.

  • Privacy Policy
  • Contact Us