Skip to main content
USENIX
  • Conferences
  • Students
Sign in

connect with us


  •  Twitter
  •  Facebook
  •  LinkedIn
  •  Google+
  •  YouTube

twitter

Tweets by @usenix

usenix conference policies

  • Event Code of Conduct
  • Conference Network Policy
  • Statement on Environmental Responsibility Policy

You are here

Home ยป DISC: A System for Distributed Data Intensive Scientific Computing
Tweet

connect with us

DISC: A System for Distributed Data Intensive Scientific Computing

Abstract: 

The increasing computation and data requirements of scientific applications have necessitated the use of distributed resources owned by collaborating parties. While existing distributed systems work well for computation that requires limited data movement, they fail in unexpected ways when the computation accesses, creates, and moves large amounts of data over wide-area networks. In this work, we analyzed the problems with existing systems and used the result of this analysis to design our own system. Realizing that it takes a long while for a new system to stabilize, we tried our best to reuse existing components. We added new components only when we could not get by with adding features to existing ones. We used our system to successfully process three terabytes of DPOSS image data in under a week by using idle CPUs in desktops and commodity clusters in the UW-Madison Computer Science Department and Starlight.

George Kola, Computer Sciences Department, University of Wisconsin-Madison

Tevfik Kosar, Computer Sciences Department, University of Wisconsin-Madison

Jaime Frey, Computer Sciences Department, University of Wisconsin-Madison

Miron Livny, Computer Sciences Department, University of Wisconsin-Madison

Robert Brunner, Department of Astronomy and NCSA, University of Illinois at Urbana-Champaign

Michael Remijan, NCSA, University of Illinois at Urbana-Champaign

BibTeX
@inproceedings {269506,
author = {George Kola and Tevfik Kosar and Jaime Frey and Miron Livny and Robert Brunner and Michael Remijan},
title = {{DISC}: A System for Distributed Data Intensive Scientific Computing },
booktitle = {First Workshop on Real, Large Distributed Systems (WORLDS 04)},
year = {2004},
address = {San Francisco, CA},
url = {https://www.usenix.org/conference/worlds-04/disc-system-distributed-data-intensive-scientific-computing},
publisher = {USENIX Association},
month = dec,
}
Download

Links

Paper: 
http://usenix.org/publications/library/proceedings/worlds04/tech/full_papers/kola/kola.pdf
Paper (HTML): 
http://usenix.org/publications/library/proceedings/worlds04/tech/full_papers/kola/kola_html/index.html
  • Log in or    Register to post comments

© USENIX

  • Privacy Policy
  • Contact Us