Large Scale in What Dimension?

Computing systems are complex entities with a diverse set of dimensions. The scale of such a system in each of these dimensions is likely to have a profound impact on how the system is designed, developed, deployed, maintained, supported, and evolved. Today, we find systems that support a large number of users deployed at a large number of sites. These systems support a large suite of applications, consist of a large number of software components, are developed by a large community, manage a large number of computer and storage elements, operate over a large range of physical distances, and/or evolve over a large number of versions and releases. In the more than two decades that we have been working on the Condor distributed resource management system, we have experienced a dramatic change in its scale in all of these dimensions. We will discuss what we learned from dealing with these changes, and what we do to prepare our project for the never-ending stream of scale changes. Miron Livny received a BSc degree in Physics and Mathematics in 1975 from the Hebrew University and MSc and PhD degrees in Computer Science from the Weizmann Institute of Science in 1978 and 1984, respectively. Since 1983 he has been on the Computer Sciences Department faculty at the University of Wisconsin—Madison, where he is currently a Professor of Computer Sciences, the director of the Center for High Throughput Computing, and leader of the Condor project. Dr. Livny's research focuses on distributed processing and data management systems and data visualization environments. His recent work includes the Condor distributed resource management system, the DEVise data visualization and exploration environment, and the BMRB repository for data from NMR spectroscopy.