LISA 2002 - Technical Program Abstract
A New Architecture for Managing Enterprise Log Data
Adam Sah - Addamark Technologies, Inc.
Pp. 121-132 of the Proceedings of LISA '02:
Sixteenth Systems Administration Conference,
USENIX Association, 2002).
Server systems invariably write detailed activity logs whose value
is widespread, whether measuring marketing campaigns, detecting
operational trends or catching fraud or intrusion. Unfortunately,
production volumes overwhelm the capacity and manageability of
traditional data management systems, such as relational databases.
Just loading 1,000,000 records is a big deal today, to say nothing of
the billions of records often seen in high-end network security,
network operations and web applications. Since the magnitude of the
problem is scaling with increases in CPU and networking speeds, it
doesn't help to wait for faster systems to catch up.
This paper discusses the issues involving large-scale log
management, and describes a new type of data management platform
called a Log Management System, which is specifically designed to cost
effectively compress, manage and analyze log records in their
original, unsummarized form. To quote Tom Lehrer, ``I have a modest
example here'' - in this case commercial software that can store and
process logs in parallel across a cluster of Linux-based PCs using a
combination of SQL and perl. The paper concludes with some lessons we
learned in building the system.
- View the full text of this paper in
PDF, and Postscript. Until November 2003, you will need your USENIX membership identification in order to access the full papers.
The Proceedings are published as a collective work, © 2002 by the USENIX Association. All Rights Reserved. Rights to individual papers remain with the author or the author's employer. Permission is granted for the noncommercial reproduction of the complete work for educational or research purposes. USENIX acknowledges all trademarks within this paper.
- If you need the latest Adobe Acrobat Reader, you can download it from Adobe's site.