Analyzing System Logs: A New View of What's Important
Sivan Sabato, Elad Yom-Tov, Aviad Tsherniak
System logs, such as the Windows Event log or the Linux system log, are an important resource for computer system management. We present a method for ranking system log messages by their estimated value to users, and generating a log view that displays the most important messages.
The ranking process uses a dataset of system logs from many computer
systems to score messages. For better scoring, unsupervised clustering
is used to identify sets of systems that behave similarly. We propose a new feature construction scheme that measures the difference in the ranking of messages by frequency, and show that it leads to better clustering results. The expected distribution of messages in a given system is estimated using the resulting clusters, and log messages are scored using this estimation. We show experimental results from tests on xSeries servers. A tool based on the described methods is being used to aid support personnel in the IBM xSeries support center.
IBM Haifa Labs, Haifa University Campus, Haifa 31905, Israel
IBM T.J. Watson Research Center
, Yorktown Heights, NY 10598