Fast Log Analysis Made Easy by Automatically Parsing Heterogeneous Logs

Thursday, November 02, 2017 - 11:30 am12:00 pm

Biplob Debnath and Will Dennis, NEC Laboratories America, Inc.


Existing log analysis tools like ELK (Elasticsearch-LogStash-Kibana), VMware LogInsight, Loggly, etc. provide platforms for indexing, monitoring, and visualizing logs. Although these tools allow users to relatively easily perform ad-hoc queries and define rules in order to generate alerts, they do not provide automated log parsing support. In particular, most of these systems use regular expressions (regex) to parse log messages. These tools assume that the administrators know how to work with regex, and make the admins manually parse and define the fields of interest. By definition, these tools support only supervised parsing as human input is essential. However, human involvement is clearly non-scalable for heterogeneous and continuously evolving log message formats in systems such as IoT, and it is humanly impossible to manually review the sheer number of log entries generated in an hour, let alone days and weeks. On top of that, writing regex-based parsing rules is long, frustrating, error-prone, and regex rules may conflict with each other especially for IoT-like systems. In this talk, we describe how we automatically generate regex rules based on the log data, which is described further in our research work, LogMine: Fast Pattern Recognition for Log Analytics, published at the CIKM 2016 conference. We also show a demo to illustrate how to integrate our solution with the popular ELK stack.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

Presentation Audio

@conference {207173,
author = {Biplob Debnath and Will Dennis},
title = {Fast Log Analysis Made Easy by Automatically Parsing Heterogeneous Logs},
year = {2017},
address = {San Francisco, CA},
publisher = {{USENIX} Association},