Fast Log Analysis Made Easy by Automatically Parsing Heterogeneous Logs

Thursday, November 02, 2017 - 11:30 am12:00 pm

Biplob Debnath and Will Dennis, NEC Laboratories America, Inc.


Existing log analysis tools like ELK (Elasticsearch-LogStash-Kibana), VMware LogInsight, Loggly, etc. provide platforms for indexing, monitoring, and visualizing logs. Although these tools allow users to relatively easily perform ad-hoc queries and define rules in order to generate alerts, they do not provide automated log parsing support. In particular, most of these systems use regular expressions (regex) to parse log messages. These tools assume that the administrators know how to work with regex, and make the admins manually parse and define the fields of interest. By definition, these tools support only supervised parsing as human input is essential. However, human involvement is clearly non-scalable for heterogeneous and continuously evolving log message formats in systems such as IoT, and it is humanly impossible to manually review the sheer number of log entries generated in an hour, let alone days and weeks. On top of that, writing regex-based parsing rules is long, frustrating, error-prone, and regex rules may conflict with each other especially for IoT-like systems. In this talk, we describe how we automatically generate regex rules based on the log data, which is described further in our research work, LogMine: Fast Pattern Recognition for Log Analytics, published at the CIKM 2016 conference. We also show a demo to illustrate how to integrate our solution with the popular ELK stack.

Biplob Debnath, NEC Laboratories America, Inc.

Dr. Biplob Debnath is a researcher at NEC Labs, where his works over the last six years have spanned building end-to-end log analtytics solutions, non-volatile memory-systems, and systems for data deduplication. His work on log analytics ships in NEC's Log Analysis Service. His PhD research on flash based key-value stores ships in Bing ObjectStore, research on data deduplication ships in Windows Server 2012, and research on caching ships in IBM Storage Array. Biplob received a Ph.D. and an M.S. from the University of Minnesota.

Will Dennis, NEC Laboratories America, Inc.

Will Dennis has been employed at NEC Laboratories America for the last 10 years, currently as a Sr. Systems Administrator in the central Information Technology Services group. In the two decades before his employment with NEC Labs, he held various IT/Operations roles in banking, healthcare and web application development (startup). Will is an avid learner and enjoys working with the many disparate technologies in use in an industrial lab setting.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

@conference {207173,
author = {Biplob Debnath and Will Dennis},
title = {Fast Log Analysis Made Easy by Automatically Parsing Heterogeneous Logs},
year = {2017},
address = {San Francisco, CA},
publisher = {USENIX Association},
month = oct

Presentation Video 

Presentation Audio