S, M, and L Logstash Architectures: Reaching for the Sky

Wednesday, November 01, 2017 - 4:00 pm5:30 pm

Jamie Riedesel, HelloSign

Abstract: 

LogStash can scale. From all-in-one boxes (S) to architectures that involve routing log-lines to separate parsing clusters managed by diverse departments (L), LogStash can do it. If you have the foundations of LogStash down, we can talk about scaling it up. From architectures with syslog as the collector and LogStash purely as a parser, to architectures where LogStash is acting as both collector and parser, you will run into scaling issues as you get bigger. We will go over scaled up and out architectures, and equip you with the knowledge of what XL might look like for you. Also, scale means more than events per second. Scale can also mean maintaining multiple years of certain types of logs. As you scale through time, you will face upgrade problems. Are you still on LogStash 1.5 because 2.x requires ElasticSearch 2.x? Or LogStash 2.4 because 5.x requires ElasticSearch 5.x? We will go over techniques to upgrade your deep history and get your architecture closer to ‘latest’.

Jamie Riedesel, HelloSign

Jamie Riedesel is a DevOps Engineer at HelloSign and has been performing acts of systems administration and engineering since 1997, and more dev-like things since 2010. She moved from corporate IT to the startup space in 2010 and experienced the good kind of culture shock. Jamie has been blogging as sysadmin1138 since 2004, a community elected moderator on ServerFault since 2010, and awarded the Chuck Yerkes community award by LOPSA in 2015.

BibTeX
@conference {207241,
author = {Jamie Riedesel},
title = {S, M, and L Logstash Architectures: Reaching for the Sky},
year = {2017},
address = {San Francisco, CA},
publisher = {USENIX Association},
month = oct
}
Who should attend: 

People with experience with LogStash, looking to find out how to make it grow with you, or do more.

Take back to work: 

People will come away with knowledge of various scaled out LogStash architectures, how to let ElasticSearch keep up, and keep their deep history searchable.

Topics include: 

LogStash, ElasticSearch, Syslog, TSDB, Collectd, AWS