LISA18 Conference Program

LISA18 Program Grid

View the program in mobile-friendly grid format.

Already Registered for LISA18?

Curious to know who else at LISA does what you do? Conferences are a great way to meet your birds-of-a-feather, and this year we've listed various roles on the LISA18 mobile app website. If you'd like, create a profile with your picture and add yourself to a role. You can choose whether to make your profile and schedule public. Then choose the contact details you would like to share, so people can find you, and you can find them.

Monday, October 29, 2018

7:30 am–5:00 pm

On-Site Registration and Badge Pickup

7:30 am–8:45 am

Continental Breakfast

8:45 am–9:00 am

Opening Remarks

Program Co-Chairs: Brendan Gregg, Netflix, and Rikki Endsley, Opensource.com

9:00 am–10:30 am

10:30 am–11:00 am

Break with Refreshments

11:00 am–12:30 pm

Talks I

Introducing Reliability Toolkit: Easy-to-Use Monitoring and Alerting

Monday, 11:00 am11:45 am

Janna Brummel and Robin van Zijll, ING

LISA18: Architecture

By definition, SREs are responsible for the reliability of sites, but what if they don’t own any sites themselves? Within ING, the largest bank of the Netherlands, BizDevOps teams are autonomous and responsible for the build and run of their services. In theory, that could make the existence of SRE obsolete, right? How can you improve availability for end customers in an environment of engineers with full service ownership? How to convince without the power of intervention? How to improve without being blameful?

We’ll explain how we, a team of 8 SREs among 1700 DevOps engineers, try to improve stability by focussing on software engineering. We created the Reliability Toolkit to help BizDevOps teams with their reliability challenges in the fields of white box monitoring and alerting while minimizing toil.

This talk will explain:

  • Our SRE team purpose and why we think our approach with heavy focus on software engineering works for our organization
  • The concept of the Reliability Toolkit and introduction of its components and their set up (Prometheus, Alertmanager, Grafana, NGINX Log Aggregator, SMS and ChatOps functionalities)
  • How we provision Reliability Toolkit
  • How we convince, onboard and educate BizDevOps teams to use the Reliability Toolkit

To conclude we will end our talk with a demo of our Reliability Toolkit

Janna Brummel, ING

Janna is IT chapter lead for the site reliability engineering squad within the Domestic Bank (Retail) for ING in the Netherlands. Her job is to help other teams within the bank to know more about their services' performance and to be able to respond more efficiently to incidents. Before this, Janna worked as business manager and dev engineer of credit cards and debit cards back end systems.

Robin van Zijll, ING

Robin is a Site Reliability Engineer @ ING and PO of the SRE Team, and has years of experience in being on-call for all services offered to our retail customers.

Incident Management at Netflix Velocity

Monday, 11:45 am12:30 pm

Dave Hahn, Netflix

LISA18: Engineering

Netflix—as a service and a system—goes through an enormous amount of change all the time. Our engineering teams make 1000s of changes a days while our customers stream 100,000,000s hour of entertainment every day. At that velocity, an outage seconds or minutes long has real and noticeable impact to our customers. Stir in some Chaos Engineering and things become even more unpredictable.

The talk begins with a story. Netflix had a healthy relationship with Chaos Monkey—our tool to ensure that instance loss didn’t affect a running service application. We’d had such good luck we extended our plans from just Chaos Monkey to more Monkeys that would do nasty things to our environment. A new entry, Latency Monkey, would help us increase the health between our microservices by injecting latency and errors at our common IPC layer. What we thought was a safe, little experiment went completely off the rails. The centralized SRE team, called CORE, realized that we’d have to think differently about outages and managing them if the company was going to to be successful moving forward.

This is the story of how the centralized SRE team at Netflix changed and adapted to help the service and our engineering teams prepare for and handle problems—big and small—when they do occur.

Key Takeaways:

  • How Netflix prepares for failures
  • Incident Handling at velocity requires special expertise
  • Preparation and training of everyone that runs services is key for quick recovery
  • You should be spending more time after an incident learning that you do during an incident managing
  • Outages being unique is an excellent goal—it takes work to make it happen

Dave Hahn, Netflix

Dave Hahn is a Senior SRE in the Cloud Operations & Reliability Engineering organization at Netflix. He has designed tools and systems used by many teams in the organization to support the Netflix service. He has decades of experiences in systems operations, networks, reliability, cron jobs, cable termination, grep, and not taking himself very seriously.

Talks II

SLO Burn—Reducing Alert Fatigue and Maintenance Cost in Systems of Any Size

Monday, 11:00 am11:45 am

Jamie Wilkinson, Google

LISA18: Architecture

Based on a true story.

As systems grow, they get more components, and more ways to fail. The alerts of the last system's design can slowly "boil the frog", and all of a sudden the SRE team finds they have no time left to address scaling problems because they're constantly firefighting. Alert fatigue sets in and the team will burn out.

Naturally maintenance work will always increase as the system itself grows. To make alerting sustainable, instead of on cause, only page on symptom, and even then by declaring what the acceptable threshold of symptom is -- also known as the SLO (and it's complement, the error budget).

Even at Google scale, many teams are yet to implement the change in their monitoring to realise SLO based alerts. But systems don't need to be the size of a planet to benefit from these patterns.

Whether you're oncall for 10 machines or 10 datacenters, in this talk a well rested champion of work/life balance will show you how to select service objectives, then to construct robust and low maintenance alerting rules using Prometheus for a live demonstration. We'll also discuss the tooling required to help make such a system retain observability in the absence of noisy caused-based alerts, now they're not telling you exactly which components are failing.

Jamie Wilkinson, Google

Jamie Wilkinson is a Site Reliability Engineer at Google. Contributing author to the "SRE Book", he has presented on contemporary topics at prominent conferences such as linux.conf.au, Monitorama, PuppetConf, Velocity, and SRECon. His interests began in monitoring and automation of small installations, but continues with human factors in automation and systems maintenance on large systems. Despite over 15 years in the industry, he is still trying to automate himself out of a job.

Serverless Data Processing and Machine Learning

Monday, 11:45 am12:30 pm

Sunil Mallya, AWS

LISA18: Architecture

Serverless computing reduces infrastructure complexity, provides fine grained billing and easy scalability. Setting up concurrent data processing infrastructure pipelines that supports many users is a complex task, moreover utilization, cost and performance are hard to tune for these pipelines. Machine Learning (ML) workloads are on the uptick, and likes of Apache Spark aim to provide an end to end data to ML story, but run in to the same complexities previously mentioned. These aren't two disjoint data and ML workflows, but share a lot in common.

In this talk, I will present a serverless data and machine learning pipeline that includes a MapReduce framework built on using Amazon S3 and AWS Lambda. We'll see how it can help alleviate issues like concurrent processing, cost and scaling. I will also showcase how machine learning algorithms like K-Means clustering can be built on top of this framework, exploiting the inherent distributed architecture. We'll then discuss the benefits and challenges of the framework with a focus on production deployments for ML models.

Sunil Mallya, AWS

Sunil is a Sr. AI Solutions Architect focused on Deep Learning in the Machine Learning Lab team at AWS working with customers in various industry verticals. Prior to that, he co-founded the neuroscience and machine learning based image analysis and video thumbnail recommendation company Neon labs. He’s also worked on building large scale low latency systems at Zynga and has an acute passion for Serverless computing. He hold a master’s degree in computer science from Brown University.

Training I

Modern Infrastructure Provisioning and CI/CD with Terraform & Terratest

Monday, 11:00 am12:30 pm
Duncan Hutty, Oracle Cloud Infrastructure
LISA18: Engineering

The days of handcrafted systems, services and infrastructure are over - it's the era of Infrastructure As Code.

We write code for orchestration tools to run configuration management systems to ensure that individual applications systems are easily reproducible. We write test suites to verify the behaviour of those applications.

In the same way, with HashiCorp's Terraform, we can write code to provision backend infrastructure such as instances, networks, storage, databases and ancillary services such as load balancers/proxies, DNS, TLS termination and more. And then reproduce the whole thing in multiple environments on many platforms including AWS, Azure, Oracle Cloud Infrastructure, GCP for testing, development and deployment.

This introduction to Terraform will show how to declaratively build and modify your infrastructure with code that can be versioned, tested and deployed with the rigour of modern release engineering, all the way to Continuous Integration and Deployment.

Duncan Hutty, Oracle Cloud Infrastructure

Duncan Hutty tries to make/find something useful and "Make it Work, Make it Right, Make it Better". Mostly this entails reading and writing code to integrate other people's software and trying to not rant too hard when sharing the vision of code & systems that are reproducible, testable and no more complex than necessary.

Training II

Linux Container Internals 1 & 2

Monday, 11:00 am12:30 pm
Scott McCarty, Red Hat
LISA18: Engineering

Have you ever wondered how Linux Containers work? How they really work, deep down inside? How does sVirt/SELinux, SECCOMP, namespaces, and isolation really work? How does the Docker Daemon work? How does Kubernetes talk to the Docker Daemon? How are container images made?

Well, we will answer these questions and more. If you want a deep technical understanding of containers, this is the lab for you. Join Red Hat engineers as we walk you through the deep, dark internals of the container host and what’s packaged in the container image. These hands on labs will give you the knowledge and confidence it takes to leverage your current Linux technical knowledge and apply it to Containers.

Scott McCarty, Red Hat

At Red Hat, Scott McCarty helps drive the roadmap around container runtimes, tools, and images within the OpenShift Container Platform and Red Hat Enterprise Linux. He also liaises with engineering teams, both at the product and upstream project level, to help drive innovation by using feedback from Red Hat customers and partners as drivers to enhance and tailor container features and capabilities for the real world of enterprise IT.

12:30 pm–2:00 pm

Lunch at the Expo

2:00 pm–3:30 pm

Talks I

Designing for Failure: How to Manage Thousands of Hosts Through Automation

Monday, 2:00 pm2:30 pm

Brandon Bercovich, Uber

LISA18: Architecture

At Uber, we run thousands of services on top many thousands of hosts using Apache Mesos with the Apache Aurora framework. This setup ensures that when a host breaks a service will automatically get rescheduled to another host, but what happens to the host? What happens when a host is still running services but is misconfigured or has a hardware fault that can be affecting the performance of the service. How about when you want to upgrade the Kernel or other software across your fleet. At Uber, we created CLM or Cluster Lifecycle Manager which is used to answer these questions in a safe and automated way. In this talk we will go through the architecture we are using to make this possible and how we are ensuring our actions don't impact services.

Brandon Bercovich, Uber

I've been working in the industry for over 18 years as a Systems Administrator, a DBA, and now a SRE. At Uber my team manages our Compute platform through automation.

Familiar Smells I’ve Detected in Your Systems Engineering Organization and How to Fix Them

Monday, 2:30 pm3:00 pm

Dave Mangot

LISA18: Engineering

Over the course of my career, I’ve had the opportunity to work with a number of organizations on their operational maturity. After doing “systems archeology” a number of times, starting at new organizations, I began recognizing certain signature “smells” that indicated that there was something that could be improved, and often, had a pretty good idea how those situations came to be.

Things like the volume of pager alerts can be indicators of poor signal to noise ratios, or overworked infrastructure, or broken architectures. Things like elaborate change control can be signs of inadequate testing, or lack of automation (as if a review by people unfamiliar with the changes makes it safer). Recovery mechanisms that are never tested are never going to actually work in the case that they are needed except in the most trivial of cases.

There are many such examples with single points of failure, competing change mechanisms, scaling challenges, outsourcing of manual automation (not a typo), badly scoped runbooks, immature monitoring, multi-generational monitoring systems, and more, that are signs that we can do better.

In this talk, we’ll talk about some fun that was had over the years, maturing different infrastructures, learning from failure and success, and how we can take lessons from “mistakes were made” scenarios to increase our performance, lower our MTTR, and help those in the systems engineering organization love their job.

Dave Mangot, N/A

Dave Mangot is the author of Mastering DevOps from Packt Publishing. He’s currently the head of Site Reliability Engineering (SRE) for the SolarWinds Cloud companies and an accomplished systems engineer with over 20 years' experience. He has held positions in various organizations, from small startups to multinational corporations such as Cable & Wireless and Salesforce, from systems administrator to architect. He has led transformations at multiple companies in operational maturity and in a deeper adherence to DevOps thinking. He enjoys time spent as a mentor, speaker, and student to so many talented members of the community.

Code-Yellow: Helping Operations Top-Heavy Teams the Smart Way

Monday, 3:00 pm3:30 pm

Michael Kehoe and Todd Palino, LinkedIn

LISA18: Culture

All engineering teams run into trouble from time to time. Alert fatigue, caused by technical debt or a failure to plan for growth, can quickly burn out SREs, overloading both development and operations with reactive work. Layer in the potential for communication problems between teams, and we can find ourselves in a place so troublesome we cannot easily see a path out. At times like this, our natural instinct as reliability engineers is to double down and fight through the issues. Often, however, we need to step back, assess the situation, and ask for help to put the team back on the road to success.

We will look at the process for Code Yellow, the term we use for this process of “righting the ship”, and discuss how to identify teams that are struggling. Through a look at three separate experiences, we will examine some of the root causes, what steps were taken, and how the engineering organization as a whole supports the process.

Michael Kehoe, LinkedIn

Michael Kehoe is a Staff SRE at LinkedIn who works on building scalable monitoring infrastructure, reliability principles and incident management. Michael previously interned at NASA Ames on their PhoneSat project. Michael's key interests lie in network engineering and automation.

Todd Palino, LinkedIn

Todd Palino is a Senior Staff Site Reliability Engineer at LinkedIn, tasked with keeping some of the largest Kafka and Zookeeper deployments fed and watered. Previously, he was a Systems Engineer at Verisign, developing service management automation for DNS, networking, hardware and operating systems. In his spare time, Todd is the developer of the open source project Burrow, a Kafka consumer monitoring tool, and can be found sharing his experience on both SRE and Apache Kafka at industry conferences and tech talks. He is also the co-author of Kafka: The Definitive Guide, available from O’Reilly Media.

Talks II

Security and Service Availability: Lessons Learned from IDaaS RCAs

Monday, 2:00 pm2:30 pm

Mark Ballew, Okta

LISA18: Engineering

Operating at scale on AWS requires teams to change how escalation was handled. In this talk we will talk about the fruits of RCAs from unexpected incidents and design considerations.

  • Escalation and human resourcing: Who gets the page
  • Encryption at rest: Hitting the limits
  • I have an error correcting modem: How bits can still be flipped

Who should attend:

Operators and engineers who want to hear notes from the front line; how an alert and outage can be analyzed to avoid a tragedy of the commons or "geez that was strange, hope it doesn't happen again" engineering culture.

Mark Ballew, Okta

I have held titles likes systems admin, site reliability engineer, and anything else you might call operations for 20 years. I specialize in computer security and infosec, building secure systems on cloud and bare metal. I have also worked at various companies doing work on identity systems, such as at Wells Fargo, Stormpath, and now Okta.

In my free time, I volunteer to help San Francisco's public transportation system and I am appointed by the mayor to give direct input into how the government should the transportation agency.

How to Be Your Security Team's Best Friend

Monday, 2:30 pm3:00 pm

Emily Cole, Agari Data, Inc.

LISA18: Engineering

Operations teams and Security teams don't always get along, and Security teams have a reputation for being grumpy and saying "no" a lot. However, there are a few simple things you can do (or may already be doing) that will make your job easier, and make your Security team less grumpy. In this session, you will get examples of security best practices as well as tips from someone who has worked in both Ops and Security.

Emily Cole, Agari Data, Inc.

Emily is currently a Senior Security Engineer for Agari Data, Inc., and spends a lot of time thinking about the ways that DevOps and Security intersect. Emily has performed critical organizational roles of security research, incident response, product security, devops engineer, system administrator, tech support, security expert, operations specialist, and project lead. Emily specializes in Unix security and is a co-author of a book on Solaris Security for the SANS Institute, and serves as a Mentor for SANS' CyberTalent Immersion Academy for Women. She currently holds GSEC, GCED, GPPA, GCIH, and ITIL certifications, and is a Certified Scrum Master.

Isolation without Containers

Monday, 3:00 pm3:30 pm

Tyler McMullen, Fastly

LISA18: Architecture

Software Fault Isolation, or SFI, is a way of preventing errors or unexpected behavior in one program from affecting others. Sandboxes, processes, containers, and VMs are all forms of SFI. SFI is a deeply important part of not only operating systems, but also browsers, and even server software.

The ways in which SFI can be implemented vary widely. Operating systems take advantage of hardware capabilities, like the MMU (Memory Management Unit). Others, like processes and containers, use facilities provided by the operating system kernel to provide isolation. Some types of sandboxing even use a combination of the compiler and runtime libraries in order to provide safety.

Each of the methods of implementing SFI have advantages and disadvantages, but we don't often think of them as different options toward a similar end goal. When we consider the growing prevalence of things like edge computing and "Internet of Things", our common patterns start to falter.

In this talk, we'll focus on how sandboxing compilers work. There are important benefits, but also major pitfalls and challenges to making it both safe and fast. We'll talk about machine code generation and optimization, trap handling, memory sandboxing, and how it all integrates into an existing system. This is all based on a real compiler and sandbox, currently in development, that is designed to run many thousands of sandboxes concurrently in server applications.

Tyler McMullen, Fastly

Tyler McMullen is CTO at Fastly, where he’s responsible for the system architecture and leads the company’s technology vision. As part of the founding team, Tyler built the first versions of Fastly’s Instant Purging system, API, and Real-time Analytics. Before Fastly, Tyler worked on text analysis and recommendations at Scribd. A self-described technology curmudgeon, he has experience in everything from web design to kernel development, and loathes all of it. Especially distributed systems.

Training I

Setting Up CI/CD Pipelines with GitLab CI and Jenkins

Monday, 2:00 pm3:30 pm
Aleksey Tsalolikhin, Vertical Sysadmin, Inc.
LISA18: Engineering

Attendees will learn how to implement CI/CD pipelines in two popular tools, Jenkins and GitLab CI.

Aleksey Tsalolikhin, Vertical Sysadmin, Inc.

Aleksey Tsalolikhin is a practitioner in operations of information systems. Aleksey strives to improve the lives of fellow practitioners through effective training in excellent technologies. Aleksey is the principal at Vertical Sysadmin, which provides on-site training and consulting in IT Operations and DevOps.

Training II

Containerize Your App Using Docker

Monday, 2:00 pm3:30 pm
Naga Sowjanya Mudunuri, Heroku
LISA18: Engineering

Have you tried replicating consistent environments in development, staging & production? Have you tried to keep the operational cost low by efficiently using the computing resources? How about scaling your application when you get popular? Containerization is one of the core tenets for operating at scale. Containers enable predictable environments for your deployments in dev/staging/production and adds to developer efficiency. Containerization lets you run your application on any compute resource whether it is AWS, GCE or a Datacenter. In this tutorial you are going to learn about the basics of Docker, a popular containerization technology, and get hands-on experience containerizing an application. You will learn how to Dockerize an application written in Go/Ruby/Node. You do not need to know all these languages. Knowing one of it helps. If you have an existing app in any of these languages you can containerize it in the class or a sample app will be provided for you to download.

Naga Sowjanya Mudunuri, Heroku

Sowjanya is a Senior Software Engineer at Heroku. She found her love and passion for Software Design and Engineering only 5 years ago. Prior to that she worked as a hardware engineer building SystemVerilog Components at Intel & Xilinx. For the past 5 years she has worked as a Full-Stack & DevOps Engineer building applications and services using Ruby, Rails, Go, Docker, Terraform. Today she is an Engineer on the Runtime team at Heroku orchestrating dynos using Go & Ruby. In her free time she is a volunteer mentor at Women Who Code San Francisco.

3:30 pm–4:00 pm

Break with Refreshments at the Expo

4:00 pm–5:30 pm

Talks I

Containers, Containers Everywhere: Deploying Games Globally at Scale

Monday, 4:00 pm4:30 pm

Jennie Lees, Riot Games

LISA18: Engineering

Creating containers and using schedulers forms only part of the picture when it comes to global deployment across a diverse set of infrastructure choices, each optimized for player value over developer speed. This talk will focus on how a game company migrated from Chef to Docker, developing its own internal standards and describing container deployment as code. Attendees will learn how small-scale container orchestration maps to global 24/7 live deployments, and take away several ideas as well as pitfalls when it comes to containers for everything (yes, even persistence).

Jennie Lees, Riot Games

Jennie is a senior software engineer at Riot Games, leading a team focused on simplifying deployment at scale. She's fascinated by the challenges of running infrastructure in a games company.

Containers and Security on Planet X

Monday, 4:30 pm5:00 pm

Michael Jennings, Los Alamos National Laboratory

LISA18: Engineering

Containers and the modern container ecosystem have thoroughly revolutionized the development, deployment, and delivery of web applications and microservices, but in many ways, the world of high-performance computing (HPC)—traditionally the innovators and trailblazers of the scalable system space—has found itself playing catch-up instead of leading the pack as it once did. No longer the target audience for the community of solution developers, the architects and innovators in HPC initially sought ways to tweak, tailor, configure, or even fork the industry-standard container platform (Docker) to address the unique needs of HPC and the scientific community. Faced with the reality that they were on their own to implement something themselves, multiple solutions emerged which dealt with many of the same challenges in fundamentally different ways and with different implications for performance, usability, reliability, scalability...and of course, security!

For system administrators in the unique world of HPC—whether in academia, government, or the technical computing industry—the current view of the container landscape can be incredibly confusing. We'll sift through the hype, dispel some misconceptions, explore the unique use case of containers in HPC, and examine the modern runtime options for high-performance containerized applications in terms of their primary differentiators—their approaches to image management, runtime execution, and security. We'll also look at future developments going on both inside and outside the HPC container arena and discuss how the landscape will be changing over the next few years. Finally we will provide some simple, straight-forward guidance to help sysadmins new to this space understand your options and make the best possible choice for addressing your own HPC container needs to ensure your users' success and your own peace-of-mind!

Michael Jennings, Los Alamos National Laboratory

Michael Jennings has been a UNIX/Linux Systems Administrator and a C/Perl developer for over 20 years and has been author of or contributor to numerous open source software projects including Eterm, Mezzanine, RPM, Warewulf, and TORQUE. Additionally, he co-founded the Caos Foundation, creators of CentOS, and has been lead developer on 3 separate Linux distributions. He currently works as a Scientist at Los Alamos National Laboratory and is the primary author/maintainer for the LBNL Node Health Check (NHC) project. He is also the Vice President of HPCXXL, the extreme-scale HPC users group.

Getting the Most Out of Your Mesos

Monday, 5:00 pm5:30 pm

David Morrison and Sunil Shah, Yelp

LISA18: Architecture

Apache Mesos is a powerful tool for running workloads across a distributed cluster of machines. It has powered Yelp’s production infrastructure since 2014 and runs an increasing variety of workloads, from traditional stateless services to batch and machine learning workloads. In this talk we describe how we use data and analytics to push Mesos to the limits, while providing a better experience for our users and our developers.

David Morrison, Yelp

David R. Morrison is a software engineer working in scheduling and optimization on the Distributed Systems team at Yelp, where he has developed autoscaling code for Yelp’s most expensive compute clusters. Previously, David worked in research and development at Inverse Limit, where he received federal funding from DARPA and Google’s ATAP program. David received his PhD in computer science from the University of Illinois, Urbana-Champaign under the supervision of Dr. Sheldon Jacobson. David has spoken at the INFORMS Business Analytics conference in 2017, at AWS re:Invent 2016, and given multiple presentations at the INFORMS Annual Meetings and other venues.

Talks II

SRE (and DevOps) at a Startup

Monday, 4:00 pm4:30 pm

Craig Sebenik, Split

LISA18: Culture

The Google "SRE book" gives a great explanation of what SRE does. That model works great for LinkedIn or Facebook. But, what happens when you loose some of those economies of scale? How do you implement those paradigms when the company (or team within a larger corporation) is only a couple of dozen people?

In this talk, I will discuss 2 different approaches; a centralized team that supports development versus a more distributed team that include developers themselves.

I have witnessed both approaches at different startups. I will finish up with the lessons learned and the pitfalls implementing either approach at another company.

Craig Sebenik, Split

Currently, an Infrastructure Engineer at Split. I have been at large companies and small ones. I have seen amazing growth (at LinkedIn and NetApp) and a couple completely fall apart. I am passionate about how SRE (and DevOps) can change how online software is developed and managed.

Cross Atlantic: Scaling Instagram Infrastructure from US to Europe

Monday, 4:30 pm5:00 pm

Sherry Xiao, Facebook

LISA18: Architecture

Deploying a service across multiple continents is difficult, especially when you have a stateful service. Facebook now has multiple datacenters across US and Europe, while Instagram infrastructure still remains only in the US. How can we scale Instagram across the ocean? What are the problems we need to solve?

One of the databases Instagram uses heavily is Cassandra. Running Cassandra with too many copies increases the complexity of maintaining this database, not to mention that having the quorum requests travel across the ocean is just... slow. So, we partitioned our dataset! The idea is to have a Cassandra European partition and a US partition, and send the users to their nearest partition.

When we started to put together the plan for deploying Instagram in current European datacenters, we encountered several problems. How do we make sure users have all the data they need stored in the same partition? When one of the European datacenters fails, how do we failover and where do we send that traffic?

This talk will cover:

  • The challenges we had during the infrastructure design and disaster recovery planning
  • How we use social hash to make sure all the data belonging to one user stays in the same partition as much as possible, and how it helps improving cache miss rate
  • The failover plan when one European datacenter fails, including how we shift the traffic around

Sherry Xiao, Facebook

I'm a Production Engineer working on scaling Instagram infrastructure. My team supports all engineering teams at Instagram, and gets involved with a large number of areas like rapidly scaling infrastructure, capacity planning, designing and practicing disaster recovery plans for Instagram.

Delete This: Decommissioning Servers at Scale

Monday, 5:00 pm5:30 pm

Anirudh Ra, Facebook

LISA18: Architecture

Facebook's datacenter footprint has increased significantly; we now have 12 locations across USA and Europe. As these new locations come online, we have had to plan for the end-of-life process: decommissioning server racks and replacing them in a timely and streamlined manner. Until recently, decommissioning a cluster entailed a lot of manual work: service oncalls were ticketed by project managers and then migrated off the old hardware onto new hardware, after which hardware was unplugged and rolled out.

We realized the need for automation that covered all of this. We started with a framework that allows for automated service migration, given a list of retiring machines and a list of replacements. We moved on to an automated process that looks at a decommission schedule and kicks off jobs to drain server clusters on time so that old racks can be taken away and new racks rolled into their place.

With this automated process in place, we have learned lessons and figured out how to minimize the time that old servers spend without services running on them before being rolled out of the datacenter. We are also exploring ways to reuse parts of this framework in other ways to increase efficiency.

Anirudh Ra, Facebook

Customer support tech turned production engineer, Anirudh tries to remember that his job is still about helping people succeed. He builds frameworks for service owners to run their services with minimal bother and enjoys baking bread and reading fiction and histories and fictional histories.

Training I

What I Wish I Knew Before Going On-call

Monday, 4:00 pm5:30 pm
Chie Shu, Dorothy Jung, and Wenting Wang, Yelp
LISA18: Culture

Being a software engineer means owning a production system—you have many users and your company's revenue relying on your products. Firefighting a broken system is a time-sensitive and stressful part of life as an engineer. New engineers entering an on-call rotation may be overwhelmed by this responsibility. How should we act in an emergency? How can we make a system emergency-friendly? In this workshop, we will share how new and future on-call engineers can be successful by guiding participants through exercises to triage real-life engineering emergency scenarios. We will also cover how on-call engineers can share learnings within an organization to prevent future fires.

Chie Shu, Yelp

Chie Shu is a backend Software Engineer at Yelp. She has worked on improving the revenue-critical Ads data pipeline to be more resilient to system failures and designed heuristics used by executives and Product Managers to assess the financial impact. She is a leader for Yelp’s Awesome Women in Engineering support group, and has organized events to foster an inclusive community for incoming women engineers and allies. Chie holds a Bachelor’s degree in Computational Biology from Cornell University.

Dorothy Jung, Software Engineer, Yelp

Dorothy Jung is a Software Engineer with multiple years of on-call experience. At Yelp she has served as a “pushmaster”, managing and monitoring company-wide deployments to production; and as a release engineering deputy, helping to set up CI/CD pipelines within the Ads organization. She was previously at DreamWorks Animation R&D, where she worked on upgrading the studio’s build management tools. Dorothy holds a bachelor’s degree in Computer Science and French from the University of California, Berkeley.

Wenting Wang, Software Engineer, Yelp

Wenting Wang is a Software Engineer with three years of industry experience. She has been on-call for different teams at Yelp: on the BizApp backend team, where she worked closely with mobile developers and monitored mobile user traffic; and on the Ads team, where she currently develops and maintains revenue-critical real-time processing systems. Wenting received her master’s degree in Computer Science from Shanghai Jiao Tong University and was previously a doctoral candidate in Computer Science focusing on distributed systems at the University of Illinois at Urbana-Champaign.

Training II

30 Years of Making Lives Easier: Perl for System Administrators

Monday, 4:00 pm5:30 pm
Ruth Holloway, cPanel, Inc.
LISA18: Engineering

A lot of system administration tasks can be made easier with some “glue” code—and Perl is an excellent glue language, useful for many things. As its creator Larry Wall says, “when is the last time you used duct tape on a duct?”

In this session, attendees will learn a little about Perl, with a focus on its use in system administration. We’ll explore Perl’s mature infrastructure, step through the basics of creating Perl programs, and discuss use cases for system administration tools in Perl.

Ruth Holloway, cPanel, Inc.

Ruth Holloway got into system administration early in her 30-year career, and has spent at least part of her time ever since working to make it easier for herself, and for the people who follow her in the admin's seat. She has been a Perl developer for over 15 years, and currently works for cPanel in Houston, TX.

Ruth is a wife, mother, grandmother, writer, activist, technologist, and mommy to the cutest dog you'll ever meet.

6:00 pm–7:00 pm

Expo Mixer

Sponsored by Apple
Join us at the LISA18 Expo for refreshments, and take the opportunity to learn about the latest products and technologies. Don’t forget to get your Expo passport stamped!

7:00 pm–11:00 pm

Birds-of-a-Feather Sessions (BoFs)

View the full schedule of BoFs on the LISA18 BoFs page.

Tuesday, October 30, 2018

7:30 am–5:00 pm

On-Site Registration and Badge Pickup

7:30 am–9:00 am

Continental Breakfast

9:00 am–10:30 am

Keynote Addresses

Make It Official: In Praise of Official Programs for Diversity & Inclusion

Tuesday, 9:00 am9:30 am

Deb Nicholson, Software Freedom Conservancy

LISA18: Culture

Diversity and inclusion programs with metrics, funding and official support from the top perform better, last longer and are more impactful. Well-articulated goals help projects tout successes, which in turn inspires more participation—so let's make it official!

Once the hard work of choosing strategies to increase diversity and improve inclusion is done, set aside some budget. It's not fair or sustainable to ask employees to do diversity work "on the side." New hiring protocols, outreach programs and promotion systems take time and training to implement if they're going to succeed. Dedicating resources to inclusion programs creates an environment where it is ok for employees to be transparent about the time and effort they're spending, which is crucial for collecting honest metrics.

Making diversity part of your company or project's success story, demonstrates sincerity and commitment that will help you attract more people that share those goals and will gladly work to achieve them.

Deb Nicholson, Software Freedom Conservancy

Deb Nicholson is a free software policy expert and a passionate community advocate. She is the Director of Community Operations at the Software Freedom Conservancy where she supports the work of its member organizations and facilitates collaboration with the wider free software community. She is also a founding organizer of the Seattle GNU/Linux Conference, an annual event dedicated to surfacing new voices and welcoming new people to the free software community. She lives with her husband and her lucky black cat in Cambridge, Massachusetts.

Anatomy of a Crime: Secure DevOps or Darknet Early Breach Detection

Tuesday, 9:30 am10:00 am

Dr. Sarah Lewis Cortes, Salesforce

LISA18: Engineering

Criminals lurk on the darknet, where child abuse forums have exploded in recent years, as well as other crime: intellectual property theft, narcotics, and carding, to name a few. Roman Seleznev convicted in 2017 and now serving 27 years in Seattle, brought darknet carding data breaches to a new level. What role does the darknet play in data breaches? What can operations and infrastructure developers consider when building their systems to help prevent breaches? We review a real life darknet breach, the Home Depot Hack, who's 2017 settlement for the first time requires darknet monitoring. We demonstrate how assumptions that hackers could never reach the RAM of Point of Sale (PoS) devices facilitated some of the largest breaches in history. Credit card information stored briefly in the plain text memory buffers in the PoS systems is just one risk that may not occur in development, or may be dismissed as far-fetched. We present considerations for operations and infrastructure developers when building systems.

Sarah Cortes, Salesforce

Dr. Sarah Lewis Cortes earned her undergraduate degree at Harvard University, studied Forensic Sciences at Boston University Medical School, and holds a PhD in Computer Science, Cybersecurity from Northeastern University, specializing in the Darknet, Privacy and Privacy Law as well as IT Security, topics on which she has published extensively. She conducts training and research with the FBI, the Alameda County Sheriff’s Office Digital Forensics Crime Lab, and other LEAs. Prior to undertaking her PhD, Sarah was SVP for Security, IT Audit and Disaster Recovery at Putnam Investments, an investment management firm with over $1 trillion in assets under management.

Do the Right Thing: Building Software in a Age of Social Responsibility

Tuesday, 10:00 am10:30 am

Jeffrey Snover, Microsoft

LISA18: Culture

In an increasingly software-driven world, our role is more critical than ever. We are literally forging the fabric of the future and we need to stop and ask ourselves what kind of world do we want to live in? Our code impacts real people in real ways. Ethical concerns, social responsibility, and inclusive design are important to the work we do, and addressing them well takes time and resources. Join Jeffrey Snover, Technical Fellow at Microsoft, to learn how modernizing development processes can free you up to focus on building great software, minimizing technical debt, protecting sensitive data, and empowering your users.

Jeffrey Snover, Microsoft

Jeffrey Snover is a Technical Fellow at Microsoft and the Chief Architect for Azure Storage and Edge Cloud Group where he focuses on Azure Stack. Snover is the inventor of Windows PowerShell, an object-based distributed automation engine, scripting language, and command line shell. He was also the Chief Architect for Windows Server. Snover joined Microsoft in 1999 as divisional architect for the Management and Services Division, providing technical direction across Microsoft's management technologies and products.

Training I

Developing Applications with Kubernetes

Tuesday, 9:00 am10:30 am
Sean Dague and Will Plusnick, IBM
LISA18: Engineering

Microservices revolutionized the way we look at app development and is now one of the most popular programming architectures. Now, Docker alongside Kubernetes is changing the way teams look at deployments of these microservices. Kubernetes provides powerful production-grade orchestration for your "Dockerized" microservices.

In this workshop, you'll get an overview of Kubernetes, and what it provides for application development. You'll then go through the process of building and deploying a microservice application on Kubernetes.

This is a hands-on-keyboard lab, everyone should come with a laptop and a desire to learn. Attendees can use minikube locally, or cloud accounts will be provided. We'll cover:

  • Kubernetes basics
  • Building Container Images
  • Deploying the application with Kubernetes
  • Upgrading and scaling the application with Kubernetes
  • Debugging your application in Kubernetes

Sean Dague, IBM

Sean Dague has been an Open Source developer for most of his professional life. He's worked on numerous Open Source projects over the years including SystemImager, OpenHPI, Xen, OpenSim, NFS Ganesha, and OpenStack.

In 2003 he created HV Open, which runs a monthly lecture series on open technology topics. It serves the Hudson Valley of New York state.

He is currently a Developer Advocate at IBM, focusing on the areas of Cloud, Containers, Infrastructure, and Open Source.

Training II

Running Excellent Retrospectives: What Happened?

Tuesday, 9:00 am10:30 am
Courtney Eckhardt, Heroku, a Salesforce company
LISA18: Engineering

Your site’s back up, you’re back in business. Do you have a way to make sure that problem doesn’t happen again? And if you do, do you like how it works?

Heroku uses a blameless retrospective process to understand and learn from our operational incidents. This tutorial will share the process we use and give you a chance to practice analyzing operational problems using the internal and external communications of a real Heroku operational incident. Along the way, we’ll discuss how Heroku developed this process, what issues we were trying to solve, and how we’re still iterating on it.

Courtney Eckhardt, Heroku, a Salesforce company

Courtney Eckhardt first got into retrospectives when she signed up for comp.risks as an undergrad (and since then, not as much has changed as we’d like to think). Her perspectives on engineering process improvement are strongly informed by the work of Kathy Sierra and Don Norman (among others).

10:30 am–11:00 am

Break with Refreshments at the Expo

11:00 am–12:30 pm

Talks I

The History of Logging @ Facebook (Abridged)

Tuesday, 11:00 am11:45 am

KC Braunschweig, Facebook

LISA18: Architecture

Ten years ago Facebook released the first version of Scribe which has formed the basis for our logging infrastructure ever since. Back then logging was mostly about aggregating files full of text but since then “logs” have evolved to include abstract streams of events and blurred the line with messaging and pub/sub systems. Scribe continues to aggregate massive amounts of log data but now also forms the basis for many real time stream processing applications at Facebook. Having a common transport layer allows us to apply new stream processing and data science tools to business or operational data with equal ease.

I'll look back at some of the early design choices that have allowed Scribe to evolve with our needs. This evolution has required us to replace major components and scale up the infrastructure by several orders of magnitude. I'll walk through the steps in this evolution with particular emphasis on our most recent upgrade: replacing the HDFS-based persistence layer with LogDevice and how we accomplished that upgrade with little user impact. 

KC Braunschweig, Facebook

KC Braunschweig has spent the last 6 years as a production engineer working on Facebook infrastructure. He's currently working on distributed coordination services built on Apache Zookeeper. He's previously worked in areas including logging, stream processing, search infrastructure and configuration management.

Container Security

Tuesday, 11:45 am12:30 pm

Daniel Walsh, Red Hat

LISA18: Engineering

This talk examines all of the technologies used to keep containers separate. We will examine concepts of what needs to be considered when containerization is happening. Whether to run your apps in containers, VMs or on bare metal. It will examine technologies like Linux Capabilities, SECCOMP, SELinux, Device Cgroups, Read Only.

I will talk about new container technologies like KATA Containers which use KVM for separation and advances in the User Namespace.

Daniel Walsh, Red Hat

Consulting engineer at Red Hat leading Container Engineering team. Including CRI-O, Buildah, -podman, containers/storage and containers/image. Docker/Moby project contributor. Led the SELinux project, concentrating on the application space and policy development.

Talks II

The Divine and Felonious Nature of Cyber Security (A DevSecOps Tale)

Tuesday, 11:00 am11:45 am

John Willis, SJ Technologies

LISA18: Engineering

Over the past 7 years of the DevOps movement, there has been a healthy tension between behavior, speed, and resilience. Empirical and analytical evidence suggests that when these three are in equilibria the outcomes create organizational performance. With behavior being an anchor of organizational performance. In DevSecOps, the motivation is to embed security patterns and practices into existing DevOps capabilities thereby creating similar equilibria with increased organizational performance. In this session, we are going to observe the IT security practices over the past 10 years that one might suggest having had a metaphorically “felonious” narrative based on the resultant outcomes. In the DevSecOps metaphorically “divine" narrative we will see once again that it’s not all just about technology, however, performance outcomes will again be heavily anchored in our behaviors.

John Willis, SJ Technologies

John Willis is VP of Devops and Digital Practices at SJ Technologies. Prior to SJT, he was a Director at Docker, after the company he co-founded (SocketPlane, SDN for containers) was acquired by Docker. Previous to founding SocketPlane, John was the DevOps Director at Dell, which he joined after the Enstratius acquisition in 2013. He has also held executive roles at Chef and Canonical. John is considered one of the founders of the Devops movement. John has authored 7 IBM Redbooks. He is also co-author the “Devops Handbook” and “Beyond the Phoenix Prearlyeaoject” with author Gene Kim.

How Houghton Mifflin Harcourt Went from Months to Minutes with Infrastructure Delivery

Tuesday, 11:45 am12:30 pm
Robert Allen
LISA18: Culture

By leveraging Apache Mesos and Apache Aurora they were able to deliver those applications while cutting annual costs by six figures and begin the process of empowering engineers to focus on innovative products and not the infrastructure. Embracing common open source practices, agile methods and open source technologies HMH began a journey of transforming how innovation occurs in its engineering organizations. This conversation focuses attention on the journey and provides candid examples of the mistakes and successes along the way as well as the practical insights of changing the hearts and minds of engineers, product owners and management. Ultimately leading to a more productive, creative, agile and happier organization.

The odyssey from "Lift and Shift" operations to a Apache Mesos centric self-service platform that establishes the technology foundations which enables engineers to go from idea to delivery in a matter of minutes has lead to a remarkable evolution in ideas and customer satisfaction. This technology exists for everyone, there is no "secret sauce". The challenges that are often less known is the "people problem" which merits understanding, preparation and sharing at all levels of an organization.

Robert Allen[node:field-speakers-institution]

An automotive technician, cabinet maker, cellist, bassist, cook, software engineer, systems admin and, most recently, Director of Engineering for platform strategy at the global education publisher Houghton Mifflin Harcourt. Robert leads the team at HMHCo known as the Bedrock Technical Services group. His team is responsible for designing, developing and supporting the systems that serve as the current and future of educational services, a unique platform that allows engineers to focus on the things that matter most to students and educators worldwide.

Training I

Up And Running Serverless: A Practical Development Workflow for AWS Serverless

Tuesday, 11:00 am12:30 pm
Tom McLaughlin, ServerlessOps
LISA18: Engineering

This workshop will walk participants through a practical development workflow for AWS serverless systems. It introduces the participants to both serverless architecture and the tools to get up and running. This workshop starts by introducing the audience to a basic monolithic 3-tier serverless web application to familiarize them with the basics of serverless architecture. It proceeds to decompose the application into microservices and introduce them to how Serverless Framework is used to create serverless services. Finally, the participant will be tasked with adding a rudimentary feature to the application as an application of what was learned in the first two modules.

Someone should leave the workshop with an introductory understanding of serverless and the knowledge to continue exploring on their own.

Tom McLaughlin, ServerlessOps

Tom is the founder of ServerlessOps and an experienced operations engineer. He started ServerlessOps after he asked the question, what would he do if servers went away? At a loss for an answer and interested in the future of his profession, he decided to pursue the answer. Tom is actively engaged in promoting serverless infrastructure and engaging with the community to learn more about their thoughts, wants, and concerns are around the topic.

Training II

Building Virtualized Labs for Security Training

Tuesday, 11:00 am12:30 pm
Joseph Maresca
LISA18: Engineering

One of the largest growing parts of information technology is the cybersecurity field. The landscape of the field is constantly changing, meaning those who interface with it on a regular basis are always learning new tips, tools, and tricks to function. This requires many of the people working in the field to have personal labs in which to work on these skills and to test out the newest tools as they become available. These same types of labs are also used by those trying to just increase their overall understanding of the field. This lecture will take a look at the virtualization options available and try to provide some insights into how to build out these labs, what sort of hardware works best, what software options are available, and after you have it working, how do you make it behave in a manner that actually replicates the real world.

Joseph Maresca

Mr. Maresca is a computer security professional who has over 12 years in the field. His work has included time working in certifications & compliance and most recently as a penetration tester in both government and commercial spaces. He currently works as a security consultant providing testing services. He has been a regular presenter at information security and hacker conferences across the US and has also provided training opportunities at some of these same events. He tries to take advantage of his vast experiences to provide a new way to look at and solve common problems that we see today.

12:30 pm–2:00 pm

Lunch at the Expo

2:00 pm–3:30 pm

Talks I

Kafka on Kubernetes: From Evaluation to Production at Intuit

Tuesday, 2:00 pm2:30 pm

Shrinand Javadekar, Intuit, Inc.

LISA18: Architecture

Kubernetes is fast becoming the platform of choice for running distributed, containerized applications in the cloud. It has great features for availability, scalability, monitoring, ease of deployment, a rich set of tools and an extremely fast-growing ecosystem that is making it ever more useful.

However, running stateful applications such as Kafka on Kubernetes is not a common practice today. At Intuit, we took an experimentation and data driven approach for evaluating Kafka on Kubernetes in AWS. In this talk, we provide details of our requirements, the experimental setup, and details of the tests. Tests included basic functional tests for produce/consume of messages, network isolation tests, cross-region produce/consume, performance and scale tests. We’ll also talk about the various tools used during this testing. We focus on the problems we ran into and how we addressed them.

This talk will demonstrate a Kubernetes cluster running Kafka along with the details of how each component is configured. Specifically, the kafka and zookeeper statefulsets, the configmaps used for storing the server.properties used by all brokers, the service objects for enabling access to the brokers and, last but not least, integration with Splunk and Wavefront for logging and monitoring respectively.

Shrinand Javadekar, Intuit, Inc.

Shrinand Javadekar is a "Kubernetes junkie" in the Platform team at Intuit. His primary focus is to make simple, scalable and reliable Kubernetes clusters, the de-facto platform for developing, deploying and running apps. In the past he has been part of large scale file system and virtualization projects at EMC and VMWare. However, his most fun stints have been working on cloud-native platforms and services at startups such as Maginatics and Applatix!

Overcoming the Challenges of Centralizing Container and Kubernetes Operations

Tuesday, 2:30 pm3:00 pm

Oleg Chunikhin, Kublr

LISA18: Engineering

Containers and Kubernetes are becoming the de-facto standard for software distribution, management, and operations. Development teams see and realize the power of these technologies to improve efficiencies, save time, and enable focus on the unique business requirements of each project, but the process of deploying, running, managing, and handling upgrades for Kubernetes is time-consuming and requires significant in-house expertise. InfoSec, infrastructure, and software operations teams, for example, face myriad challenges when managing a new set of tools and technologies as they integrate them into an existing enterprise infrastructure.

In this session, Oleg Chunikhin, CTO at Kublr, will provide an overview of the general architecture of a centralized Kubernetes operations layer based on open-source components such as Prometheus, Graphana, ELK Stack, KeyCloak, etc. From there, he will outline the unique challenges to consider for each of the following critical elements before embarking on a containerization and Kubernetes strategy:

  • Centralized monitoring and log collection; audit
  • Identity and access management
  • Backup and disaster recovery
  • Infrastructure management

By looking at each of these areas, attendees will leave with an understanding of how to ensure successful container and Kubernetes ‘Day 2’ operations.

Oleg Chunikhin, Kublr

Oleg Chunikhin is CTO of Kublr. With nearly 20 years of software architecture and development experience, Oleg is responsible for defining the company’s technology strategy and innovative standards, while forging growth plans. To this end, Oleg has championed the standardization of DevOps in all the company does. He is also committed to driving the adoption of automation and AI applications in innovation-driven industries such as genetics, 3D printing, robotics, neural interfaces, mesh technologies, etc. A popular speaker on all things Kubernetes, Oleg’s speaking credentials include Interop ITX (Las Vegas), Cloud Computing Expo (NYC), and the 21st Cloud Expo (Silicon Valley).

Automating Multi-service Deployments on Kubernetes

Tuesday, 3:00 pm3:30 pm

Michael Hrivnak, Red Hat, Inc.

LISA18: Engineering

Many applications consist of multiple services, such as a database, API service, and frontend. Provisioning them as a single application in Kubernetes can be a challenge, especially if one or more services runs outside your cluster.

The Service Catalog provides a new way to publish, provision, and manage applications on Kubernetes through the use of Service Brokers. The Automation Broker allows users to leverage Ansible Automation to define and orchestrate simple to complex multi-service deployments.

In this session you will learn:

  • How to provision a multi-service application on Kubernetes using the Automation Broker.
  • How to include external service provisioning in your application’s deployment.
  • How to package Ansible Playbooks into a single meta-container for orchestrating the deployment of your application.
  • How to publish your own applications in the Kubernetes Service Catalog.

Michael Hrivnak, Red Hat, Inc.

Michael Hrivnak is a Principal Software Engineer at Red Hat. During his time as Team Lead for the Pulp project, he became involved in solving real-world container orchestration problems. He now works in that domain as part of the Automation Broker project. With experience in both software and systems engineering, he is excited to be writing software for systems engineers. Michael is passionate about open source software, live music, general aviation, and reducing energy consumption.

Talks II

Debugging Linux Issues with eBPF

Tuesday, 2:00 pm2:30 pm

Ivan Babrou, Cloudflare

LISA18: Engineering

This is a technical dive into how we used eBPF to solve real-world issues uncovered during an innocent OS upgrade. We'll see how we debugged 10x CPU increase in Kafka after Debian upgrade and what lessons we learned. We'll get from high-level effects like increased CPU to flamegraphs showing us where the problem lies to tracing timers and functions calls in the Linux kernel.

The focus is on tools what operational engineers can use to debug performance issues in production. This particular issue happened at Cloudflare on a Kafka cluster doing 100Gbps of ingress and many multiple of that egress.

This is also an introductory talk to a training on ebpf_exporter by Alexander Huyhn.

Ivan Babrou, Cloudflare

Ivan is a Performance Engineer at Cloudflare. He spends his days finding performance bottlenecks, fixing them and making sure large chunk of internet runs as fast and as efficiently as possible.

Debugging and Optimizing User Experience with Deep Visual Analytics

Tuesday, 2:30 pm3:00 pm

Sarvesh Nagpal and Toby Walker, Microsoft

LISA18: Architecture

Go beyond standard analytics and dive into a world where you can see your product from your user’s perspective. In the wild west of devices and browsers, what you think you are serving and what your users see can be completely different—devices, plugin, proxies and networks can all change and degrade the quality of your experience. And these problems are expensive. In one case at Microsoft a user experience issue caused millions of dollars in lost revenue.

In this talk we'll share how to close the gap between what the user experience you think you are providing and what your actual users experience. By combining rich client and server telemetry, integrating it with big data visualization and ML-based detection, we show how you detect user experience issues and understand their impact on your product. We’ll share case studies of how we apply this to Microsoft’s large scale product in solving real-world problems and identifying opportunities that drive business growth.

Come join us to learn more and see how this can help solve your own product issues.

Sarvesh Nagpal, Microsoft

Sarvesh leads the performance team at Bing, Microsoft and is passionate about solving complex data problems with rich visualizations. Sarvesh holds a M.S. in Computer Science from Columbia University, NY.

What Breaks Our Systems: A Taxonomy of Black Swans

Tuesday, 3:00 pm3:30 pm

Laura Nolan

LISA18: Engineering

Black swan events: unforeseen, unanticipated, and catastrophic issues. These are the incidents that take our systems down, hard, and keep them down for a long time.

By definition, you cannot predict true black swans. But black swans often fall into categories that we’ve seen before.

This talk examines those categories, and how we can harden our systems against these categories of events, which include unforeseen hard capacity limits, cascading failures, queries of death, hidden system dependencies, various forms of deadlock, and more.

Laura Nolan, N/A

Laura Nolan’s background is in Site Reliability Engineering, software engineering, distributed systems and computer science. She wrote the 'Managing Critical State' chapter in the O'Reilly ‘Site Reliability Engineering’ book, and is co-chair of SREcon18 Europe/Middle East/Africa. Laura is currently enjoying a well-earned sabbatical (and tinkering with some of her own projects) after 15 years in industry, most recently at Google.

Training I

Deploying a Load-Balanced Python Web Application with Amazon Web Services

Tuesday, 2:00 pm3:30 pm
Nicholas Hunt-Walker, Starbucks Corporation
LISA18: Engineering

For many developers, the experience of deploying an application consists of finding a server to host the app, copying it over, and serving using some system like Nginx or Apache.

And that's ok!

It'll work for small applications that aren't meant to handle large loads.

However, if your product is growing or you're adding a new service to an existing, mature application, chances are you're going to need to learn how to scale your web application.

You can do it manually, of course, but Amazon Web Services provides great tools for managing and scaling your applications automatically.

In this tutorial we'll walk through how to set up those services in the AWS environment, enabling your web application to maintain reliability whether big or small.

Nicholas Hunt-Walker, Starbucks Corporation

After 5.5 years working toward a Ph.D. in astrophysics, I left the field to move into teaching software development at a Seattle-area coding school. Since that move, I've been able to have a hand in training over 100 new software developers. I now work as an application developer at Starbucks on the Emerging Technologies team, helping foster innovation and produce proofs of concept for Starbucks Technology. I'm fiercely interested in good mentorship both as a mentee and a mentor of others, and am always looking for ways to grow personally and professionally.

Training II

A Hacker’s View Of Your Network—Analyzing Your Network With Nmap

Tuesday, 2:00 pm3:30 pm
Joe Schottman, N/A
LISA18: Engineering

Nmap is a common open-source network mapping and exploration tool used by attackers to enumerate your servers and services. It also features a Lua-based script engine that can be used to rapidly detect vulnerabilities across your network, helping you stay one step in front of threats. Learn the basics of what your network looks like to attackers and how to use their tool to monitor your perimeter, automate vulnerability scanning, and help keep your systems secure.

Joe Schottman, N/A

Joe is a former web developer and system administrator now working in security but whose job duties still manage to end up being "other duties as required." His professional experience includes online video in higher education, high volume news websites, and financial industries. He is focused on testing and working smarter, increasing collaboration between offensive and defensive staff, and helping people understand the underlying concepts rather than relying on ineffable processes and procedures.

3:30 pm–4:00 pm

Break with Refreshments

4:00 pm–5:30 pm

Talks I

MySQL Infrastructure Testing Automation at GitHub

Tuesday, 4:00 pm4:30 pm

Jonah Berquist and Gillian Gunson, GitHub

LISA18: Engineering

The database team at GitHub is tasked with keeping the data available and with maintaining its integrity. Our infrastructure automates away much of our operation, but automation requires trust, and trust is gained by testing. This session highlights three examples of infrastructure testing automation that helps us sleep better at night:

  • Backups: auto-restores and backup validation. Making backup data accessible to our engineers. What metrics and alerts we have in place.
  • Failovers: how we continuously test our failover mechanism, orchestrator. How we setup a failover scenario, what defines a successful failover, how we automate away the cleanup. What we do in production.
  • Schema migrations: how we ensure that gh-ost, our schema migration tool, which keeps rewriting our (and your!) data, does the right thing. How we test new branches in production without putting production data at risk.

Jonah Berquist, GitHub

Jonah is the Engineering Manager of the Database Infrastructure team at GitHub. He previously worked as a Senior DBA at Twitter and he began his career with databases as a remote DBA for a variety of customers at Blue Gecko. He enjoys looking at graphs and writing scripts to do his job for him.

Gillian Gunson, GitHub

Gillian has been a database infrastructure engineer at GitHub for two years. Her focus has been on performance troubleshooting and observability. Previous employers include Okta, PalominoDB, Oracle, and Disney. Gillian is based in Vancouver, BC, Canada.

Change Management for Humans

Tuesday, 4:30 pm5:00 pm

Tiffany Longworth, Puppet

LISA18: Culture

Unfortunately, rolling out changes for humans is not as easy as merging a pull request. How many times have you seen a new project management tool get rolled out, sometimes with much fanfare and polish, and just not get adopted? Have you ever seen your company announce a new business unit which led to a minor revolt? How scary is the word “reorganization”? If you have ideas on how your group’s processes could be better, want to launch a new tool that will work better as your company grows, or have to adjust the way you do things to meet new regulations, learning the basics of change management will help you to get your plans going, launch them effectively, and ensure they stick around.

I’ve helped implement several tools, projects, and process changes over the years. In this talk, I’ll walk you through the basics of organizational change management with specific examples about

  • Why it’s so hard
  • Points to consider while implementing
  • Tactics to increase adoption
  • Keeping the change alive once launched
  • Pitfalls to avoid

Tiffany Longworth, Puppet

Tiffany Longworth is a Site Reliability Engineer at Puppet. She has launched successful projects large and small, but has also worked on projects that were spectacular failures! She likes using her background as a Marine, her training as an English teacher, automation, and cat gifs as much as possible.

Operations Reform: Tom Sawyer-ing Your Way to Operational Excellence

Tuesday, 5:00 pm5:30 pm

Thomas Limoncelli, Stack Overflow, Inc.

LISA18: Culture

How do you reform your operations organization without demotivating the entire team? They’re proud of the mess they maintain and may be very resistant to change. How can you remove blockers to a devops transformation in operations?

In this talk, Tom will show how the operations of 150 services were reformed without blame or hurt feelings. Like Tom Sawyer convincing others to paint a fence, the team was self-motivated to fix broken processes and reform the system themselves.

Rather than a detailed 500-question audit, the teams were asked to complete a 4-question “gut check” for each subservice. Because the process took minutes instead of weeks, the team could easily produce the result. Because it was a self-evaluation, they were motivated to transform and improve their operational practices. No one felt shamed because it was a self-assessment.

No one felt blamed because services were criticized, not people. The technique makes it easy to judge new projects based on both “theory of constraints”. Shared solutions were easier to identify. The system helped drive training and coaching plans, made visible the business impact of tech debt, and created a data-driven management system where KPIs had failed.

Soon the program was expanded to include more gut checks and more services. This eased the organization into more formal assessments, and provided management roll-ups by service, team and organization.

Lastly, Tom will show how to apply these techniques in any organization.

Thomas Limoncelli, Stack Overflow, Inc.

Tom is the SRE Manager at StackOverflow.com and internationally recognized co-author of books such as the newly updated Volume 1: The Practice of System and Network Administration, 3rd Edition. He is also known for Time Management for Sysadmins and Volume 2: The Practice of Cloud System Administration. He is an internationally recognized author, speaker, system administrator, and DevOps advocate. He’s previously worked at small and large companies including Google, Bell Labs/Lucent, and AT&T. His blog is http://EverythingSysadmin.com and he tweets @YesThatTom.

Talks II

Managing OS Release Transitions at Netflix Scale

Tuesday, 4:00 pm4:30 pm

Edward Hunter, Netflix

LISA18: Culture

Netflix runs over 150k+ instances of Ubuntu inside the Amazon cloud (AWS) supporting hundreds of micro-services to serve over 125m customers worldwide. A small team of engineers is responsible for maintaining and evolving the base OS (BaseAMI) on which every service depends. Over the past year or so we have migrated the majority of the fleet from Ubuntu's Trusty release to Xenial. When Bionic released we were ready to start moving services very shortly after the release date.

Our goals with the migration were simple:

  1. Don't break Netflix
  2. Minimize developer pain/complexity during the migration
  3. Be ready for the next release of Ubuntu as soon as practical after it's release

Meeting these goals required changes to packaging, tools and processes. This talk will reveal some of what we do to manage the OS and allow Netflix to deploy it quickly to thousands of VMs on a daily basis. It will also look at what it takes to stay up-to-date with patches and other changes in the ecosystem all while supporting our users, both internal and external, 24x7.

Edward Hunter, Netflix

Ed is an engineering leader at Netflix responsible for Performance, OS, and Capacity engineering. Prior to Netflix he spent time at Juniper Networks as a director of OS engineering for Junos. He also spent many years at Sun Microsystems managing part of the Solaris team, working in Sunsoft and Sunlabs. He finished his time at Sun as chief of staff to the CTO.

Solving all the Problems with systemd

Tuesday, 4:30 pm5:00 pm

Alvaro Leiva Geisse, instagram

LISA18: Engineering

Often system administrators have to choose one of two options: On one end, traditional service management has a service starting with all privileges, and a full view of your system, and on the other end we have containers, with a restrictive, more controlled view of your system. But, with a modern kernel and systemd, it is no longer one or the other, but you can actually take the best of both approaches and decide which components to apply to your service.

Do you like the concept of packaging dependencies of containers, but also like the idea of sharing the network with your server from a traditional service manager? Do you want to restrict the access to the files on your system from containers, but also want to be able to manage your service from your server like traditional service management allows you? It turns out that you can have it all.

In this presentation I will show all the service techniques to deploy services in Linux that use and abuse systemd, from spinning up a simple service, to actually running your service isolated on a systemd container, and everything in the middle. I'll also show you how to use these features with other traditional techniques, like socket and path activation, service watchdog. scheduling tasks to be executed later on, and what happens when a service goes down.

You already have systemd installed on your server...Why not take full advantage of its capacities?

Alvaro Leiva Geisse, instagram

I love Python, I grew up in a small town in Chile and one weekend, 16 years ago, I had the flu and could not go out. I decided to learn how to code in Python and that was the beginning of the road that would move us all to Northern California so that I could join the Production Engineering team at Instagram. I also like eating and cooking (in that order).

Unikraft: Unikernels Made Easy

Tuesday, 5:00 pm5:30 pm

Simon Kuenzer, NEC Laboratories Europe GmbH

LISA18: Architecture

Unikernels are still terrible to develop and maintain. Applications have to be ported manually to non-standard OSes before gaining from impressive benefits, like superb performance, great isolation, and a small trusted compute. Unikernels can be instantiated in tens of milliseconds or less. They are tiny with low memory footprints of a few MBs or even KBs. They can achieve high network throughput of 10-40 Gb/s with a single CPU core and they enable running thousands of concurrent instances.

We are going to present the Xen/Linux Foundation's open source Unikraft project. Its high level goal is to provide an automated tool to build unikernels without requiring the time-consuming, expert work as today. In addition, Unikraft targets support for multiple "platforms": Xen, KVM, containers and bare-metal. Images are automatically produced for multiple of these platforms without requiring any additional time from users.

Simon Kuenzer, NEC Laboratories Europe GmbH

Simon is a senior systems researcher passionate in virtualization and Unikernels. He works for NEC Labs since the past 6 years and has expertise in NFV and fast packet frameworks like Intel DPDK and Netmap. He is the maintainer of Unikraft, a Unikernel build framework that got released open source as incubator under the Xen project and Linux Foundation. Simon does currently his Ph.D. at University of Liege and received his Diploma degree in computer science with focus on robotics and operating systems at Karlsruhe Institute of Technology (KIT) in Germany.

Training I

De-Obfuscating Talent Acquisition

Tuesday, 4:00 pm5:30 pm
Lori Barfield, RaiseMe
LISA18: Culture

Managers: Is your team’s workload growing faster than you can hire new talent? Do you agonize over which candidate would be best for your team? Do you have trouble persuading candidates to say yes, even when it looks like a perfect fit? Come to our workshop, we can help.

Executives: Do you have a manager under you who has open headcount requisitions so stale, they have birthdays? Send them to our workshop, we can help.

HR Professionals: Do candidate quizzes and cookie cutter requirements complicate the selection process so much, you’re turning off the real talent? Let us help you design a better process, a process that will resonate with you as well as your engineering managers.

The engineering team expansion process is daunting. When you look closely at how the sausage is made at most companies, it’s clear that engineering managers aren’t usually aware of best practices. The few who are lucky enough to receive strong mentoring can leverage that good advice their whole careers; the rest have to learn by making mistakes.

The goal of this class is to give attendees a distillation of best practices for creating new job positions and filling them. Taught by a long time engineering leader and hiring manager, it has a unique analytical approach that will resonate with engineers. This workshop will be the mentor they always needed.

Lori Barfield, RaiseMe

Lori joined her first Internet startup as a senior UNIX system administrator. When that company went public, she was hooked, and helping smaller firms prevail against well established rivals has been her passion ever since. She is currently COO at VerticalSysadmin and a chair at the SCaLE and ShellCon conferences in Southern California. In 2017 she developed RaiseMe, a unique career development effort for nonprofits, which has successfully helped people obtain their first engineering roles in the information security industry.

Lori is a mother of five and enjoys the occasional escape to the woods with her husband.

Training II

Attacking & Auditing Docker Containers

Tuesday, 4:00 pm5:30 pm
Madhu Akula, Appsecco
LISA18: Engineering

Developers and Operations teams (DevOps) have moved towards containers and modern technologies. Attackers are catching up with these technologies and finding security flaws in them. In this workshop, we will look at how we can test for security issues and vulnerabilities in Dockerised environments. Throughout the workshop we will learn how we can find security misconfigurations, insecure defaults and container escape techniques to gain access to host operating system (or) clusters. In the workshop, we will look at real world scenarios where attackers compromised containers to gain the access to applications, data and other assets.

By the end of workshop participants will be able to:

  1. Understand Docker security architecture
  2. Audit containerised environments
  3. Perform container escapes to get access to host environments

The participants will get the following:

  1. A Gitbook(pdf, epub, mobi) with complete workshop content
  2. Virtual machines to learn & practice
  3. Other references to learn more about topics covered in the workshop

Madhu Akula, Appsecco

Madhu is a security ninja and published author. Madhu’s research papers are frequently selected for major security industry conferences including Defcon 26,24 , Blackhat USA 2018, Appsec EU 2018, All Day DevOps (2016, 2017), DevSecCon (London, Singapore, Boston), DevOpsDays India, c0c0n, Serverless Summit, null, etc. His research has identified many vulnerabilities in over 200 organisations including US Department of Homeland Security, Google, Microsoft, Yahoo, Adobe, LinkedIn, Ebay, At&t, Blackberry, Cisco, Barracuda etc. He is co-author of Security Automation with Ansible2 book published by Packt Publishing in December 2017, which is listed as a resource by the RedHat Ansible itself.

6:00 pm–8:00 pm

Conference Reception

Take a break from your laptops, and join us at our annual LISA Reception for dinner, drinks, and the opportunity to connect with other attendees, speakers, and conference organizers.

8:00 pm–11:00 pm

Birds-of-a-Feather Sessions (BoFs)

View the full schedule of BoFs on the LISA18 BoFs page.

Wednesday, October 31, 2018

8:00 am–12:00 pm

On-Site Registration and Badge Pickup

7:30 am–9:00 am

Continental Breakfast

9:00 am–10:30 am

Talks I

How Our Security Requirements Turned Us into Accidental Chaos Engineers

Wednesday, 9:00 am9:30 am

Paul Carleton, Stripe Inc.

LISA18: Engineering

This talk will cover a security focused project that evolved into a chaos injection system.

The system is called “Lifespan Management” and it enforces a lifespan on a cloud hosted VM. After the lifespan expires, the host is terminated, and a replacement is brought up. It has the benefits of making it easier to apply fixes for CVE’s (CVE comes out on day X, we know hosts will age out by day Y), and reducing the value of a compromised machine (“I’ve finally captured a host! It’s being shutdown?? No!”)

This seemed simple enough, but the complexity it uncovered made for a fun, year-long adventure in chaos engineering.

In this talk, I’ll cover the evolution of the system, and some lessons we learned along the way like:

  • All termination API calls are not created equal
  • Zero failing health checks does not mean a host is healthy
  • Answering “Was this the chaos system?” quickly is essential

I’ll also include anecdotes like how it helped with Spectre/Meltdown mitigations, how it mercilessly killed all our kubernetes workers, and how it locked us out of our QA environment.

Paul Carleton, Stripe Inc.

I am an infrastructure engineer on the Cloud team at Stripe and I want to make systems that fix themselves. Outside of work, I write about tech, hike, bike, and spend too much time thinking about where the moon is in relation to Earth.

Securing a Security Company

Wednesday, 9:30 am10:00 am

Patrick Cable, Threat Stack, Inc.

LISA18: Culture

Security is hard. Organizations and businesses tend to sacrifice security for speed, which often leads to undesirable security outcomes for organizations. There's good news though: system engineers, administrators, ops professionals of the world are in a unique spot to make security in their organization better! This is especially true for engineers in smaller organizations and startups, because you don't need to be a Security Person™ to make an organization more secure.

In this talk we'll dig into how a security company thinks and acts about security internally - and the lessons you can take away from it. What did we start with? Where are our pain points? Where are we going? We'll talk about threat models, the pain of constraints, how you can get into trouble with cryptography, the importance of UX, vendor assessments and incident response. At the end, you'll have cultural, engineering, and architecture ideas to take back to your organization and implement.

Patrick Cable, Threat Stack, Inc.

Patrick Cable is a Sr. Infrastructure Security Engineer at Threat Stack. He works to ensure the security of the Threat Stack Platform by collaborating with other departments, implementing security tools, and building new technology to make security easier for everyone in the organization. Prior to working at Threat Stack, Patrick was Associate Staff in the Secure and Resilient Systems Group at MIT Lincoln Laboratory where he worked on improving cloud security in research environments.

We Already Have Nice Things, Use Them!

Wednesday, 10:00 am10:30 am

Sabice Arkenvirr

LISA18: Engineering

An alarming number of companies are using custom tools to provide functionality readily provided by standard, and frequently open source, tools in the systems administrator's toolbox. From deploying servers to services, and maintenance throughout the life of a deployment, there is much to be gained by using an industry standard, and there are very few instances when something entirely custom is actually required.

This talk will delineate the most painful pitfalls of rolling your own tooling. It will also walk through the rare scenarios where custom solutions are acceptable and the best way to approach the process of writing and maintaining these projects.

Sabice Arkenvirr[node:field-speakers-institution]

Sabice is an automation & operations engineer and former consultant based out of San Francisco. She has worked remotely and on site with customers across the industry to help improve their processes and create a clear path forward for maintenance and security of their systems.

Talks II

Five-sigma Network Events (and How to Find Them)

Wednesday, 9:00 am9:30 am

John O'Neil, Edgewise Networks

LISA18: Engineering

Networks are complex systems and too often, despite their best effort, no one knows everything about what's going on. And most of the knowledge about the network is about typical activity. But what about the atypical activity?

There are many reasons to want to find unusual behavior in your network. The biggest reason is that it may be a sign of something new and unexpected—rather than the usual stuff—driving the activity. This doesn't necessarily imply that a network intrusion in underway. There are many other possibilities, both innocuous and dangerous. In any case, though, unusual behavior is probably something you want to know.

There are a variety of tools related to "anomaly detection" or "outlier detection," and this talk isn't about any of them. Instead, this talk is an introduction to writing your own tools for detecting unusual network events. We'll use Python, with some easily available pip installations, and look at some simple approaches to the problem that answer some interesting questions and scale well.

The code will be made available, but the point is not that the code is a complete solution—the point is, rather, that it's a starting point for creating tools that tell netops folks what they want (and need) to know. 

John O'Neil, Edgewise Networks

John O’Neil is the Data Scientist at Edgewise Networks. He writes and designs software for data analysis and analytics, search engines, natural language processing and machine learning. He has a PhD in linguistics from Harvard University, and is the author of more than twenty papers in Computer Science, Linguistics, and associated fields, and has given talks at numerous professional and academic conferences.

Reducing Chaos in Production: Testing vs. Monitoring

Wednesday, 9:30 am10:00 am

Robert Treat, OmniTI

LISA18: Engineering

While no one disputes the good in finding and fixing issues before deploying to production, relying on traditional testing methods in the age of data-intensive, internet scale software has proven to be incomplete. The ability to identify and fix production issues quickly is crucial and requires insight into usage patterns and trends across the entire application architecture. This talk touches on deficiencies of common testing methods, provide real-world examples of discovering odd edge cases with both testing and monitoring, and offers recommendations on metric instrumentation to help companies identify and act on business-affecting problems.

Robert Treat, OmniTI

After a career spent building data-intensive, mission-critical systems at fortune 500 companies across the e-commerce, healthcare, financial services, and SaaS industries, Robert currently spends his days as CEO of OmniTI, a technical services firm focused on providing web services and infrastructure management. A long time open source contributor, he is a published author and speaks at conferences worldwide on topics such as open source, databases, and managing operations at scale. He occasionally blogs at http://xzilla.net.

The Packet Explosion—How a Next Generation Network Device Turned Inadvertently into a Load Generator

Wednesday, 10:00 am10:30 am

Sascha Mühlbach, 1&1 Mail & Media Development & Technology GmbH

LISA18: Engineering

The heterogeneous architecture of modern next generation network equipment consists of multiple high speed packet processors in the form of ASICs and reconfigurable hardware, e.g., FPGAs. While the latter brings great flexibility and performance, on the other hand debugging and fixing firmware issues on that level is quite tricky. Especially, if standard support procedures cannot be applied easily.

Based on a real issue we had in our production environment with one of our network devices, that started to flood the network with millions of packets, I will give a brief overview of the architecture of such equipment and the pitfalls that might exist in case of an issue. To visualize this, I will use the source code of a simple packet processor deployed on an FPGA network development board to demonstrate how a simple bug can generate a line-rate 10G packet flood triggered by a malformed packet.

Sascha Mühlbach, 1&1 Mail & Media Development & Technology GmbH

Sascha Mühlbach currently holds the position of Expert Infrastructure Architect at 1&1 Mail & Media, a sub-division of 1&1 Internet providing free and paid E-Mail services, a news portal and cloud storage to customers worldwide under various brands (e.g., WEB.DE, GMX, Mail.com). In this position he is responsible for the global operations and advancement strategy of the applications and systems infrastructure. Since 2003 he has been working as System Engineer, Architect and Product Manager for different international companies in the telecommunication and hosting sector. He further holds a PhD in Computer Science with the focus on network security.

Training I

Inclusion of Women and Underrepresented Individuals in IT Workplaces

Wednesday, 9:00 am10:30 am
Jeri-Elayne Smith, The Citadel, The Military College of South Carolina
LISA18: Culture

In this experiential training session, individuals will discuss and reflect upon inclusion in the workplace. After taking an individual focus, they will learn a research-based model of Ubuntic Inclusion. The session will end with each attendee creating the start of an "Inclusion Action Plan."

Jeri-Elayne Smith, The Citadel, The Military College of South Carolina

J. Goosby Smith, Ph.D. is an Associate Professor of Management and Assistant Provost for Diversity Equity and Inclusion at The Citadel, The Military College of South Carolina located in Charleston. Smith holds Ph.D. and M.B.A. degrees in Organizational Behavior from Case Western Reserve University, and a B.S. in Computer Science from Spelman College. After a decade in IT for Digital Equipment Corporation (DEC) and other firms, Smith made the transition to a career as a diversity and inclusion professional.

Training II

Systems Tracing and Trace Visualization Lab—Part I

Wednesday, 9:00 am10:30 am
Suchakrapani Sharma, Shiftleft Inc; Geneviève Bastien, École Polytechnique de Montréal; Mohamad Gebai, SUSE
LISA18: Engineering

Trace Visualization Lab is a two-part lab session that introduces participants to system tracing and trace visualizations that are an invaluable technique to understand in-depth system behavior and reach root-cause of problems faster than just CLI tracing modes. The focus of this lab is on post-mortem analysis (the system failed and we want to understand the root cause). The tutorial first introduces attendees to system tracing, trace collection, storage/aggregation/filtering and eventually visualization techniques such as flamecharts, flamegraphs, timeline views, critical flow view (inter-process flow), container and VM views, resource usage etc).

The introductory sessions (part I) also explain why and when is systems tracing required and its role in supporting related performance anlaysis techniques such as distributed tracing, profiling, debugging and service log analysis. Various visual techniques will be shown and hands on activities that showcase how different "views" can help in scenarios such as resource contention, latency analysis would be carried out.

For this lab, primary trace collection will be through LTTng, Perf/Ftrace and primary trace visualization system will be Trace Compass.

Suchakrapani Sharma, Shiftleft Inc

Suchakra is currently a Scientist at ShiftLeft Inc. He completed his PhD in Computer Engineering from École Polytechnique de Montréal where he worked on eBPF and hardware-assisted tracing techniques for advanced systems performance analysis. He has been involved in research on systems performance analysis and has delivered talks at LISA 2017 (San Francisco), Tracing Summit 2015 and 2016 (Seattle and Berlin) where he has demonstrated advanced kernel and userspace tracing tools. He also developed one of the first hardware-trace based VM analysis techniques. More information about him can be found at : https://suchakra.wordpress.com/about

Geneviève Bastien, École Polytechnique de Montréal

Geneviève Bastien is a research associate at the Dorsal Laboratory of École Polytechnique de Montréal. She is a contributor to the Trace Compass and LTTng projects. Her mission is to make the students' life easier when in comes to prototyping cool new analyses and to make sure that their contributions make it into the hands of the end users. She's also involved in Community Wireless Networks in Montréal and in organizations that promote the use of free software in the public sphere.

Mohamad Gebai, SUSE

Mohamad Gebai is a Senior Software Engineer at SUSE. He currently works on software-defined storage solutions, notably Ceph, and has previously been part of the Azure Storage team at Microsoft. He has also been a research associate and a guest lecturer at Polytechnique Montreal. He lectured on Cloud computing technologies.

10:30 am–11:00 am

Break with Refreshments

11:00 am–12:30 pm

Talks I

Engineering an IP Portfolio

Wednesday, 11:00 am11:45 am

Matthew Horton, Arnold & Porter

LISA18: Culture

A strong IP portfolio can be a powerful asset to a company. Of course, it provides economic value to the company or inventors. But a well-orchestrated IP portfolio development effort can also stimulate inventiveness and problem-solving, foster a culture of innovation, and enhance the prestige of the company.

This talk will focus on the ways that engineers can help capture IP value for their organizations. Engineers play a crucial role in the development of a robust IP portfolio. Streamlining communication between the legal and engineering teams and the processes for developing and capturing new developments gives the company a valuable advantage in achieving this goal. This talk will discuss strategies on improving these communication lines, and advise engineers on how to take a more active role in the development of the company's IP portfolio.

In addition, the talk will touch on broad strokes of IP law, glaring issues relevant to the computing industry, like what is “patent-eligible subject matter” after the Supreme Court’s Alice decision, and how Oracle v. Google is affecting the ability to copyright software code. We will also address practical applications and consideration of various portfolio development strategies in the day-to-day setting, such as whether a company should pursue an IP program when the company is primarily an open source shop and how that could be accomplished.

Matthew Horton, Arnold & Porter

Matt Horton counsels clients on strategic intellectual property portfolio development, focusing on business-oriented and industry-oriented portfolio strategies. Matt also supports IP litigation and cybersecurity efforts. He has extensive information technology and cybersecurity experience, serving as a systems engineer and cybersecurity engineer on teams responsible for federal network infrastructures. Matt holds an LLM in Intellectual Property from The George Washington University Law School, and a JD from Tulane University. He also has a Masters in Computer & Information Technology from the University of Pennsylvania, and a BS in Information Technology from George Mason University, with a concentration in Networking and Cybersecurity.

It's Dangerous to Go Alone: Leveling Up a Heroic Team

Wednesday, 11:45 am12:30 pm

Aly Fulton, Sensu

LISA18: Culture

Building and maintaining systems and the teams behind them successfully is a lot like building and maintaining successful raiding teams in massively multiplayer online role-playing games (MMORPGs). Your quest, should you choose to accept it, is to listen to the parallels and use the lessons to strengthen your biggest asset-–your team–-so they can focus on building resilient systems.

This talk uses real-world examples from working in tech and leading gaming communities to highlight:

  • How to best communicate in distributed, remote environments.
  • The necessity of diverse team make-ups and respecting our differences (no "Dev vs Ops").
  • The benefits of "blameless" culture, the freedom to experiment, and the value of learning from the previous failures.
  • Strategies to foster individual and team growth, while recruiting and retaining diverse talent.

Talks II

Keeping the Balance: Load Balancing Demystified

Wednesday, 11:00 am11:45 am

Murali Suriar, Google, and Laura Nolan

LISA18: Architecture

Can you explain the entire path that an IP packet takes from your users to your binary? What about a web request? Do you understand the tradeoffs that different kinds of load balancing techniques make? If not, this talk is for you.

Load balancing is hard, and it is made up of many disparate technologies. It cuts across network, transport and application layers. We'll describe different flavours of load balancing (network, naming, application) and how they work.

We will then discuss example use cases, and which load balancing approaches are most appropriate in each case. We'll also relate these to several design patterns for high-availability services that have developed over the years. Finally, we'll relate the techniques we've been discussing to well-known open source technologies and to the major cloud load balancing services.

Murali Suriar, Google

Murali Suriar is lapsed computer science graduate, turned network engineer, turned SRE. Currently working at Google running cluster filesystem and locking services. Left Google to get on a boat. Got bored and came back.

Laura Nolan, N/A

Laura Nolan’s background is in Site Reliability Engineering, software engineering, distributed systems and computer science. She wrote the 'Managing Critical State' chapter in the O'Reilly ‘Site Reliability Engineering’ book, and is co-chair of SREcon18 Europe/Middle East/Africa. Laura is currently enjoying a well-earned sabbatical (and tinkering with some of her own projects) after 15 years in industry, most recently at Google.

Apache Kafka and KSQL in Action: Let’s Build a Streaming Data Pipeline!

Wednesday, 11:45 am12:30 pm

Robin Moffatt, Confluent

LISA18: Engineering

Have you ever thought that you needed to be a programmer to do stream processing and build streaming data pipelines? Think again!

Apache Kafka is a distributed, scalable, and fault-tolerant streaming platform, providing low-latency pub-sub messaging coupled with native storage and stream processing capabilities. Integrating Kafka with RDBMS, NoSQL, and object stores is simple with the Kafka Connect API, which is part of Apache Kafka. KSQL is the open-source SQL streaming engine for Apache Kafka, and makes it possible to build stream processing applications at scale, written using a familiar SQL interface.

In this talk we’ll explain the architectural reasoning for Apache Kafka and the benefits of real-time integration, and we’ll build a streaming data pipeline using nothing but our bare hands, the Kafka Connect API, and KSQL.

Gasp as we filter events in real time! Be amazed at how we can enrich streams of data with data from RDBMS! Be astonished at the power of streaming aggregates for anomaly detection!

This will be a practical talk, after which attendees will have a clear idea of the power of stream processing, and how to get started with it using the open-source Apache Kafka and KSQL projects.

Robin Moffatt, Confluent

Robin is a Developer Advocate at Confluent, as well as an Oracle ACE Director and Developer Champion. His career has always involved data, from the old worlds of COBOL and DB2, through the worlds of Oracle and Hadoop, and into the current world with Kafka. His particular interests are analytics, systems architecture, performance testing and optimization. He blogs at http://cnfl.io/rmoff and http://rmoff.net/ (and previously http://ritt.md/rmoff) and can be found tweeting grumpy geek thoughts as @rmoff. Outside of work he enjoys drinking good beer and eating fried breakfasts, although generally not at the same time.

Training I

How to Write Effective Training

Wednesday, 11:00 am11:45 am
Branson Matheson, Cisco
LISA18: Engineering

Ever wanted to teach a class but didn't know how to get started? This presentation will walk students through an organized process that can help take a class idea .. develop an outline, class materials, exercises and finally a proposal suitable for a conference or event. We will discuss how to use idea construction to break an objective into component parts and then discuss different methods for teaching those components so that a student can grasp the concept and apply it. We will write simple workshop exercises that then help reinforce the objective and give students enough information to be able to take the idea back to their environment and apply it. We'll then go over general proposal requirements and some tips on how to write a winning submission.

Branson Matheson, Cisco

Branson is a 30-year veteran of system architecture, administration, and security. He started as a cryptologist for the US Navy and has since worked on NASA shuttle and aerospace projects, TSA security and monitoring systems, secure mobile communications, and Internet search engines. He has also run his own company while continuing to support many open source projects. Branson speaks to and trains sysadmins and security personnel world wide; and he is currently a senior technical lead for Cisco Cloud Services. Branson has several credentials; and generally likes to spend time responding to the statement "I bet you can't...."

How to Give Effective Training

Wednesday, 11:45 am12:30 pm
Branson Matheson, Cisco
LISA18: Culture

As conference attendees, we have respect for those that stand before us to give forth upon topics diverse and engaging. It's always been said that public speaking is a challenge, and infront of technical peers it can seem more difficult but it doesn't have to be. We will discuss taking a training course and developing it into an entertaining presentation that will facilitate the learning process and leave a lasting impression upon attendees. We will discuss several methods for preparing to give a presentation, ways to improve existing material and tips and techniques for enhancing the experience for both students and instructor. Shakespeare said "the play's the thing" and while you won't be ready to perform as Elphaba, you will have a toolkit of methods and thoughts to improve your presentations.

Branson Matheson, Cisco

Branson is a 30-year veteran of system architecture, administration, and security. He started as a cryptologist for the US Navy and has since worked on NASA shuttle and aerospace projects, TSA security and monitoring systems, secure mobile communications, and Internet search engines. He has also run his own company while continuing to support many open source projects. Branson speaks to and trains sysadmins and security personnel world wide; and he is currently a senior technical lead for Cisco Cloud Services. Branson has several credentials; and generally likes to spend time responding to the statement "I bet you can't...."

Training II

Systems Tracing and Trace Visualization Lab—Part II

Wednesday, 11:00 am11:45 am
Suchakrapani Sharma, Shiftleft Inc; Geneviève Bastien, École Polytechnique de Montréal; Mohamad Gebai, SUSE
LISA18: Engineering

Trace Visualization Lab is a two-part lab session that introduces participants to system tracing and trace visualizations that are an invaluable technique to understand in-depth system behavior and reach root-cause of problems faster than just CLI tracing modes. The focus of this lab is on post-mortem analysis (the system failed and we want to understand the root cause). The tutorial first introduces attendees to system tracing, trace collection, storage/aggregation/filtering and eventually visualization techniques such as flamecharts, flamegraphs, timeline views, critical flow view (inter-process flow), container and VM views, resource usage etc).

The advanced sessions (part 2) cater to views in Trace Compass that support analysis of containerized workloads, VMs, IRQs and creating custom views in Trace Compass. The session also challenges users to solve a scenario and hunt bugs in real workloads that use all visual tools and techniques in tandem. Attending "Trace Visualization Lab - Part I" is a pre-requisite for this session.

For this lab, primary trace collection will be through LTTng, Perf/Ftrace and primary trace visualization system will be Trace Compass.

Suchakrapani Sharma, Shiftleft Inc

Suchakra is currently a Scientist at ShiftLeft Inc. He completed his PhD in Computer Engineering from École Polytechnique de Montréal where he worked on eBPF and hardware-assisted tracing techniques for advanced systems performance analysis. He has been involved in research on systems performance analysis and has delivered talks at LISA 2017 (San Francisco), Tracing Summit 2015 and 2016 (Seattle and Berlin) where he has demonstrated advanced kernel and userspace tracing tools. He also developed one of the first hardware-trace based VM analysis techniques. More information about him can be found at : https://suchakra.wordpress.com/about

Geneviève Bastien, École Polytechnique de Montréal

Geneviève Bastien is a research associate at the Dorsal Laboratory of École Polytechnique de Montréal. She is a contributor to the Trace Compass and LTTng projects. Her mission is to make the students' life easier when in comes to prototyping cool new analyses and to make sure that their contributions make it into the hands of the end users. She's also involved in Community Wireless Networks in Montréal and in organizations that promote the use of free software in the public sphere.

Mohamad Gebai, SUSE

Mohamad Gebai is a Senior Software Engineer at SUSE. He currently works on software-defined storage solutions, notably Ceph, and has previously been part of the Azure Storage team at Microsoft. He has also been a research associate and a guest lecturer at Polytechnique Montreal. He lectured on Cloud computing technologies.

Hidden Linux Metrics with Prometheus eBPF Exporter

Wednesday, 11:45 am12:30 pm
Alexander Huynh, Cloudflare, and Ivan Babrou
LISA18: Engineering

While there are plenty of readily available metrics for monitoring Linux kernel, many gems remain hidden. With the help of recent developments in eBPF, it is now possible to run safe programs in the kernel to collect arbitrary information with little to no overhead. A few examples include:

  • Disk latency and io size histograms
  • Run queue (scheduler) latency
  • Page cache efficiency
  • Directory cache efficiency
  • LLC (aka L3 cache) efficiency
  • Kernel timer counters
  • System-wide TCP retransmits

Practically any event from "perf list" output and any kernel function can be traced, analyzed and turned into a Prometheus metric with almost arbitrary labels attached to it.

If you are already familiar with BCC tools, you may think if ebpf_exporter as bcc tools turned into prometheus metrics.

In this tutorial we’ll go over eBPF basics, how to write programs and get insights into a running system.

https://github.com/cloudflare/ebpf_exporter

Ivan Babrou, Cloudflare

Ivan is a Performance Engineer at Cloudflare. He spends his days finding performance bottlenecks, fixing them and making sure large chunk of internet runs as fast and as efficiently as possible.

12:30 pm–2:00 pm

Conference Luncheon

2:00 pm–3:30 pm

Talks I

Mastering Near-Real-Time Telemetry and Big Data: Invaluable Superpowers for Ordinary SREs

Wednesday, 2:00 pm2:30 pm

Ivan Ivanov, Netflix

LISA18: Engineering

One of fundamental requirements for typical SRE team is the capability to have solid operational insights into the holistic state of the supported system. This usually involves collecting, aggregating, correlating, visualizing and reacting on data generated by diverse set of data sources.

As part of this talk I will go through the high level approach, sample data sources, and some implementation details, used by the Netflix Open Connect CDN Reliability engineering team while supporting the infrastructure and services hosted on thousands of physical servers and providing streaming video delivery for hundreds of millions of clients.

I will cover some samples of the usage of Hive, Presto, Spark, Elasticsearch, Tableau, and Netflix developed tools for monitoring, alerting, debugging, long term analysis, planning, etc. I will also speak about the benefits of correlating detailed data from server and client reported telemetry.

For clarity—this is not a talk about how to build, implement, develop or support Big Data or near real time telemetry systems. This is all about how you can use them as a platform and a powerful toolset for making your operations team stronger.

Ivan Ivanov, Netflix

Ivan is a Senior CDN Reliability Engineer on the Netflix Open Connect team. He has been designing, deploying, supporting, and optimizing online services on a global scale in various operations roles for the last 17+ years. He is focusing on service reliability, availability, scalability, quality of experience (QoE), service optimizations, investigating and troubleshooting server/client/network/application related issues. Prior to joining Netflix Ivan was Principal Service Engineer at Microsoft working on Windows Update, Microsoft.com and Windows Store.

Datastore Axes: Choosing the Scalability Direction You Need

Wednesday, 2:30 pm3:00 pm

Nicolai Plum, Booking.com Ltd

LISA18: Architecture

This talk will discuss the different orthogonal axes that data storage systems scale up along. Storage system designs can grow successfully in one or more of read rate, write rate, data size and data complexity. They cannot grow in all of these directions; modern highly scalable storage systems are a compromise. This talk will discuss the restrictions of hardware that force the need for smart software, then describe how to scale up an RDBMS and NoSQL datastores. It will cover strong and weak points of a variety of system architectures with examples of how they succeeded and failed in operational use, as well as the types of solution that can be applied to solve challenges in organisational objectives that need scalability along particular axes.

Nicolai Plum, Booking.com Ltd

Currently working as a Database Engineer in the Database Engineering Team of Booking.com, previously systems administration and systems architecture positions at Booking.com and Systems Administrator positions at UUNET UK.

Make Your System Firmware Faster, More Flexible and Reliable with LinuxBoot

Wednesday, 3:00 pm3:30 pm

Andrea Barberio and David Hendricks, Facebook

LISA18: Engineering

UEFI has largely replaced the traditional BIOS in servers, but it is still often limited and has bugs which can take a long time to be fixed by vendors. LinuxBoot, an open source effort, offers a solution to these and other problems by embedding Linux into a machine's firmware, and using it to initialise hardware and start the boot process. Amongst other advantages, this provides significantly faster boot time, enables us to more securely netboot systems over HTTPS and gives access to real debugging tooling. We will talk about the advantages of running LinuxBoot for reliability, speed and flexibility, and how you can try it today.

Andrea Barberio, Facebook

Production Engineer at Facebook Dublin.

Currently working on LinuxBoot and Provisioning services. Previously worked on Facebook's DNS infrastructure and AWS's network monitoring systems as a software engineer.

My interests span from network analysis and security, to reverse code engineering, from systems lifecycle to firmware automation. More about me at https://insomniac.slackware.it.

David Hendricks, Facebook

David ("dhendrix") has been involved with open source firmware and tools for many years, first as an intern at Los Alamos National Laboratory and later as a software engineer at Google working on ChromeOS and at Facebook working on Open Compute Project and Telecom Infrastructure Project. He is working to address challenges of development, deployment, maintenance, and security of system firmware at scale.

Talks II

Modern HTTP Routing

Wednesday, 2:00 pm2:30 pm

Sandor Szücs, Zalando SE

LISA18: Engineering

Modern HTTP routing should support traffic switching, provide fine granular visibility, add resiliency patterns, be extensible, and should be easily configured by development teams. At Zalando SE, we run more than 80 Kubernetes clusters and developed open source tools to support the mentioned features. One core component, skipper, we internally call the swiss army knife of HTTP. It provides filters to modify HTTP data, headers, and body. It allows us to specify route matching with predicates. Using these principles we can support deployment patterns like blue-green, shadow traffic, or A/B tests.

Shadow traffic is one less known feature that enables users to test new architecture with production traffic, without disrupting production.

Modern l7 software load balancers have to be simple to scale, provide visibility in depth, and enhance resiliency. Retries, circuit breaker, or rate limits are interesting cases that can support your applications being more resilient to failure. How to do rate limits in a growing and shrinking cluster, that means changing number of replicas in the backend and also load balancer instances?

The talk will show a modern HTTP router and it's features and show some practical use cases from our production setup that can be reused by the audience.

Sandor Szücs, Zalando SE

Sandor Szücs builds infrastructure software with Kubernetes, that enable feature teams to be productive. He strongly believes in devops and open source. He likes to code Go and focus on the Ingress part of Kubernetes. Sandor was formerly a System Engineer managing production systems in baremetal datacenters with LXC, Puppet, hardware load balancers and storage systems.

Taking Over & Managing Large Messy Systems

Wednesday, 2:30 pm3:00 pm

Steve Mushero, ChinaNetCloud

LISA18: Engineering

People love to build new systems from scratch, but the real world is full of large & messy systems with limited documentation, manual processes, huge technical debt, and a myriad of problems. These often need to be taken over, stabilized, managed, optimized & upgraded for the long-term, usually without downtime, a process can take a week, a month, or a year. This is a large-scale Managed Service Provider's look at their last decade of the tools and processes to do this on 1, 10, and 100+ million-user systems on the ever-chaotic Chinese Internet.

Steve Mushero, ChinaNetCloud

Steve Mushero is CEO of Ops Platform provider OpsStack.io, and CEO of ChinaNetCloud, China's first Internet Managed Service Provider, AWS Partner, and manager of hundreds of large-scale (up to hundreds of millions of users each) systems. He's previously been CTO in a variety of organizations in Silicon Valley, Seattle, New York, and around the world.

Reborn in IT

Wednesday, 3:00 pm3:30 pm

Lori Barfield, RaiseMe

LISA18: Culture

Some of the most talented people we work with come into the IT profession orthogonally, after starting out in different fields at the beginning of their careers. Also, seasoned engineers who have always been in IT often find themselves in a rut after a few years, and want to shift to a different industry vertical. Making the move into management or out of management are similar challenges. These are all re-born candidates, and they are uniquely qualified to handle some of the hardest jobs in IT. Yet, they all face the same challenge: How to get picked for their dream role if they are coming from a different background. First they have to do battle with the notoriously cookie-cutter hiring guidelines in the IT Industry.

Based on a successful non-profit career development program in Southern California, this talk addresses the big challenges with making IT engineering dreams come true:

  • Which certifications and training programs are the most respected
  • What you should and shouldn't do with your resume
  • What types of roles and employers you should target
  • How to help recruiters help you
  • Sample language for hiring managers trying to justify a special placement
  • How LUGs and Conferences help

Lori Barfield, RaiseMe

Lori joined her first Internet startup as a senior UNIX system administrator. When that company went public, she was hooked, and helping smaller firms prevail against well established rivals has been her passion ever since. She is currently COO at VerticalSysadmin and a chair at the SCaLE and ShellCon conferences in Southern California. In 2017 she developed RaiseMe, a unique career development effort for nonprofits, which has successfully helped people obtain their first engineering roles in the information security industry.

Lori is a mother of five and enjoys the occasional escape to the woods with her husband.

Training I

Test Infrastructure Driven Design

Wednesday, 2:00 pm3:30 pm

Joaquin Menchaca, RocketLawyer

LISA18: Architecture

Many organizations struggle to eliminate tech debt with existing bubble gum and scripts or change configuration code, or look toward adopting immutable production patterns with Docker and Kubernetes. The question is how to do this and build confidence that when modules or whole systems are swapped out everything will still work.

Joaquin Menchaca, RocketLawyer

Joaquin started in the industry straight out of highschool at Apple Computer as a QA Engineer and later a Tool Developer. As QA roles transformed, he moved into information systems as system administrator, systems engineer, and now a Senior DevOps Engineer (Yes, he knows DevOps is a philosophy). Currently, Joaquin is partnered with RocketLawyer, helping them transform their organization to apply DevOps principles that will break down silos and foster collaboration.

Training II

Linux Systems Troubleshooting

Wednesday, 2:00 pm3:30 pm
Thomas Uphill, Narrabilis
LISA18: Engineering

In the days of immutable infrastructure, what's the point of troubleshooting anything?

In my experience, when there is a problem with the image/application/container, the immutable infrastructure will just keep recreating the problem. Knowing how to diagnose common problems is still an important skill. Moreover, the machines that control images and containers are longer running than the containers they run, problems on these machines still need to be found and fixed.

In this tutorial we will look at the problems seen with Linux systems. We'll start with how various subsystems work: networking; filesystems; users/groups; and permissions. We'll then move on to look at tools to inspect running systems and of course strace. The focus will be on "off the shelf" tools and how to use them.

Thomas Uphill, Narrabilis

Thomas is a veteran System Administrator who has recently switched to a development role. He's the author of several books on Puppet and has spoken at past LISA conferences on a variety of topics. His primary work environment is Linux and he's a VIM user.

3:30 pm–4:00 pm

Break with Refreshments

4:00 pm–5:00 pm

Keynote Addresses

Serverless Ops: What to Do / This is What We Do, When the Server Goes Away

Wednesday, 4:00 pm4:30 pm

Tom McLaughlin, ServerlessOps

LISA18: Engineering

With the rise of serverless architecture, many of the common day-to-day operations tasks will change dramatically, if not disappear completely. We as Operations professionals will be challenged to redefine our roles and responsibilities within the technology organization as serverless abstracts away the server and its respective OS to cloud service providers. No stranger to this scenario, we will not only be tasked with solving these engineering obstacles introduced by the new serverless paradigm, but we will also need to prove our value to the business in the face of a changing technology landscape… again.

This is a combination professional/cultural and technical talk. We’ll start by discussing the disruption that serverless presents to operations and why. While DevOps and public cloud are becoming commonplace, serverless is the beginning of a new disruption cycle. We need to understand why serverless is disruptive and learn from the lessons of the past.

The talk will continue on to discuss the value of operations work and understanding the relationship between work and value. Not all work has the same value and we need to understand this so we prioritize the best use of our time.

Finally I will walk through the current state of serverless engineering and tools, and show how and where we fit in. For our career longevity and security we need to understand how we fit.

A few areas I’ll discuss include:

  • DevOps and public cloud… And how serverless is starting a new disruption cycle 10 years later.
  • Understanding and determining the value of your work
  • Moving up the value chain and closer to customer success metrics.
  • Team reorganization to better align with business success.
  • Serverless architectural decision making
  • Performance management and cost containment
  • Failure monitoring and service handling
  • Security risk and concerns

If you’re an Operations engineer and you had all your host and OS related work removed from you, would you know what to do to stay busy and demonstrate your value to your organization?

Tom McLaughlin, ServerlessOps

Tom is the founder of ServerlessOps and an experienced operations engineer. He started ServerlessOps after he asked the question, what would he do if servers went away? At a loss for an answer and interested in the future of his profession, he decided to pursue the answer. Tom is actively engaged in promoting serverless infrastructure and engaging with the community to learn more about their thoughts, wants, and concerns are around the topic.

5:00 pm–5:15 pm

Closing Remarks

Program Co-Chairs: Brendan Gregg, Netflix, and Rikki Endsley, Opensource.com