Conference Program

FBAR has enabled Facebook's production engineering teams to automate break/fix responses to many of the single-host events, such as hardware failures or application daemon crashes, that occur on a daily basis, leaving engineers to focus on larger, more complex and interesting problems.

To provide more opportunities and time for engineers to take on more meaningful work, the team developed automated responses for the less frequent, but larger-scale proactive work necessary to maintain a healthy infrastructure. This work takes on different shapes, including top-of-rack switch replacements, disruptive BIOS/firmware updates, or work on power supply (backup or primary).

This talk will use a (fictitious) example of a set of racks undergoing maintenance to give an overview of how the automation provided by FBAR to handle single-host repairs was expanded to cover proactive maintenance work, including an explanation of the automated maintenance process, the API engineers implement to automate the work, and the way it interfaces with humans when automation won't (or fails to) work. It will also include a few small lessons we learned along the way, and explain why a simple approach covers much of the use cases with few drawbacks.

Romain is a manager in Facebook's Production Engineering organization, working on the team that maintains Facebook's Auto Remediation tool (FBAR). He originally came in to Facebook as part of the Site Reliability Operations team and has spent the last five years focused on safely automating operational tasks. Most recently, this has taken the shape of creating new tooling that allows us to automate maintenance affecting multiple machines (and multiple racks) at a time. The team has spent the last two years refining the process and keeping automation simple while covering the majority of use cases.

Backblaze is not a traditional unicorn or unicorn-aspirant. We take a bootstrapped and simple low-cost approach to operations that differs from the current cloud-based trends. This has some key advantages in terms of not requiring heavy VC backing and all that entails, but also presents a different set of challenges. I’ll be describing some of those challenges from the point of view of someone who recently moved from large scale and is now adapting to a smaller scale.

Elliott Sims is a Senior Sysadmin at Backblaze. Previously, he worked at Facebook from 2009-2015 building and maintaining infrastructure through exponential growth.

Instagram deploys code 40 times a day to its fleet of thousands of webservers and userbase of 400M monthly active users automatically when engineers land changes. This talk describes the iterative approach we took to building this system, the problems we faced along the way, the solutions we implemented, and the key principles which enable this to work. These principles include a reliable and fast test suite automatically run during code review and before and after landing; an automated canary to detect significant breakages before being widely deployed; good visibility of and stop mechanisms for the automation; good site monitoring and alarming; and a fast rollback mechanism.

The rise of utility computing has revolutionized much about the way organizations think about infrastructure and back-end serving systems compared to the ""olden days"" of dedicated physical data centers. However, in the final analysis, success is still driven by meeting your SLAs. If services are up and sufficiently performant, you win. If not, you lose.

In the traditional data center environment, fighting the uptime battle was typically driven by a philosophy I call ""Operational Materialism."" The primary goal of OM is preventing failures at the infrastructure layer, and mechanisms for making this happen are plentiful and well-understood, many of which boil down to simply spending enough money to have at least N+1 of anything that might fail and create significant downtime as a result. Redundant power supplies, NIC bonding, replicated SANs, and hot-standby servers are some of the common artifacts of an OM world.

In the cloud, however, Operational Materialism cannot succeed. Although the typical cloud provider tends to be holistically reliable, there are no guarantees that any individual virtual instance will not randomly or intermittently drop off the network or be terminated outright. Yet we still need to keep our services up and running and meet our SLAs, and thus we need a different mindset that accounts for the fundamentally opaque and ephemeral nature of the public cloud.

In this talk, I will present an alternative to OM, a worldview that I refer to as "Operational Buddhism." Like traditional Buddhism, OB has Four Noble Truths:

1. Cloud-based servers can fail at any time for any reason.

2. Trying to prevent this server failure is an endless source of suffering for DBAs and SREs alike.

3. Accepting the impermanence of individual servers, we can focus on designing systems that are failure-resilient, rather than failure-resistant.

4. We can escape the cycle of suffering and create a better experience for our customers, users, and colleagues.

To illustrate these concepts with concrete examples, I will discuss how configuration management, automation, and service discovery help us to practice Operational Buddhism at Pinterest for both stateful (MySQL, HBase) and stateless (web) services. Moreover, as our path is not the only road to infrastructure enlightenment, I'll also talk about some of the roads not taken, including the debate over Infrastructure-as-a-Service (IaaS) vs. Platform-as-a-Service (PaaS).

Ernie Souhrada is a database engineer on the SRE team at Pinterest where his current focus is on improving the performance and operational efficiency of a petabyte-scale hybrid deployment of MySQL, HBase, and Redis. Over the past two decades, Ernie has worked in almost every aspect of information technology, from network engineering and software development to systems administration and information security. When not slinging data or thinking about infrastructure, he can be found on a ski slope, at a sushi bar, or in search of the next great psytrance track. Ernie holds a B.S. in mathematics and a B.A. in political science from Arizona State University.

The systems world has wholly accepted the DevOps way of thinking. However, the network world has not started this, mainly due to a lack of knowledge and cultural resistance to change. This presentation tries to show the network world how to start the DevOps change, in a non threatening manner.

Leslie Carr is currently on the board of directors at SFMIX and happily running away from the responsibilities of day jobs. Before becoming a funemployed bum, Leslie was a devops engineer at Cumulus Networks. Previous to that, she was on the production side of the world at many large websites, such as Google, Craigslist, and Wikimedia. She is a lover and user of open source and automation, and she dreams of robots taking over all of our jobs one day.

It can be easy to come up with a TCO analysis that would challenge any public cloud and make you think, "let's go in-house!" What are the challenges and is it really worth it? The TubeMogul Operation team went thru the technical challenges at building a private cloud. In this presentation you will learn how the team went from a R&D to an automated deployment of a bare-metal servers to finally migrate a large workload from a Public Cloud to its own Private Cloud infrastructure. We will detail how the team dealt with unexpected issues and also how we chose the hardware, estimated capacity, stay cost effective, improve overall performance of the system, and bring better control and visibility.

This talk will cover the technical detail of:

• Evaluating OpenStack, Building and automating a CI environment for a mix of bare metal and cloud servers.  
• What are the network limitations of OpenStack and how we creatively leverage VLANs to handle large packet per seconds. 
• How to efficiently monitor your cloud infrastructure 
• Find quickly your bottlenecks 
• What we missed and should be consider before moving in house 
• Lesson Learned and Post Cost Analysis

Nicolas Brousse is Senior Director of Operations Engineering at TubeMogul(NASDAQ: TUBE). The company's sixth employee and first operations hire, Nicolas has grown TubeMogul's infrastructure over the past seven years from several machines to over two thousand servers that handle billions of requests per day for clients like Allstate, Chrysler, Heineken and Hotels.com.

Adept at adapting quickly to ongoing business needs and constraints, Nicolas leads a global team of site reliability engineers and database architects that monitor TubeMogul's infrastructure 24/7 and adhere to "DevOps" methodology. Nicolas is a frequent speaker at top U.S. technology conferences and regularly gives advice to other operations engineers. Prior to relocating to the U.S. to join TubeMogul, Nicolas worked in technology for over 15 years, managing heavy traffic and large user databases for companies like MultiMania, Lycos and Kewego. Nicolas lives in Richmond, CA, and is an avid fisherman and aspiring cowboy.

Despite our best efforts, our systems fail. Sometimes it’s our fault - code that we wrote or bugs that we caused. But sometimes the fault is with systems that we rely on.

ZooKeeper is a very useful distributed system that is often used as a building block for other distributed systems, like Kafka and Spark. It is used by PagerDuty for many critical systems, and for five months it failed on us a lot.

We will walk through the process of finding and fixing one cause of many of these failures. You will learn how to use various tools to stress test the network, some intricate details of how ZooKeeper works, and possibly more than you wanted to know about TCP including an example of machines having a different view of the state of a TCP stream.

Donny Nadolny is a Scala developer at PagerDuty, working on improving the reliability of their backend systems. He spends a large amount of time investigating problems experienced with distributed systems like Cassandra and ZooKeeper.

From system performance to application metrics, we continue to further our understanding of what to monitor, why, and how to present it appropriately to the various audiences who need to act on this information. Yet there are things across our environment that we agree we can't measure because they are unquantifiable. That doesn't mean that there is zero signal to be analyzed and monitored.

We can look at open source software that is in wide use, yet becomes stale and unusable after years due to the atrophy of maintainers keeping it up to date with security and integrations with other software, or implementation of new features that keep it useful. How do you measure the health of your current implemented software solutions so that you know when to start planning change, or committing intentional time to a project?

How do you know when the cost of Amazon's Relational Database Service is more costly than having in-house qualified DBAs?

How do you measure the value of sending your employees to one conference over another? How do you convey the value of going to one conference over another?

In this talk, I'll tackle these questions in addition to sharing other observations about monitoring within our environments with the goal of inspiring others to examine available signals, their impact, and the value of monitoring.

Facebook began deploying IPv6 only clusters in 2014. This talk covers the challenges & solutions with provisioning in a v6 only world from hardware to software.

Lightning Talks are back-to-back five-minute presentations on just about anything.

Is your root cause really “human error”? How did your environment let the human make the error? How did their error take down the service? How many outages did humans prevent? Can your dev teams’ priorities be aligned with reliability, instead of only with churning out features?

At Heroku, we do ops as a service -- reliability is our product. If we go down, we take thousands of businesses with us. In SRE, we push for reliability and resiliency in designs, sure, but it’s more than that. We iterate on process, automation, tooling, and incident response, because people are at the heart of everything we do.

Lex Neva is probably not a super-villain. He has 6 years of experience keeping large services running, including Linden Lab's Second Life, DeviantArt.com, and his current position as a Heroku SRE. While originally trained in computer science, he’s found that he most enjoys applying his software engineering skills to operations. A veteran of many large incidents, he has strong opinions on incident response, on-call sustainability, and reliable infrastructure design, and he currently runs SRE Weekly.

Courtney comes from a background in customer support and internet anti-abuse policy. She combines this human-focused experience with the principle of Conway’s Law and the work of Kathy Sierra and Don Norman into a wide-reaching and humane concept of operational reliability.

I will show how my team has been using HashiCorp's Terraform for a year to manage our AWS infrastructure. I'll go over a small intro to Terraform, then jump into some internal design decisions, internal best practices, and a short demo to help solidify some of the principles.

Kelvin is a Systems Engineer for Adobe where he is working in their Digital Marketing BU. There he has been using Terraform and Puppet to transform the way the ops team does business. Before that he was working as a "full stack" ops guy for MX where he racked servers, deployed code, and generally administered systems. He loves automation, making sure everything is committed to version control, and completing fulfilling and interesting projects. You can find him on Twitter @zxjinn.

An organization performs best when it is made up of a diverse group of people, opinions and ideas from a variety of locations and backgrounds. As a manager, your job is to balance this diversity to support and grow a strong team poised for individual and organizational success. Learning how to best motivate a group of disparate and distinct people with unique talents best prepares you to overcome unforeseen situations and obstacles as you scale your operations organization.

At Fastly, we’ve built a world-class team of ops experts from many backgrounds and timezones around the world, amidst immense growth and with operational pressure. This talk will cover strategies for overcoming individual and organizational management challenges in a globally diverse environment. We’ll discuss strategies for moving your team toward a shared positive future amidst friction in a fast paced 24x7 operational environment. We’ll explore different people management challenges and discuss methods to work through the grumpiest admin. This talk will leave you thinking of new methods for focusing your efforts on total team strength and looking where you want to go instead of where you’re afraid of heading.

This talk discusses techniques and philosophy for making oncall more humane / reasonable. We will discuss terminology, how to sort through "alert overload," suggestions for tooling to help, and processes you can put in place to get development & management on board. This comes from years of taking over a managing various oncall services for different architectures, often from extreme states of disrepair. 

With more than 15 years of experience building database-backed, internet-based systems, Robert currently spends his days as CEO of OmniTI, a technical services firm focused on helping clients with web application and infrastructure management at scale. Author and speaker at conferences worldwide, Robert is a recognized expert within the industry on topics such as Open Source, databases, and managing web operations at scale. He occasionally blogs at xzilla.net.