SREcon15 Program

Striving for Imperfection: Using an error budget to move fast without compromising high reliability

You may assume Site Reliability Engineers aim to build systems that never go down. What that fails to realize is that 100% reliability is almost never the goal. Instead, our task is to trade off reliability against the many other goals we have for our services. SREs want to provide great service to end users and customers, and also have the flexibility to change the systems often and quickly. We want to ensure that the queries and the revenue keep flowing, and do so as efficiently as possible, provisioning as little excess as necessary to deliver good service. Taking an engineering approach to meeting these goals means we need to make these tradeoffs measurable, and this is where error budgets come in. Your error budget is a measure of risk, it is the amount of headroom you have above your SLA. Being smart about how you manage and spend this error budget is one of the best tools that SRE has to meet the various contending goals that services at Internet scale present.

Marc Alvidrez is a Senior Staff Site Reliability Engineer with Google. He joined the company in 2004, and starting as an early SRE he has led a variety of teams responsible for both infrastructure and major user-facing services. These have included the first team responsible for Google File System (GFS), and the teams responsible for Google's Display and AdSense advertising serving systems, Google+ and Google Photos. Prior to Google he held systems engineering roles at Vodafone and Internet startup Topica, where he was the Director of Operations.

There are many different approaches for reliability work, and different needs depending on each company's needs and infrastructure. One might spend time dealing with private clouds, public clouds, in-house software, third-party software, developing, watching, instrumenting, and debugging. Sometimes we need to build, sometimes we need to optimize, sometimes we need to un-break things.

How do we build teams that adapt to the different needs, and how can we hire effectively for each one of those cases? What are the different approaches we can take for the work? In this panel we'll discuss some of that, and compare different approaches from different organizations actively working on systems reliability.

Stephanie Dean has built, grown, and supported various types and sizes of operational engineering teams for the past 10 years at companies such as Amazon, Facebook, and Twitter. She's now living the life enabling teams to build out Infrastructure as a TPM at Dropbox.

Dave Barr has been in the industry for 20+ years and a Site Reliability Engineer with Google, StumbleUpon, and Twitter. He now manages several SRE teams at Twitter, covering core Twitter services, Search, and Ads serving.

Abe Hassan is a Site Reliability Manager at Google, working on the Search Indexing team. Prior to joining Google, Abe managed the operations team at Say Media and at Six Apart, responsible for the web operations of blogging services LiveJournal and Typepad.

Site reliability is not just about keeping your service running, users clicking, and transactions humming along. Your service does not end at your network interconnect. Reliability has to include principles and practices of safe computing for your users, their data, and all of the other systems that your site connects to. Bake security in from the beginning to protect your site, your users, and user data from an increasing array of assailants: DevOps needs InfoSec too.

Threats to the stability of a service can come from poor code, unreliable hardware, or bad network or power connections, but they also come from people who want to exploit the power and resources which you have accumulated to operate and grow your site. Hardware failures and other random acts of nature will test the abilities of your operations teams, but they do not present the same threat as a sentient enemy. And any site with resources of value will have intelligent, resourceful predators. If you haven't seen them, you are not looking in the right places.

This panel of experienced anti-abuse and security experts will cover the threat space for services, from fraudulent accounts to nation state actors who want your user's data, to scammers who want to use your infrastructure to wreak havoc on other portions of the internet ecosystem. Learn about developing a data security action plan for your site.

Join us for an adventure to foil the dark side, and be careful not to blink.

Kurt Andersen has been active in the anti-abuse community for over 15 years and currently leads the Growth and Comm SRE team at LinkedIn, as well as serving as one of the Program Committee Chairs for the Messaging, Malware and Mobile Anti-Abuse Working Group (M3AAWG). He has spoken at M3AAWG, Velocity, and SANOG on various aspects of reliability, authentication and security.

Ryan Boyd manages the global email delivery team at Groupon, which is responsible for email delivery in 48 countries around the world. Ryan’s team integrates within the engineering and marketing organizations at Groupon to maintain industry-leading deliverability rates through establishing and maintaining best practices for email delivery and security.

Pete Cheslock is the head of Threat Stack's operations and support teams. He has over 15 years' experience in DevOps, and understands the challenges and and issues faced by security, development and operations professionals every day, and how we can help deal with them.

Mark Risher runs the Spam & Abuse team at Google, protecting services across the company from attacks of all kinds. Previously, he was CEO and Co-Founder of Impermium (which Google acquired in 2014), and "Spam Czar" at Yahoo! He has regularly presented worldwide to government, industry, and the media about spam, abuse and cyber security issues.

Cory Scott​ joined LinkedIn in 2013 and currently leads the House Security team, which is responsible for application and infrastructure security across all of LinkedIn. Prior to joining LinkedIn, he was the director of Matasano Security, an information security consultancy. He has spoken at BlackHat, USENIX LISA, SANS, and OWASP.

Jamie Tomasello is the technology director at AccessNow.Org, an international human rights organization dedicated to defending the digital rights of at-risk users. Jamie has been combating Internet abuse and addressing policy issues for more than the past 13 years. With a background in applied behavior analysis and pattern recognition, Jamie has focused her efforts in analyzing technical data points underlying cybercrime, identifying non-obvious data relationships, and profiling cybercriminals.

Join us for a panel discussing various challenges facing the education of SREs. The panel will focus on new SREs (coming out of higher education), ongoing education and how we as a group educate developers.

Training the next generation of SREs coming in to our field means figuring out what information is important from other fields as well as trying to pass on lessons learned from years of experience. Also, we need to listen to new people that bring an unfettered perspective.

Educating SREs within a company present slightly different issues. Namely, keeping up to date on tools that may be very specific to the processes used at any one company. Companies may choose to follow industry norms or may do something totally different. Companies need to keep their employees up to date with there specific goals.

Lastly, developers have their own priorities, but we need their help to emit metrics, add caching as well as a host of other code-level changes. We can learn from their experiences and also educate them to our best practices.

Craig Sebenik works for a startup as the only SRE (infrastructure engineer). This presents plenty of opportunities to empower developers to maintain their own code. But it also presents a number of challenges with educating devs on adding metrics, improving deployment, etc. Craig recently left LinkedIn where he was an SRE and led a few different initiatives aimed at training SREs. There are valuable lessons from both large and small companies. Craig has a passion for education and is very interested in sharing knowledge throughout our industry.

David Mah is a Dropbox SRE characterized by his paranoia when dealing with infrastructure. Sometimes people consider this to be 'attention to detail'; other times it would just be considered paranoia. His current focus is on building protections in Dropbox's in-house storage system to avoid losing user data. He recently graduated from the University of Washington and brings the perspective of one who didn't know much about SRE until fairly recently.

Andrew Widdowson is a Staff Software Engineer in Site Reliability Engineering at Google. In addition to being a long time member of the Search SRE team, he is the tech lead for "SRE EDU", Google's internal efforts to support a culture of teaching of and from SREs. Andrew also leads a team of engineers that fight abusers, scrapers, and attackers of the search stack. In his spare time, Andrew serves as a chair of the Carnegie Mellon University School of Computer Science Alumni Advisory Board.