Deploying Changes to Production in the Age of the Microservice

Thursday, 2017, August 31 - 09:0009:50

Samantha Schaevitz, Google


You decoupled your APIs from their implementations and put them behind RPC interfaces. You build and deploy services independently. You code health is impeccable. You put your user data in a persistent, replicated, and consistent store, where it belongs. Your developer velocity has skyrocketed.

Now we have new problems. We’ve got N independent services with M edges of interaction between them. That’s N services that need to be built, tested, and deployed on the infrastructure that expected you to have one service whose mess of entanglement was a secret you had with the compiler.

How do we deploy N binaries with N sources of static configuration and M sources of runtime configuration safely without losing our collective minds? In this talk, I’ll share some of how we grew that aforementioned N from 1 to many in Gmail. Specifically:

  • Consistent naming schemas for services, environments -
  • Maintaining lightweight, easy-to-change production configuration abstraction layers
  • Release early, often
  • Canary everything by sharding into more A/B environments than you'd think you’d need 
  • Encourage backwards compatibility in all APIs 
  • Validate and test all configuration before changing global state

And, of course, some of things we (Gmail) learned by breaking things along the way.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

@conference {205506,
author = {Samantha Schaevitz},
title = {Deploying Changes to Production in the Age of the Microservice},
year = {2017},
address = {Dublin},
publisher = {{USENIX} Association},