Infrastructure Kata and Moving a Large Enterprise to the Cloud

Up until a few years ago, Yelp had a traditional datacenter-based, bare-metal infrastructure for the production web site. The flexibility and dynamic nature of cloud storage and data processing environments was our first introduction to the cloud, and was soon heavily adopted internally. As the company continued to expand, both in its home (U.S.-based) territory, and internationally, scaling infrastructure in traditional data centers became expensive and constraining.

Therefore the operations team’s board-level goals became to build out production environments in the cloud, initially in one region, and then into other (more remote) regions. Taking an application and infrastructure built for a traditional environment and porting it to run in the cloud is not always a simple endeavour. Each build-out has undertaken a significant amount of work to clean up and iterate on our architecture, however at each step pragmatic compromises have had to be made to deliver business value.

This talk will recap the first several iterations of this build-out, concentrating on the problems that we encountered both technically with the new deployments and socially as the team scaled and became international. The focus will be on the "Kata" of each step—deliberate meditations on the nature of our engineering and social problems, and the improvements we made by repeatedly solving the same problem(s) again and again in an iterative manner as we built out each new environment.

Equal coverage will be given to things which worked out well (being a wise investment in time or not too painful to ignore till the next iteration), and those which didn’t work—where we spent resources automating things not on the critical path, or chose not to tackle an issue which caused issues later.

Tomas works on infrastructure automation and hybrid cloud deployments at large scale for Yelp. He speaks regularly at devops events and other technical conferences on a number of topics between testing, development, architecture, automation and systems administration. Tom came to the dark side of systems and devops after being a professional perl developer for many years. He's an avid open source contributor and lives in London with his fiancé and far too many cats.