The Chicken or the Egg: Creating a Stable Preprod Environment as a Public Cloud Provider

As a public cloud company, we needed an internal reliable environment to safely test changes before shipping to Production. Supporting our infrastructure requires validating an increasingly widening array of components and services. This required our SRE team to build and maintain a full-featured replica of Production from scratch. This talk tells the story of how we brought "Testcloud" online. We’ll discuss a technical overview of the challenges we faced building our "Testcloud" including how we tackled the unique operational challenges that come with building isolated ecosystems. These include how we evaluated and adapted existing industry examples, challenges in achieving parity with production, partitioning production from lower environments, and managing circular dependencies. The target audience is anyone who finds themselves having to test software code or physical components while managing customer expectations.