Managing Project Migrations at Scale

Problem: Migrate 1400 build projects to a container-based build environment.

Challenges:

• Projects were spread across many teams. We had to create a migration plan that would reduce the overhead on each team.
• Fixed deadline for completing work (no stragglers!)
• New build environment significantly different from the old one—some tools not available, path changes, resource restrictions.
• Autonomous development teams with different project configurations

Solutions:

• Transparently modify build helper programs where possible to detect environment (old versus new) and "do the right thing"
• Automate as much of the migration as possible so dev teams don't have to understand the internals of the build system
• Scripts to assess migration risk for each project (risky projects received direct attention from releng team)
• Begin standardization effort to make future automation easier
• Tools to assist teams in identifying correct containers in which to run
• Daily standups helped the distributed team stay focused and improved communication

Lessons Learned:

• Standardization is still the key to providing support at scale
• Most engineers don't care what their build-related files look like
• deally, engineers should not have to know anything about the build system