MySQL Infrastructure Testing Automation at GitHub

Website Maintenance Alert

Due to scheduled maintenance, the USENIX website will not be available on Tuesday, December 17, from 10:00 am to 2:00 pm Pacific Daylight Time (UTC -7). We apologize for the inconvenience.

If you are trying to register for Enigma 2020, please complete your registration before or after this time period.

Tuesday, October 30, 2018 - 4:00 pm4:30 pm

Jonah Berquist and Gillian Gunson, GitHub

Abstract: 

The database team at GitHub is tasked with keeping the data available and with maintaining its integrity. Our infrastructure automates away much of our operation, but automation requires trust, and trust is gained by testing. This session highlights three examples of infrastructure testing automation that helps us sleep better at night:

  • Backups: auto-restores and backup validation. Making backup data accessible to our engineers. What metrics and alerts we have in place.
  • Failovers: how we continuously test our failover mechanism, orchestrator. How we setup a failover scenario, what defines a successful failover, how we automate away the cleanup. What we do in production.
  • Schema migrations: how we ensure that gh-ost, our schema migration tool, which keeps rewriting our (and your!) data, does the right thing. How we test new branches in production without putting production data at risk.

Jonah Berquist, GitHub

Jonah is the Engineering Manager of the Database Infrastructure team at GitHub. He previously worked as a Senior DBA at Twitter and he began his career with databases as a remote DBA for a variety of customers at Blue Gecko. He enjoys looking at graphs and writing scripts to do his job for him.

Gillian Gunson, GitHub

Gillian has been a database infrastructure engineer at GitHub for two years. Her focus has been on performance troubleshooting and observability. Previous employers include Okta, PalominoDB, Oracle, and Disney. Gillian is based in Vancouver, BC, Canada.

BibTeX
@conference {221756,
author = {Jonah Berquist and Gillian Gunson},
title = {MySQL Infrastructure Testing Automation at GitHub},
year = {2018},
address = {Nashville, TN},
publisher = {{USENIX} Association},
month = oct,
}