Lessons Learned from Our Main Database Migrations at Facebook

Website Maintenance Alert

Due to scheduled maintenance, the USENIX website will not be available on Tuesday, December 17, from 10:00 am to 2:00 pm Pacific Daylight Time (UTC -7). We apologize for the inconvenience.

If you are trying to register for Enigma 2020, please complete your registration before or after this time period.

Friday, 2018, August 31 - 14:0014:50

Yoshinori Matsunobu, Facebook

Abstract: 

At Facebook, we created a new MySQL storage engine called MyRocks. Our objective was to migrate one of our main databases (UDB) from compressed InnoDB to MyRocks and reduce the amount of storage and number of servers used by half. In August 2017, we finished converting from InnoDB to MyRocks in UDB. The migration was very carefully planned and executed, and it took nearly a year, but that was not the end of the migration. SREs needed to continue to operate MyRocks databases reliably. It was also important to find any production issue and to mitigate or fix it before it became critical. Since MyRocks was a new database, we encountered several issues after running in production. In this session, I will introduce several interesting production issues that we have faced, and how we have fixed them. Some of the issues were very hard to predict. These will be interesting for attendees to learn too.

Attendees will learn the following topics.

  • What is MyRocks, and why it was beneficial for large services like Facebook
  • What should be considered for production database migration
  • How migration should be executed
  • Learning 4-6 real production issues

Yoshinori Matsunobu, Facebook

Yoshinori Matsunobu is a Production Engineer at Facebook, and is leading MyRocks project and deployment. Yoshinori has been around the MySQL community for over 10 years. He was a senior consultant at MySQL Inc. from 2006 to 2010. Yoshinori created a couple of useful open source product/tools, including MHA (automated MySQL master failover tool) and quickstack.

BibTeX
@inproceedings {218919,
author = {Yoshinori Matsunobu},
title = {Lessons Learned from Our Main Database Migrations at Facebook},
booktitle = {SREcon18 Europe/Middle East/Africa (SREcon18 Europe)},
year = {2018},
address = {Dusseldorf},
url = {https://www.usenix.org/node/218920},
publisher = {{USENIX} Association},
month = aug,
}