Kernel Upgrades at Facebook

Thursday, 2018, August 30 - 12:0012:30

Pradeep Nayak Udupi Kadbet

Abstract: 

The goal of this talk is to explain the importance of automating your kernel upgrades and why you should invest time in building automation which reliably and continuously enforces newer kernels on your hosts.

The Kernel Team at Facebook is in charge of the Linux kernel used at Facebook, along with other 'system level' packages that go with. The kernel team works on tasks like:

  • Merging upstream changes into the Facebook Linux Kernel
  • Creating custom kernel changes for our needs
  • Investigating Linux-related performance issues and failures
  • Periodically building and initial testing of new Facebook kernel rpms

MySQL is one of the primary data stores which Facebook relies on. We have tens of thousands of database hosts which run on linux boxes with different kernel versions. No kernel is perfect and often time database hosts hit kernel bugs which impact production traffic. The remediation often is to upgrade to newer kernels which have these fixes.

In this talk I will go over some of the kernel bugs which impacted our production database servers and how we invested time in developing an automation framework to enforce new kernels on our database hosts in a continuous fashion at Facebook scale. I will also go over how MySQL Infrastructure at Facebook adopted this and is successfully upgrading tens of thousands of database servers without impacting production traffic.

Pradeep Nayak Udupi Kadbet

Pradeep is a Production Engineer at Facebook and works with MySQL Infrastructure. He loves hacking code in python and builds bots to do things for him. While he is not working, he enjoys traveling and clicking pictures.

SREcon18 Europe/Middle East/Africa Open Access Videos
Sponsored by Indeed

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

Presentation Audio

BibTeX
@inproceedings {218945,
author = {Pradeep Nayak Udupi Kadbet},
title = {Kernel Upgrades at Facebook},
booktitle = {SREcon18 Europe/Middle East/Africa (SREcon18 Europe)},
year = {2018},
address = {Dusseldorf},
url = {https://www.usenix.org/node/218946},
publisher = {{USENIX} Association},
}