An SRE Guide to Linux Kernel Upgrades

Wednesday, June 14, 2023 - 3:50 pm4:45 pm

Ignat Korchagin, Cloudflare


The Linux Kernel lies at the heart of many high profile services and applications. And since the kernel code executes at the highest privilege level it is very important to keep up with kernel updates to ensure the production systems are patched in a timely manner for numerous security vulnerabilities discovered almost every day.

Yet, because the kernel code executes at the highest privilege level and a kernel bug usually crashes the whole system, many SREs, production engineers and system administrators try to avoid upgrading the kernel too often just for the sake of stability. In many companies we have seen a tendency to create more obstacles to Linux kernel releases (requiring more approvals, harder update justifications, requiring more time in canary testing etc). But introducing all these obstacles and not treating kernel updates like any other software updates usually significantly increases the risk for the company and their service of being exploited.

One of the reasons SREs and production engineers are too afraid of ANY kernel upgrade is that they don’t actually know the details about Linux kernel release process and policy. This talk tries to demystify Linux Kernel releases and provides a guide on how to distinguish a kernel bugfix release from a feature release. We also try to explore why commonly established perceptions and patterns around production kernel releases are wrong and how you actually risk the stability of your systems by not releasing the kernel regularly. In the end we describe how kernel releases are implemented in our company and propose possible approaches to deploy kernel upgrades regularly with minimal risk.

Ignat Korchagin, Cloudflare

Ignat is a systems engineer at Cloudflare working mostly on Linux. Ignat’s interests are cryptography, hacking, and low-level programming. Before Cloudflare, Ignat worked as a senior security engineer for Samsung Electronics’ Mobile Communications Division. His solutions may be found in many older Samsung smart phones and tablets. Ignat started his career as a security researcher in the Ukrainian government’s communications services.

@conference {288275,
author = {Ignat Korchagin},
title = {An {SRE} Guide to Linux Kernel Upgrades},
year = {2023},
address = {Singapore},
publisher = {USENIX Association},
month = jun

Presentation Video