Alexander Huynh, Cloudflare, and Ivan Babrou
While there are plenty of readily available metrics for monitoring Linux kernel, many gems remain hidden. With the help of recent developments in eBPF, it is now possible to run safe programs in the kernel to collect arbitrary information with little to no overhead. A few examples include:
- Disk latency and io size histograms
- Run queue (scheduler) latency
- Page cache efficiency
- Directory cache efficiency
- LLC (aka L3 cache) efficiency
- Kernel timer counters
- System-wide TCP retransmits
Practically any event from "perf list" output and any kernel function can be traced, analyzed and turned into a Prometheus metric with almost arbitrary labels attached to it.
If you are already familiar with BCC tools, you may think if ebpf_exporter as bcc tools turned into prometheus metrics.
In this tutorial we’ll go over eBPF basics, how to write programs and get insights into a running system.
Alexander Huynh, Cloudflare
Performance engineer, specializing in fine tuning existing systems, and the R&D of future systems.
Ivan Babrou, Cloudflare
Ivan is a Performance Engineer at Cloudflare. He spends his days finding performance bottlenecks, fixing them and making sure large chunk of internet runs as fast and as efficiently as possible.
Open Access Media
USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.
This content is available to:
author = {Alexander Huynh and Ivan Babrou},
title = {Hidden Linux Metrics with Prometheus {eBPF} Exporter},
year = {2018},
address = {Nashville, TN},
publisher = {USENIX Association},
month = oct
}
Download Training Materials (Attendees Only)Systems engineers interested in eBPF, as well as systems integrators interested in exposing these traditionally-isolated eBPF results into a larger metrics pipelines, should attend.
Especially recommended to attendees of Ivan Babrou's talk presentation, since this is its practical supplement.
- eBPF is a safe and performant way to get hidden metrics out of your system
- There are readily available tools (bcc tools) using eBPF to get insights into a running system
- You can efficiently track important kernel metrics over time to make changes confidently
Prometheus exporter for eBPF metrics, Linux kernel internals
- Laptop running a modern Linux, either VM or native
- Working knowledge of YAML
- Basic knowledge of C
- Interest in Linux kernel inner workings