Hidden Linux Metrics with Prometheus eBPF Exporter

Wednesday, October 31, 2018 - 11:45 am12:30 pm

Alexander Huynh, Cloudflare, and Ivan Babrou

Abstract: 

While there are plenty of readily available metrics for monitoring Linux kernel, many gems remain hidden. With the help of recent developments in eBPF, it is now possible to run safe programs in the kernel to collect arbitrary information with little to no overhead. A few examples include:

  • Disk latency and io size histograms
  • Run queue (scheduler) latency
  • Page cache efficiency
  • Directory cache efficiency
  • LLC (aka L3 cache) efficiency
  • Kernel timer counters
  • System-wide TCP retransmits

Practically any event from "perf list" output and any kernel function can be traced, analyzed and turned into a Prometheus metric with almost arbitrary labels attached to it.

If you are already familiar with BCC tools, you may think if ebpf_exporter as bcc tools turned into prometheus metrics.

In this tutorial we’ll go over eBPF basics, how to write programs and get insights into a running system.

https://github.com/cloudflare/ebpf_exporter

Ivan Babrou, Cloudflare

Ivan is a Performance Engineer at Cloudflare. He spends his days finding performance bottlenecks, fixing them and making sure large chunk of internet runs as fast and as efficiently as possible.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

This content is available to:

Download Training Materials (Attendees Only)
BibTeX
@conference {221812,
author = {Alexander Huynh and Ivan Babrou},
title = {Hidden Linux Metrics with Prometheus eBPF Exporter},
year = {2018},
address = {Nashville, TN},
publisher = {{USENIX} Association},
month = oct,
}
Who should attend: 

Systems engineers interested in eBPF, as well as systems integrators interested in exposing these traditionally-isolated eBPF results into a larger metrics pipelines, should attend.

Especially recommended to attendees of Ivan Babrou's talk presentation, since this is its practical supplement.

Take back to work: 
  • eBPF is a safe and performant way to get hidden metrics out of your system
  • There are readily available tools (bcc tools) using eBPF to get insights into a running system
  • You can efficiently track important kernel metrics over time to make changes confidently
Topics include: 

Prometheus exporter for eBPF metrics, Linux kernel internals

Prerequisites: 
  • Laptop running a modern Linux, either VM or native
  • Working knowledge of YAML
  • Basic knowledge of C
  • Interest in Linux kernel inner workings