Systems Tracing and Trace Visualization Lab—Part I

Wednesday, October 31, 2018 - 9:00 am10:30 am

Suchakrapani Sharma, Shiftleft Inc; Geneviève Bastien, École Polytechnique de Montréal; Mohamad Gebai, SUSE


Trace Visualization Lab is a two-part lab session that introduces participants to system tracing and trace visualizations that are an invaluable technique to understand in-depth system behavior and reach root-cause of problems faster than just CLI tracing modes. The focus of this lab is on post-mortem analysis (the system failed and we want to understand the root cause). The tutorial first introduces attendees to system tracing, trace collection, storage/aggregation/filtering and eventually visualization techniques such as flamecharts, flamegraphs, timeline views, critical flow view (inter-process flow), container and VM views, resource usage etc).

The introductory sessions (part I) also explain why and when is systems tracing required and its role in supporting related performance anlaysis techniques such as distributed tracing, profiling, debugging and service log analysis. Various visual techniques will be shown and hands on activities that showcase how different "views" can help in scenarios such as resource contention, latency analysis would be carried out.

For this lab, primary trace collection will be through LTTng, Perf/Ftrace and primary trace visualization system will be Trace Compass.

The Trace Visualization Lab has been open sourced and is publicly accessible at:

Suchakrapani Sharma, Shiftleft Inc

Suchakra is currently a Scientist at ShiftLeft Inc. He completed his PhD in Computer Engineering from École Polytechnique de Montréal where he worked on eBPF and hardware-assisted tracing techniques for advanced systems performance analysis. He has been involved in research on systems performance analysis and has delivered talks at LISA 2017 (San Francisco), Tracing Summit 2015 and 2016 (Seattle and Berlin) where he has demonstrated advanced kernel and userspace tracing tools. He also developed one of the first hardware-trace based VM analysis techniques. More information about him can be found at :

Geneviève Bastien, École Polytechnique de Montréal

Geneviève Bastien is a research associate at the Dorsal Laboratory of École Polytechnique de Montréal. She is a contributor to the Trace Compass and LTTng projects. Her mission is to make the students' life easier when in comes to prototyping cool new analyses and to make sure that their contributions make it into the hands of the end users. She's also involved in Community Wireless Networks in Montréal and in organizations that promote the use of free software in the public sphere.

Mohamad Gebai, SUSE

Mohamad Gebai is a Senior Software Engineer at SUSE. He currently works on software-defined storage solutions, notably Ceph, and has previously been part of the Azure Storage team at Microsoft. He has also been a research associate and a guest lecturer at Polytechnique Montreal. He lectured on Cloud computing technologies.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

This content is available to:

@conference {221832,
author = {Suchakrapani Sharma and Genevi{\`e}ve Bastien and Mohamad Gebai},
title = {Systems Tracing and Trace Visualization {Lab{\textemdash}Part} I},
year = {2018},
address = {Nashville, TN},
publisher = {USENIX Association},
month = oct
Download Training Materials (Attendees Only)
Who should attend: 

Site Reliability Engineers, developers, performance engineers, infrastructure engineers, systems engineers

Take back to work: 
  • Learn overview of performance anlaysis tools and techniques
  • Understand when system tracing is needed and when other tools are sufficient
  • Learn how to gather system traces
  • Learn how to visualize and analyze traces to understand system behavior
  • Use different "views" available in Trace Compass to find root cause of elusive performance bugs
  • Learn how to design and code custom views in Trace Compass tailored for their usecase
Topics include: 

Systems Performance, Performance Debugging, Tracing, Post-mortem analysis of systems, Operating Systems (Linux), Trace Visualizations


Basic understanding of operating systems and networking. Prior experience of frustration during debugging/bug hunting and systems engineering is valuable as well but not a hard pre-requisite.