Visibility into Loggers (and Other Low Level Services)—Seeing the Trees from the Forest

Monday, March 25, 2019 - 4:25 pm4:55 pm

Danny Chen, Bloomberg LP


Loggers and tracers have become crucial components of computing systems, providing invaluable visibility into the runtime behavior of our software. Ironically, these vital components are opaque when it comes to their own runtime behavior. We typically only look at logging as suspects in performance-related incidents as part of post-mortem analysis.

Why do we have such blind spots when it comes to components that are pervasively used in our systems? We explore possible explanations and present example solutions.

Danny Chen, Bloomberg LP

Danny Chen started his career almost 40 years ago as a UNIX performance engineer at Bell Laboratories where he was a co-developer of one of first general purpose UNIX kernel tracing facilities (USENIX/1988: CASPER the Friendly Daemon). He also contributed to the SVR4 virtual memory implementation (USENIX/1990: "Insuring Improved VM Performance - Some No-Fault Policies). Since then he has worked on a wide variety of systems ranging from low latency market data to distributed transaction managers—all while tailing logs.

SREcon19 Americas Open Access Videos Sponsored by

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

@conference {229515,
author = {Danny Chen},
title = {Visibility into Loggers (and Other Low Level {Services){\textemdash}Seeing} the Trees from the Forest},
year = {2019},
address = {Brooklyn, NY},
publisher = {USENIX Association},
month = mar,

Presentation Video