Interview with Brendan Gregg, winner of the 2013 LISA Award for Outstanding Achievement in System Administration

In 2010, Joyent's Brendan Gregg gave his first USENIX LISA conference talk. He says he didn't feel qualified to give the talk, but a colleague encouraged him to submit the proposal, Visualizations for Performance Analysis (and More). Only a few short years later, Brendan Gregg was on stage at LISA '13 to accept the 2013 LISA Award for Outstanding Achievement in System Administration.

Brendan is the lead performance engineer at Joyent and the author of the recently released Systems Performance book, but he still identifies himself as a sys admin. Looking around the LISA '13 expo hall entrance, Brendan says, “This crowd feels more like my crowd than any of the others.”

I asked Brendan how it felt to win the LISA Outstanding Achievement in System Administration award, which means he's in the company of well-known and respected sys admins dating back to 1993. He admitted that winning an award previously given to admins like Evi Nemeth is intimidating, because she's one of his role models and he has owned all of her system administration books.

Brendan says that the competition for the award was stiff, but he's encouraged by how USENIX gives the awards. Rather than being awarded for creating a particular sys admin tool, the award is given to people who also participate in the greater sys admin community and help in other ways, for example, by writing tech books or troubleshooting guides. He likes that the award is given for outstanding achievement, “which can recognize all forms of contributions,” he says.

Recently, Brendan's focus leans toward methodologies. When he was a junior sys admin, Brendan says that performance analysis knowledge was passed down from senior sys admins, or it wasn't passed down and you had to be a part of a “secret priesthood” to acquire it. As he became an expert in performance analysis and began to teach performance classes, he realized that analysts had techniques that helped solve problems, but they hadn't written them down. That's when he realized that he needed to document methodologies to pass on to his students. In fact, he develops methodologies and then tests them out on students and mentees, requests their feedback, and determines what works.

A few years ago, Brendan published a methodology called the USE Method, which checks system performance health and identifies bottlenecks and errors. Although he was using the USE Method years ago, Brendan hadn't written it down, but teaching it to other admins helped him realize that it was useful, and documenting it was a way to give back to the sys admin community. Now Brendan says the USE Method is becoming a standard methodology for performance classes.

Brendan says he has published more than 300 performance tools for different operating systems, but the tools aren't helpful if admins don't know which tools to run. Brendan realized that in addition to providing tools, you need to offer guidance on which tool to use and how to use it. “Part of documenting methodologies is out of guilt for making the problem even worse by putting more tools into the community,” he says. In his new Systems Performance book, he documented approximately 30 methodologies.

Check back for our LISA '13 videos soon, and in the meantime, you can watch Brendan's LISA '12 talk in which he discusses the USE Method and several other analysis methods.

More news from LISA '13:

• USENIX: Flame graph shows system performance in a new light
• USENIX: Red Hat prepares for 64-bit ARM servers
• Security expert seeks to make surveillance costly again
• Is there a link between the low cost of storage and government surveillance?
• Introduction to Chef
• Recap of Gluster Community Day at USENIX LISA
• Evil Genius 101
• Continuous Integration with Jenkins
• Getting to Senior: Sysadmin Level IV and Beyond
• Non-Abstract Large System Design for Sysadmins
• More Power with PowerShell - Class Review
• Six Days of LISA '13
• The 2013 Check Yerkes Award Goes to...
• LISA13 Conference Review / Overview Thing

Are we missing your coverage of LISA '13? If so, post the link in the comments or email it to rikki@usenix.org.