Interview with Chastity Blackwell on Her Upcoming LISA17 Talk on Documentation

I’m a fan of documentation. I have been, ever since I first inherited an infrastructure and had to try to figure out what the previous admin had done. I try to write decent documentation—documentation that I wish had been was there when I first tried to learn about a particular system. I also find myself doing so much that what I can keep in cache in my brain is limited, and stuff I’ve worked on tends to “page out,” so to speak. Writing good docs is an excellent way to do your future self a favor.

Because I want to be able to rely on my documentation, it’s something that I actively try to make better. I was excited to see that Chastity Blackwell, someone I really respect from the community, is coming to LISA17 to talk about "The 7 Deadly Sins of Documentation"! I wanted to take a few minutes to ask her some questions about how she documents and what is important to her, in preparation for her talk on Wednesday!

Matt Simmons: Hi Chastity! Thanks so much for your time! Can you introduce yourself, and give some of your background as to what made you first start caring strongly about documentation?

Chastity Blackwell: My name is Chastity Blackwell and I'm an SRE at Yelp. I've been working in Operations for about 15 years, but I don't really have a traditional background in programming or CS. In college, I studied creative writing and history and I've done some freelance writing in the past, so writing in general is something that's important to me.

As someone who entered the field without as strong a technical background as some of my peers, having good documentation has been very helpful.

MS: What advice do you have for people who are in the situation where they are members of a team where there isn’t a strong social pressure to document?

CB: Even in situations where there's no strong social pressure to document, I think everyone tends to do their own informal documentation by jotting notes in a file, a ticket, or a notebook. I don't know that I've ever worked somewhere where everything is entirely done by oral tradition. The trick is to turn these informal little documents into something that is actually useful for everyone. At some point, that does require someone taking the initiative to create some kind of repository for this accumulated knowledge.

If there's no one in your organization or your team that is taking that initiative, you can do it yourself. It doesn't have to be particularly fancy, especially to start—a git repo with a bunch of text files is better than a bunch of text files scattered across various laptops—but, the larger your organization or infrastructure, the more organized documentation needs to be.

MS: One of the problems I’ve seen throughout my career is out of date documentation, because it might have been done by a previous team member, or it was done when it was thought of and neglected. What do you do to make sure that docs are maintained and living, rather than abandoned and neglected?

CB: This is a really common problem and during my talk, I share a number of ways to tackle the issue. Most importantly, you've got to make updating documentation an actual goal when you allocate time and resources to a project. Too often it’s considered more of a nice-to-have, or something that's assumed will get done over the course of the rest of the project, but that's just not the case. You've got to actually treat documentation like a deliverable, just like code, and make sure that it's useful by having it reviewed.

To ensure that's as easy as possible, make sure that your documentation isn't onerous to update. Consolidate all documentation into a single place (or at least a single place for each type of documentation) and make sure that place is discoverable and searchable. Good docs tend to help perpetuate themselves; if you know there's a good doc on X piece of infrastructure right here, it's easy to consult it—and if something changes, you know you can go in there and fix it.

MS: What do you think the biggest problem with documentation is? Is there a magic bullet to fix it? If you could describe an infrastructure that had perfect documentation, what would it look like?

CB: This is hard to say, but I think the root of most documentation problems is that it isn't considered important. If you don't give folks time to actually document things, treat it like a task that can be pawned off on junior members of the team, or worse, throw up your hands and say it's impossible to document your rapidly changing infrastructure, there's obviously no way your documentation is going to be great. Unfortunately, there’s no magic bullet. Poor documentation is a cultural problem that needs strong leadership to fix.

MS: If you could describe an infrastructure that had perfect documentation, what would it look like?

CB: I don't think there's really any way to have perfect documentation. Similar to how something is always broken in a modern distributed systems infrastructure, some part of your documentation is going to be out-of-date or incorrect. Good documentation isn't just having all your runbooks and technical docs in a carefully curated repository; it has to include all the culture and processes around it so that when you do find that something is wrong, it can be fixed easily. Unless you're entombing your infrastructure in amber, it's always going to be a moving target, and your docs are going to be too.

MS: Do you think you should build and then document, document and then build, or some combination?

CB: I think it has to be a back and forth. Nothing you build is going to emerge from the end of that process in exactly the way it was conceived of from the start, but you don't want to go into that process without some sort of plan. At Yelp, we have formal proposals for large projects that describe the nature of the system, but those are rarely the documentation that you're going back to and reading to deal with the day-to-day work. In addition, the system is likely going to change over the course of its lifetime, and you'll need to update the documentation to suit.

MS: What background or familiarity should we have before we come to your talk?

CB: I think my talk should be accessible to most folks in operations without needing any special knowledge. I'm mostly talking about how to write better and improve your process around document creation, not specific tools or technologies.

Thanks so much for your time, Chastity! I’m really looking forward to your talk!

You can catch "The 7 Deadly Sins of Documentation" on Wednesday, November 1 at 4:00 pm!