From Firefighting to Proactive Work: the Journey of a Small Infrastructure Team in a Hyper Growth Environment

Thursday, 2017, August 31 - 09:5010:20

Alex Gerlic, Intercom

Abstract: 

Due to an incident on our main datastore, we react and spent an entire week trying to keep Intercom up, with the help of 20 engineers from other teams. During this tough week, we had obliged to drop any other projects and focus on building a firefighting organization.

After the urgency period, it became evident to us that we need to focus on reactive work to prevent the incident from happening again. It was the launch-pad for the conception of a brand-new organization for our team, focusing on ownership and high impact work.

Few months after, results ruled in favour of our hard work: we’ve reduced system interruptions by more than 80% ! But good news and radical changes also come with consequences: we need to deal with multiple implications and drastically change our way to work as a team

During this talk we will cover:  

  • our journey from a firefighting to a proactive work organization.
  • good and bad organizational decisions we made
  • impacts on the morale of the team

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

Presentation Audio

BibTeX
@conference {205510,
author = {Alex Gerlic},
title = {From Firefighting to Proactive Work: the Journey of a Small Infrastructure Team in a Hyper Growth Environment},
year = {2017},
address = {Dublin},
publisher = {{USENIX} Association},
}