In July of 2021, USENIX and the LISA steering committee brought the sad news to the systems engineering community that LISA, the industry's longest-running systems conference, was closing down after 35 years. Thirty five years! When the first LISA workshop happened in 1987, the concept of the HTTP protocol was still two years off, the Linux kernel was three years away, and the SSH protocol wouldn't be around for another eight years. Today, we take those technologies for granted, but LISA helped push those and more into the prominence they have today.
At its largest size, LISA attracted more than 1000 attendees and had over 100 talks, workshops, paper presentations, tutorials, Guru sessions, and BoFs each year. The conference pushed the industry in new directions, and it pushed its attendees' careers in new directions. It became a yearly family reunion where a community of systems engineering folk got together to see old friends and colleagues and welcome new members while talking about where we currently are and where we need to go next. It was also a place where attendees could talk directly to luminaries in the field — the authors of the tools and the experts in the methods we used every day.
With LISA gone, it's time to look back over the years at what it was, how it got to be so popular, and the many places where we can still see its legacy in the systems community. LISA’s history and the history of the last 35 years of computing are very intertwined, and both can be split into three eras spanning about a decade each: The Early Years (1987 to about 2000), The DotCom Bust to The Great Recession (about 2001 to about 2009), and The Final Years (about 2010 to 2021).
The Early Years
The very first LISA, then called the Large Installation System Administrator’s Workshop, was held in April of 1987 in Philadelphia and was organized by Rob Kolstad and Max Vasilatos. The very first LISA call for participation summarized its purpose:
This workshop will bring together system administrators trying to conquer UNIX's historical bias towards smaller systems. It is believed these administrators battle many of the same problems repeatedly and can share their unique solutions to some problems in order to avoid duplication of effort as UNIX grows to run in ever larger installations. System managers of shops with over 100 users (on one or several processors) will find this workshop particularly valuable.
When we asked Rob what he remembered from the origins of LISA, he quoted that CFP, but he also added “One of the parameters we told people defining 'Large Installation' was 'or if you manage 1GB or more of disk space'”. From a simple “100 users or 1GB of disk space” beginning, an industry-leading event was born.
That first CFP included example topics that could be discussed during the workshop and requested that attendees submit a one-to-two page summary of their site and a problem they had solved. Quoting the CFP again:
Some topics to be considered include: large file systems (dumps, networked file systems), password file administration (including YP), large mail system administration, USENET/News/Notes administration, mixed vendor (and version) environments, load control and batch systems, handy new utilities, and large LANs.
Each participant submits (electronically, to {allegra,sun,ihnp4,uiucdcs}!convex!kolstad}) a one or two page single-spaced summary describing a solution to some problem from the topics above (or something equally as interesting/important). Use the first paragraph to describe the properties of the site and anything that makes it unique (e.g., distributed, large, supercomputers, mixed-vendors). Follow with a description of the problem and a description of the solution (detailed enough that fellow administrators can implement it). Also, please include with your submission a set of five (or so) topics that you'd like to hear about.
Themes at this first workshop and other early LISAs seem like archaeological curiosities now, but it’s easy to see how they molded and influenced the industry. A journey through the archives of the first ten years of conference talks shows that topics like monitoring, capacity planning, configuration management, and performance tuning were already of interest to the community. However, they were all at a different scale than we are used to today: monitoring usage of individual workstations, planning disk capacity on a single server, automating the deployment of printer configurations, and improving performance of mail servers.
Just as the technology being discussed at the conference evolved over time, so did the conference infrastructure. Tony Del Porto, long time USENIX sysadmin, described the earliest terminal rooms as having a single Sun server with a dialup SLIP connection and a bunch of terminals and a manual scheduling process:
Conference attendees lined up at the terminal room door, wrote their name on a sign up sheet, and then had a turn at a terminal. Over time the "terminals" changed from dumb terminals to PCs to BYOD, but the concept of providing a way for SysAdmins to work remotely while attending the conference, and the challenges of making that happen, didn't.
Tony went on to describe how the infrastructure evolved in more detail:
At first, conference venues had no internet connectivity, so months prior to the event the terminal room coordinator would arrange for a temporary T1 line or a microwave point-to-point connection. Later, those months were spent convincing the conference venue to let a bunch of SysAdmins take over their network infrastructure and use every megabit of existing bandwidth available. Then, the weekend before the conference, the terminal room coordinator led a merry band of volunteers in building the conference network. They installed *NIX on rented computers, configured network switches and routers, taped down ethernet cable, hung wireless access points, and then monitored it all for a week under the watchful eye — and with the helpful suggestions of — hundreds of their peers. In the last years of LISA, the terminal room was an opportunity for systems engineers to touch physical hardware, a novelty in the era of infrastructure as code. At the end of the week, every cable was pulled up and coiled in the method handed down from Evi Nemeth and boxed up with the rest of the network equipment owned or loaned, and shelved until the next year.
LISA was an instant hit, and each year was bigger than the next. Rob Kolstad noted to us, "Attendance grew in a good way —as did profitability. I'm thinking it was one-of-a-kind for a long time." By the early 1990s, LISA had grown from a two-day workshop to a full five day conference. As it grew, it added more tracks and features, eventually reaching a stable six days of talks, tutorials, workshops, and many other events. Among those events was the vendor exhibition floor, which also started small and grew over the years. Lee Damon, long-time LISA attendee and chair of LISA in 2004, gave us a quick description of what the early show floor looked like:
I coordinated the vendor space for LISA V. We gave each of them half of an 8-ft banquet table, an electrical receptacle, and a trash can. I was quite impressed with some of the booths that were created in such a space.
Alongside LISA’s growth, the computing industry as a whole was growing. By the mid 1990s, everybody was riding the wave of the DotCom bubble. LISA was the place to find “those computer geeks”, and it became a popular recruitment and marketing event. Companies spent lavishly at the conference trying to attract the best talent they could find.
From the DotCom Bust to the Great Recession: 2001 - 2009
The seemingly infinite money and influence that computing enjoyed in the late 1990s came crashing down in the early 2000s. Lots of companies folded, and the ones who survived quickly learned the importance of scalability and efficiency. These topics were behind much of the work presented at earlier LISAs, and as the crash started to recover, a second wave of more efficient and agile companies started to emerge.
In its first decade, LISA was a largely academic conference: it had a strong research papers track, and many of its speakers and attendees were from universities and research laboratories. During its second decade, the research and engineering work that these sites did to automate and scale their systems were rapidly picked up by the next wave of dot-com companies. LISA shifted from having a mainly academic focus to serving the industry it helped create.
In 2002, Jim Reese, Chief Operations Engineer at Google, gave the LISA keynote titled "Scaling the Web: An Overview of Google (A Linux Cluster for Fun and Profit)". In its description, Google looks quaint by today's standards:
Want to know how to build an Internet search engine that indexes several terabytes of data—over 3 billion Web documents—and serves it up at a rate of thousands of requests per second? (Hint: Start with a farm of 10,000+ Linux servers.) This talk will cover the technology behind Google: company overview, search parameters and results, hardware and query load balancing, Linux cluster topology, scalability, fault tolerance, and more.
Over this decade, we saw the modern software stack emerge. Fighting spam by hand gave way to AI and ML algorithms that did the same and much more; automating bespoke sendmail configurations gave way to deploying hundreds or thousands of web servers with the same configurations; high availability file servers gave way to constantly available web services; terabytes of backup storage gave way to petabytes of tiered storage; and virtualization and the infrastructure behind it grew each year. "The Cloud", the source of so many jokes then and still today, became a viable place to get work done, whether it was remote or on-premise. Talks at LISA covered all of these topics and more, and late night conversations in hotel lobbies helped advance all of these technologies.
By the mid 2000s, LISA had settled on a pretty stable schedule of six days long, with workshops running Sunday through Tuesday, technical sessions running Wednesday through Friday, and tutorials running all week.
LISA Technical sessions were generally split into four tracks during this time: one that was focused on academic papers; one that was focused on Guru sessions, or “office hours” with an expert in a field; and two focused on talks by industry experts. Technical sessions ran all day, from 9am through 5:30 pm, resulting in a wide variety of sessions for attendees to choose from. A frequent comment among LISA attendees was that there was too much good content to choose from, which is probably the best kind of complaint to get.
Tutorials at LISA were taught by some of the greatest minds in the industry and covered an enormous range of topics of interest to sysadmins: networking, security, documentation, programming, hiring, and more. They ran in full-day or half-day lengths, and there were frequently more than ten tutorials available on any given day. This was a time when managing systems was becoming an important profession, and much of the community learned both the basics and advanced topics from the LISA tutorial sessions.
While LISA reached a fairly steady size and schedule during these years, one particular conference during this decade stood out among attendees and was talked about for years to come. LISA 2003, held in San Diego, California, took place while wildfires were raging in the hills of southern California. Carolyn Rowland, chair of LISA 2012, described her own experience that year:
There was ash falling all over the rose gardens at the hotel and the sky was orange. The swimming pools were also grey with ash. It was like a dystopian novel.
Josh Simon, a long-time LISA attendee, picked up on the orange theme too:
2003 (San Diego Town and Country) was the year of the fire. The fact that the conference CFP and program graphics were all in orange and yellow tones was a complete coincidence... or Ellie and her team had a lot of 'splainin' to do.
By the end of the ‘00s, LISA had helped grow the system administration and engineering community from a job that people fell into to a profession that people sought out, and it helped define how large-scale systems were built. But as the decade closed, individuals in the community were starting to specialize on individual technologies and the community itself was splitting out into smaller sub-communities. Change was on its way.
The Final Years and The Fragmentation of System Administration: 2010 - 2021
Ben Rockwood's 2011 keynote titled "The DevOps Transformation" set the stage for the last decade of LISA. While the term "DevOps" is wildly overused today, Ben effectively used it to tie together many of the movements that were happening at the time and that have continued to permeate the LISA community. As he noted, system development and system operations were starting to overlap more and more, with developers managing more of their own infrastructure and sysadmins treating more of their infrastructure as software. The system administration landscape was changing, a trend that Ben continued to describe at LISA in 2014 with his talk titled “I am SysAdmin (And So Can You!)”.
By the 2010s, most sites that LISA attendees hailed from had grown large enough that no single person could hold all of a site's system administration information in their head. Members of the LISA community were becoming more specialized in their areas of expertise, and it was common to hear pockets of conversation in the halls talking much more in-depth about specific system engineering topics than had happened in the past.
This specialization led to a fantastic set of talks each year covering security, performance, networking, monitoring, file systems, and more, all given by people who were defining those topics. It also led the community to dig deeper into the work they were passionate about. Even the conference infrastructure started to specialize: the Terminal Room from early LISAs turned into LISA Build in 2014, an event that gave volunteers the opportunity to build the entire conference wired and wireless network in the days leading up to the technical sessions.
During this time, we saw several of the earliest LISA topics dramatically mature. Configuration management, one of the earliest specialization topics at LISA, eventually led to the infrastructure-as-code approach to system management that companies of all sizes use today; performance engineering, which was just known as "system tuning" in LISA's early days, became an important practice; monitoring, originally performed using a handful of bespoke shell scripts, led to an entire industry devoted to monitoring systems; and many others. During this decade, LISA proved what a success it had been over the previous twenty years.
That doesn't mean the recipe for the conference stayed the same, though. LISA’s organizers focused on casting a wider net for talks during this time, and creating a community that attracted more diverse speakers and attendees than the stereotypical “computer guy.” This was the result of intentional and continued work over many LISAs. Tutorials speaker Courtney Eckhardt mentioned:
LISA had better diversity among the speakers than in its audience, and those of us who spoke and attended noticed, and it made it clear to us that we were truly welcome because our voices were being promoted. I successfully recruited friends to attend and speak based on this and they also had great experiences.
And yet, LISA also became a victim of its own success. It was staggeringly large, with over 100 sessions spread over six days and a yearly attendance of over 1,000 people. Specialization and sub-communities were the obvious result, with day-long workshops and co-located events working to help keep the community together. Eventually, this specialization led to one of the most important points in the LISA timeline, when the LISA SRE community grew large enough to spin up an entire new conference in 2014. SREcon has grown and flourished since then, making it one of the largest and most successful LISA subgroups of all.
In 2018, LISA started its own transformation by returning its focus to its core community: systems engineers who worked close to hardware and operating systems. Its length dropped down to three days to better match similar USENIX conferences, and its content was curated to match its more narrowed focus. This format worked well and was used again in 2019, which unfortunately ended up being the final in-person LISA.
In early 2020, right before the LISA '20 CFP was due to be released, the whole world went into lockdown due to COVID-19. As a result, LISA '20 was canceled, and LISA '21 moved to an online-only event. While COVID didn't directly take LISA as one of its victims, it did have a large effect on the USENIX Association as a whole and on the plan to continue shifting LISA's focus to system engineering. While USENIX saw growth in other conferences, such as SREcon and Enigma, LISA saw the opposite as the industry evolved and specialized. After 2021's conference, the LISA steering committee concluded it was time to wind down the conference.
Wrapping it Up: The End of an Era
Thirty-four years ago, the original LISA CFP targeted sites that managed more than 100 users on a couple of systems. Today, many members of the LISA community work for organizations that handle many, many orders of magnitude more users and systems. Just as no one person can run an entire site anymore, no one conference can handle our community anymore. LISA started as a place where a relatively small set of people got together to talk about their problems managing critical systems, and it quickly grew beyond its initial sizes. As you look around the industry today, you may think that we lost something big in LISA’s closing. But if you look more closely, you’ll see that it is alive and well in what it created and inspired: many conferences covering systems engineering topics, whether sponsored by USENIX or by other organizations, can trace some of their origins back to LISA, and many of the tools, programming languages, and practices that we use today were developed, refined, and first presented at LISA.
Mike Julian summarized LISA’s closing very well on Twitter:
It's true though--I think the event has run its course. It's served its mission and served it well.
The industry has advanced, thanks, in part, to LISA and to the thousands-upon-thousands of people it influenced over 30+ years.
So, thanks for a great run LISA.
You did well.
We'll take it from here.
LISA may be gone as an event, but it is still alive and strong in the community it helped grow. We’re all thankful for what LISA gave us, and we’re all looking forward to what the next 35 years of advanced systems brings us.
Acknowledgements
Just as LISA was not a single-person show, this article was not a single-person effort. I’d like to thank all of the people who helped with writing and researching this article: Laura Nolan, for driving the original idea, as well as Adam Moskowicz, Brendan Gregg, Courtney Eckhardt, Greg Rose, Josh Simon, Lee Damon, Pat Cable, Pat Wilson, Rik Farrow, Rob Kolstad, and Tony Del Porto for their historical and editing contributions. In addition, I want to give a shout out to everybody who responded on Twitter, Slack, and everywhere else with their memories and appreciation for LISA.
