Why Order Matters: Turing Equivalence in Automated Systems Administration
Steve Traugott - TerraLuna, LLC Lance Brown - National Institute of Environmental
Pp. 99-120 of the Proceedings of LISA '02:
Sixteenth Systems Administration Conference,
USENIX Association, 2002).
Hosts in a well-architected enterprise infrastructure are self-administered; they perform their own maintenance and upgrades. By
definition, self-administered hosts execute self-modifying code. They
do not behave according to simple state machine rules, but can
incorporate complex feedback loops and evolutionary recursion.
The implications of this behavior are of immediate concern to the
reliability, security, and ownership costs of enterprise and mission-critical computing. In retrospect, it appears that the same concerns
also apply to manually-administered machines, in which administrators
use tools that execute in the context of the target disk to change the
contents of the same disk. The self-modifying behavior of both manual
and automatic administration techniques helps explain the difficulty
and expense of maintaining high availability and security in
The practice of infrastructure architecture tool design exists to
bring order to this self-referential chaos. Conventional systems
administration can be greatly improved upon through discipline,
culture, and adoption of practices better fitted to enterprise needs.
Creating a low-cost maintenance strategy largely remains an art. What
can we do to put this art into the hands of relatively junior
administrators? We think that part of the answer includes adopting a
well-proven strategy for maintenance tools, based in part upon the
theoretical properties of computing.
In this paper, we equate self-administered hosts to Turing
machines in order to help build a theoretical foundation for
understanding this behavior. We discuss some tools that provide
mechanisms for reliably managing self-administered hosts, using
deterministic ordering techniques.
Based on our findings, it appears that no tool, written in any
language, can predictably administer an enterprise infrastructure
without maintaining a deterministic, repeatable order of changes on
each host. The runtime environment for any tool always executes in the
context of the target operating system; changes can affect the
behavior of the tool itself, creating circular dependencies. The
behavior of these changes may be difficult to predict in advance, so
testing is necessary to validate changed hosts. Once changes have been
validated in testing they must be replicated in production in the same
order in which they were tested, due to these same circular
The least-cost method of managing multiple hosts also appears to
be deterministic ordering. All other known management methods seem to
include either more testing or higher risk for each host managed.
This paper is a living document; revisions and discussion can be
found at Infrastructures.Org, a project of TerraLuna, LLC.
- View the full text of this paper in
PDF, and Postscript. Until November 2003, you will need your USENIX membership identification in order to access the full papers.
The Proceedings are published as a collective work, © 2002 by the USENIX Association. All Rights Reserved. Rights to individual papers remain with the author or the author's employer. Permission is granted for the noncommercial reproduction of the complete work for educational or research purposes. USENIX acknowledges all trademarks within this paper.
- If you need the latest Adobe Acrobat Reader, you can download it from Adobe's site.