Eleventh Systems Administration Conference (LISA '97)
SAN DIEGO, CA
October 26-31, 1997
Generation X in IT
Randy Johnson and Harris Kern, R&H Associates Inc.
Summary by Carolyn M. Hennings
Randy Johnson and Harris Kern spoke about the characteristics of a
portion of today's workforce referred to as Generation X and the impact
it has on traditional IT departments. The challenge to existing IT
departments is identifying the nature of the Generation X workforce,
clarifying why these characteristics are potentially an issue, and
determining how to manage the situation in the future.
In the early 1990s, industry labelled Generation X persons born
between 1964 and 1978 as "slackers"; however, most are
entrepreneurial, like change and diversity, and are technically
literate. In contrast, the traditional IT organization was built on
control and discipline.
As technology has moved away from a single machine to a networked
computing model, the nature of the IT business has changed. The
speakers noted that IT departments had historically relinquished
control of personal computers and local area networks. IT management
has come to the realization that these are essential elements of the
success of mission-critical applications. As a result, there must be
Johnson and Kern suggested IT management focus on the following areas:
Teamwork. Encourage people to work together and rely on individuals to
do their jobs.
Communication. Improve communication within the organization and with
Involvement. Rather than direction from above, involve the team in
decisions and planning.
People. Encourage a "can do, be smart" attitude with some discipline.
Process. Institute the minimum and sufficient processes to support the
They suggested that this could be considered "creating Generation Y."
These people and relationships will be needed to build successful IT
organizations. The IT department must become a true services
organization. To accomplish this, the department must win back the
responsibility for technology decisions, reculture the staff to support
diversity and change, market and sell the services, train staff
members, and focus on customer satisfaction.
The department must communicate within the IT organization and with
customers. Defining architectures, standards, support agreements, and
objectives will make great strides in this area. The definition and
support of the infrastructure from the desktop to the network, data
center, and operations is an essential step. Defining "production" and
what it means to the customer in terms of reliability, availability,
and serviceability goes a long way in opening communication and
System management processes with standards and procedures modified from
the mainframe discipline are necessary steps. The speakers cautioned
organizations against bureaucracy and suggested focusing on producing
only "minimum and sufficient documentation." Implemen-ting deployment
methodologies and processes was strongly encouraged, as well as
developing tools for automating these processes.
REFEREED PAPERS TRACK
Summaries by Bruce Alan Wynn
Implementing a Generalized Tool for Network Monitoring
Marcus J. Ranum, Kent Landfield, Mike Stolarchuk, Mark Sienkiewicz,
Andrew Lambeth, and Eric Wall, Network Flight Recorder Inc.
Most network administrators realize that it is impossible to make a
network unbreachable; the key to network security is to make your site
more difficult than another so would-be intruders find easier pickings
In this presentation, Ranum further postulated that when a network
break-in does occur, the best reaction (after repelling the invader) is
to determine how access was gained so you can block that hole in your
security. To do this, the author presents us with an architecture and
toolkit for building network traffic analysis and event records: the
Network Flight Recorder (NFR). The name reflects the similarity of
purpose to that of an aircraft's flight recorder, or "black box," which
can be analyzed after an event to determine the root cause.
Further, he postulated that information about network traffic over time
may be used for trend analysis: identifying approaching bottlenecks as
traffic increases, monitoring the use of key applications, and even
monitoring the network traffic at peak usage periods in order to plan
the best time for network maintenance. Thus, this information would be
useful for network managers in planning their future growth.
The NFR monitors a promiscuous packet interface in order to pass
visible traffic to an internally programmed decision engine. This
engine uses filters, which are written in a high-level filter
description language, read into the engine, compiled, and preserved as
byte-code instructions for fast execution. Events that pass through the
filters are passed to a combination of statistical and logging back-end
programs. The output of these back-ends can be represented graphically
as histograms or as raw data.
Ranum can be reached at <firstname.lastname@example.org>; the complete NFR source
code, including documentation, Java class source, decision engine, and
space manager, is currently available from <https://www.nfr.net>for noncommercial research use.
Extensible, Scalable Monitoring for Clusters of Computers
Eric Anderson, University of California, Berkeley
The Cluster Administration using Relational Databases (CARD) system is
capable of monitoring large clusters of cooperating computers. Using a
Java applet as its primary interface, CARD allows users to monitor the
cluster through their browser.
CARD monitors system statistics such as CPU utilization, disk usage,
and executing processes. These data are stored in a relational database
for ease and flexibility of retrieval. This allows new CARD subsystems
to access the data without modifying the old subsystems. CARD also
includes a Java applet that graphically displays information about the
data. This visualization tool utilizes statistical aggregation to
display increasing amounts of data without increasing the amount of
screen space used. The resulting information loss is reduced by varying
shades of the same color to display dispersion.
Anderson can be reached at <email@example.com>. CARD is
available from <https://now.cs.berkeley.edu/Sysadmin/esm/intro.html>.
Monitoring Application Use with License Server Logs
Jon Finke, Polytechnic Institute
Many companies purchase software licenses using their best estimate of
the number required. Often, the only time this number changes is when
users need additional licenses. A side effect of this is that many
companies pay for unused software licenses. In this presentation, Jon
Finke described a tool for monitoring the use of licensed software
applications by examining license server logs.
This tool evolved from one designed to track workstation usage by
monitoring entries in the wtmp files. Because most license
servers record similar information (albeit in often radically different
formats), the tool was modified to monitor license use.
Information can be displayed in a spreadsheet or as a series of linear
graphs. The graphs provide an easy visual estimate of the number of
software licenses actually in use at a given point in time, or over a
period of time. Analysis of this information can quickly uncover
unneeded licenses at your site.
Currently, the tool interfaces with Xess (a commercial spreadsheet
available from Applied Information Services), Oracle, and Simon
(available from <ftp://ftp.rpi.edu/pub/its-release/simon/README.simon>).
Finke can be contacted at <firstname.lastname@example.org>.
Session: The Business of System Administration
Summaries by Brad C. Johnson
Automating 24x7 Support Response to Telephone Requests
Peter Scott, California Institute of Technology
Scott has designed a system, called helpline, that provides automated
answering of a help desk telephone during nonpeak hours and is used for
notifying on-call staff of emergencies within a short amount of time
(minutes or seconds) once a situation is logged in the system
(scheduler) database. This system was designed mainly to be cheap and
therefore mostly applicable to sites with low support budgets. The
system is comprised of source code written in Perl, the main scheduler
information base written in SGML, and two dedicated modems one
for incoming calls (for problem reporting) and one for outgoing calls
The rationale for creating helpline is that most other available
software that was sufficient to provide automated support cost more
than $100,000. Several tools that cost less were discovered, but they
did not provide sufficient notification methods (such as voice, pager,
and email according to a schedule). Recent entries into this market
include a Telamon product called Tel Alert, which requires proprietary
hardware, and VoiceGuide from Katalina Technologies, which runs only on
Windows. There is also available some freeware software called tpage,
but it concentrates on pagers, not on voice phones.
The key to the system is a voice-capable modem. When an incoming call
is answered by the modem daemon, it presents a standard hierarchical
phone menu a series of prerecorded sound files that are linked
to the appropriate menu choice. Independent of the phone menu system is
the notifier and scheduler component. When an emergency notification
occurs, the scheduler parses a schedule file (written in SGML) to
determine who is on call at the time, determines what profile (i.e.,
action) is appropriate based on the time and situation, and takes the
action to contact the designated on-call person. Multiple actions can
be associated with an event, and if the primary notification method
fails, alternate methods can be invoked.
Unfortunately, this software may not be completed for a long time (if
ever) because funding and staff have been assigned to other projects,
although the current state of the source code is available for review.
Send email with contact information and the reason for your request to
<email@example.com>. Additionally, in its current state, there are
some significant well-known deficiencies such as data synchronization
problems (which require specialized modem software), (over)sensitivity
to the load of the local host (a host that is assumed to be reliably
available), and virtually no available hard copy documentation.
Turning the Corner: Upgrading Yourself from "System Clerk" to "System Advocate"
Tom Limoncelli, Bell Labs, Lucent Technologies
Limoncelli believes that many administrators can be classified in one
of two ways: as a system clerk or as a system advocate. A system clerk
takes orders, focuses mainly on clerical tasks, and performs many
duties manually. A system advocate is focused on making things better,
automates redundant tasks, works issues and plans from soup to nuts,
and treats users as customers to create respectful, understanding
partnerships for resolving problems. The key to job satisfaction,
feeling better, and getting better raises is to make the transition
from clerk to advocate.
Making a successful transition to system advocate requires converting
bad (subservient) habits into good (cooperative) ones, creating spare
time for better communication and quality time for planning and
research, and automating mundane and repetitive tasks.
Although changing habits is always hard, it's important to concentrate
on getting a single success. Follow that with another and another, and
over time these experiences will accumulate and become the foundation
for good habits.
To find spare time, people need to think outside of the box and be more
critical and selective about where their time is spent. Suggestions for
regaining time include stop reading USENET, get off mailing lists, take
a time management course, filter mail (and just delete the ones that
you can't get to in a reasonable time e.g., at the end of the
week), and meet with your boss (or key customer) to prioritize your
tasks and remove extraneous activities from your workload.
Automating tasks, within the realm of a system administrator, requires
competency in programming languages such as Perl, Awk, and MAKE. These
are languages that have proven to be robust and provide the
functionality that is necessary to automate complex tasks.
Transforming the role of clerk to advocate is hard and requires a
change in attitude and working style to improve the quality of work
life, provide more value to customers, and create a more professional
and rewarding environment. However, the effort required to make this
transition is worth it. Simply put, vendors can automate the clerical
side of system administration, but no vendor can automate the value of
a system advocate.
How to Control and Manage Change in a Commercial Data Center Without
Losing Your Mind
Sally J. Howden and Frank B. Northrup, Distributed Computing
Howden and Northrup presented a methodology to ensure rigor and control
over changes to a customer's computing environment. They (strongly)
believe that the vast majority of problems created today are caused by
change. When change occurs unsuccessfully, the result can range from
lost productivity to financial loss. Change is defined as any action
that has the potential to change the environment and must consider the
impact from software, hardware, and people. Using the rigorous method
that was outlined will lower the overall risk and time spent on
problems. They believe that this rigor is required for all changes, not
just for significant or complex ones.
There are eight main steps outlined in this methodology: (1) Establish
and document a base line for the entire environment. (2) Understand the
characteristics of the change. (3) Test the changes in both an informal
test and formal preproduction environment. (4) Fully document the
change before, during, and after implementation. (5) Review the change
with all involved parties before placing it into the production
environment. (6) Define a detailed back-out strategy if the change fails in the
production environment. (7) Provide training and education for all
parties involved in the change. (8) Periodically revisit the roles and
responsibilities associated with the change.
The authors were quite firm about testing a change in three physically
distinct and separate environments. The first phase includes (unit)
testing of the change on the host(s) involved in development. The
second phase requires testing in a preproduction environment that, in
the best case, is an exact duplicate of the production environment. The
third phase is placing the change in the actual production environment.
When pressed on the suitability of using this (heavyweight) process on
all changes, the authors stated that the highest priority activities
are to fully document change logs and to create thorough work plans.
The paper notes, however, that although this process does generate a
amount of work by the administrators before a given change, it has
(over time) shown to reduce the overall time spent especially
for repeated tasks, when transferring information to other staff, when
secondary staff are on duty, and when diagnosing problems.
Session: System Design Perspectives
Summaries by Mark K. Mellis
Developing Interim Systems
Jennifer Caetta, NASA Jet Propulsion Laboratory
Caetta addressed the opportunities presented by building systems in the
real world and keeping them running in the face of budgetary
She discussed the role of interim systems in a computing environment
systems that bridge the gap between today's operational
necessities and the upgrades that are due three years from now. She
presented the principles behind her system design philosophy, including
her extensions to the existing body of work in the area. Supporting the
more academic discourse are a number of cogent examples from her work
supporting the Radio Science Systems Group at JPL. I especially enjoyed
her description of interfacing a legacy stand-alone DSP to a
SparcStation 5 via the DSP's console serial port that exposed the
original programmer's assumption that no one would type more than 1,024
commands at the console without rebooting.
Caetta described points to consider when evaluating potential interim
systems projects, leveraging projects to provide options when the
promised replacement system is delayed or canceled, and truly creative
strategies for financing system development.
A Large Scale Data Warehouse Application Case Study
Dan Pollack, America Online
Pollack described the design and implementation of a
greater-than-one-terabyte data warehouse used by his organization for
decision support. He addressed such issues as sizing, tuning, backups,
performance tradeoffs and day-to-day operations.
He presented in a straightforward manner the problems faced by truly
large computing systems: terabytes of disk, gigabytes of RAM,
double-digit numbers of CPUs, 50 Mbyte/sec backup rates all in a
single system. America Online has more than nine million customers, and
when you keep even a little bit of data on each of them, it adds up
fast. When you manipulate that data, it is always computationally
The bulk of the presentation discussed the design of the mass storage
IO subsystem, detailing various RAID configurations, controller
contention factors, backup issues, and nearline storage of "dormant"
data sets. It was a fascinating examination of how to balance the
requirements of data availability, raw throughput, and the state of the
art in UNIX computation systems. He also described the compromises made
in the system design to allow for manageable system administration. For
instance, if AOL strictly followed the database vendor's
recommendations, they would have needed to use several hundred file
systems to house their data set. By judicious use of very large file
systems so as to avoid disk and controller contention, they were able
to use a few large (!) file systems and stripe the two gigabyte data
files across multiple spindles, thereby preserving both system
performance and their own sanity.
Shuse At Two: Multi-Host Account Administration
Henry Spencer, SP Systems
Spencer's presentation described his experiences in implementing and
maintaining the Shuse system he first described at LISA '96. He details
the adaptation of Shuse to support a wholesale ISP business and its
further evolution at its original home, Sheridan College, and imparted
further software engineering and system design wisdom.
Shuse is a multi-host administration system for managing user accounts
in large user communities, into the tens of thousands of users. It uses
a centralized architecture. It is written almost entirely in the expect
language. (There are only about one hundred lines of C in the system.)
Shuse was initially deployed at Sheridan College in 1995.
Perhaps the most significant force acting on Shuse was its adaptation
for ISP use. Spencer described the changes needed, such as a
distributed account maintenance UI, and reflected that along with
exposing Sheridan-specific assumptions, the exercise also revealed
unanticipated synergy, with features requested by the ISP being adopted
A principal area of improvement has been in generalizing useful
facilities. Spencer observed in his paper, "Every time we've put effort
into cleaning up and generalizing Shuse's innards, we've regretted not
doing it sooner. Many things have become easier this way; many of the
remaining internal nuisances are concentrated in areas which haven't
had such an overhaul lately."
Other improvements have been in eliminating shared knowledge by making
data-transmission formats self-describing, and in the ripping out of
"bright ideas" that turned out to be dim and replacing them with
simpler approaches. These efforts have payed off handsomely by making
later changes easier.
Spencer went on to describe changes in the administrative interfaces of
Shuse, and in its error recovery and reporting.
Shuse is still not available to the general public, but Spencer
encourages those who might be interested in using Shuse to contact him
Spencer's paper is the second in what I hope will become a series on
Shuse. As a system designer and implementor myself, I look forward to
objective presentations of experiences with computing systems. It's a
real treat when I can follow the growth of a system and learn how it
has changed in response to real-world pressures and constraints. Often
papers describe a system that has just been deployed or is in the
process of being deployed; it is rare to see how that system has grown
and what the development team has learned from it.
Session: Works in Progress
Summaries by Bruce Alan Wynn
Service Level Monitoring
Jim Trocki, Transmeta Corp.
Many system and network administrators have developed their own simple
tools for automating system monitoring. The problem, proposes Jim
Trocki, is that these tools often evolve into something unlike the
original and in fact are not "designed" at all.
Instead, Jim presents us with mon: a Perl 5 utility, developed
on Linux and tested on Solaris. mon attempts to solve 85% of
the typical monitoring problems. The authors developed mon
based upon these guidelines:
Simple works best.
Separate testing code from alert generation code.
Status must be tracked over time.
The mon tool accepts input from external events and "monitors"
(programs that test conditions and return a true/false value). The
mon processes then examine these data and decide which should
be presented directly to clients and which should trigger an alarm.
The authors are currently expanding the functionality of mon
to include dependency checking of events, severity escalation, alert
acknowledgments via the client, "I'm okay now" events, asynchronous
events, a Web interface, and a better name.
The current version of mon is available at
Jim Trocki can be reached at <firstname.lastname@example.org>.
License Management: LICCNTL Control License Protected
Software Tools Conveniently
Wilfried Gaensheimer, Siemens AG
Gaensheimer presented an overview of a number of tools that can help
control and monitor the use of software licenses. The tools can also
generate reports of license use over time.
For additional information on these tools, contact Gaensheimer at
Todd Williams, MacNeal-Schwendler Corp.
One of the less exciting tasks that system and network administrators
are often faced with is that of taking a physical inventory. Typical
reasons for this requirement include:
Maintenance contract renewal
Charge backs for resource use
Identifying the type of machinery
Williams began tracking his inventory by including comments in the
system's "hosts" files, but quickly outgrew this mechanism when devices
appeared that did not have an IP address, and when the amount of
information desired made the "hosts" table unwieldy.
Instead, Williams developed a database to track this information. He
developed procedures to keep this information up to date as machinery
moves in and out of the work site.
For additional information on these software tools for tracking
inventory, contact Todd Williams at <email@example.com>.
Steve Tylock, Kodak
Although it may initially seem a surprising topic for a technical
conference, Tylock reintroduced the basic values of a Fortune 500
respect for the dignity of the individual
continuous improvement and personal renewal
Instead of applying these to the company itself, Tylock suggested that
system and network administrators could increase their professionalism
and efficiency by applying these basic values to their daily work.
For more information on this topic, contact Steve Tylock at
Extending a Problem-Tracking System with PDAs
Dave Barren, Convergent Group
Many system and network administrators use one type of problem-tracking
system or another. But because working on the typical system or network
problem often means working away from one's desk, administrators must
keep track of ticket status independently of the tracking system. When
administrators return to their desk, they must "dump" the information
into the tracking system, hoping that they don't mis-key data or get
interrupted by another crisis.
To help alleviate this problem, Barren suggests using a PDA to track
ticket status. Barren has developed a relatively simple program for his
Pilot that allows him to download the tickets, work the problems, track
ticket status on the Pilot, then return to his desk and upload the
changes in ticket status in one easy step.
This allows Barren to work on more tickets before returning to his desk
and increases the validity of the tracking system. Barren hopes to
encourage more users to implement this plan so that the increased
number of Pilots will allow him to upload ticket status information at
virtually any desk instead of returning to his own.
For additional information on this concept and the software tools
Barren has developed, contact him at <firstname.lastname@example.org>.
Survey of SMTP
Dave Parter, University of Wisconsin
One of the beautiful things about the Simple Mail Transfer Protocol is
that it allows people to use any number of transfer agents to deliver
electronic mail across the world. The down side is that there is a
hodgepodge of versions and "brands" of transfer agents in use, and
nobody really knows what is in use these days. Except, perhaps, Dave
To examine this issue, Parter monitored the incoming mail at his site
for a short period of time. For each site that sent mail to his, he
tested the SMTP greeting and tried to identify the type and version of
the agent. His results:
Parter was able to identify 140 distinct versions of sendmail in use in
this small sampling.
Where, Parter asks, do we go from here with these data? He isn't sure.
If you would like to discuss these findings, or conduct your own
survey, contact Parter at <email@example.com>.
Session: Net Gains
Summaries by Mark K. Mellis
Creating a Network for Lucent Bell Labs Research South
Tom Limoncelli, Tom Reingold, Ravi Narayan, and Ralph Loura, Bell
Labs, Lucent Technologies
This presentation described how, as a result of the split of AT&T
Bell Labs Research into AT&T Labs and Lucent Bell Labs, they
transitioned from an "organically grown" network consisting of four
main user communities and ten main IP nets (out of a total of 40 class
C IP nets) to a systematically designed network with two main user
communities on four main IP nets, renumbering, rewiring, cleaning up,
and "storming the hallways" as they went.
Unlike many projects of this scope, the authors planned the work as a
phased transition, using techniques such as running multiple IP
networks on the same media and operating the legacy NIS configuration
in parallel with the new config to transition slowly to the new
configuration, rather than make all the changes during an extended down
time and discover a critical error at the end. They relate their
experiences in detail, including a comprehensive set of lessons learned
about strategy, end-user communications, and morale maintenance. ("Yell
a loud chant before you storm the hallways. It psyches you up and makes
your users more willing to get out of the way.'')
Having been faced with a network unfortunately typical in its
complexity, and real-world constraints on system downtime, this group
described their thought processes and methodologies for solving one of
the problems of our time, corporate reorganization. In the face of
obstacles such as not having access to the union-run wiring closets and
"The Broken Network Conundrum," where one must decide between fixing
things and explaining to the users why they don't work, they divided
their networks, fixed the problems, and got a cool T-shirt with a
picture of a chainsaw on it, to boot.
Some the tools constructed for this project are available at
Pinpointing System Performance Issues
Douglas L. Urner, BSDI
Urner gave us a well-structured presentation that within the context of
a case study on Web server performance optimization presents a
systematic model for tuning services from the network connection,
through the application and operating system, all the way to the
hardware. His paper is a vest-pocket text on how to make it go faster,
regardless of what "it" might be.
Urner began the paper by describing an overview of system tuning:
methodology, architecture, configuration, application tuning, and
kernel tuning. He discussed the need to understand the specifics of the
problem at hand protocol performance, application knowledge,
data collection and reduction. He then described tuning at the
subsystem level, including file system, network, kernel, and memory. He
presented a detailed explanation of disk subsystem performance, then
went on to examine CPU performance, kernel tuning, and profiling both
application and kernel code.
Urner's paper is about optimizing Web server performance, but it is
really about much more. He describes, in detail, how to look at
performance optimization in general. He encourages readers to develop
their intuition and to establish reasonable bounds on performance. By
estimating optimal performance, the system designer can determine which
of the many "knobs" in an application environment are worth "turning",
and help set reasonable expectations on what can be accomplished
through system tuning.
Session: Configuration Management
Summaries by Karl Buck
The first two papers deal with the actual implementations of tools
written to handle the specific problems. The third paper is an attempt
to get a higher level view of where configuration management is today
and make suggestions for improving existing CM models.
Automation of Site Configuration Management
Jon Finke, Rensselaer Polytechnic Institute
Finke presented his implementation of a system that not only tracks
interesting physical configuration aspects of UNIX servers, but also
stores and displays dependencies between the servers and the services
that they provide. The configuration management system has an Oracle
engine and outputs data to a Web tree, making for a very extensible,
useful tool. For instance, if a license server is to be updated, one
can find out not only all the other services that will be affected, but
also the severity of those outages and who to contact for those
services. Source code is available; see
Chaos Out of Order: A Simple, Scalable File Distribution Facility
for "Intentionally Heterogeneous" Networks
Alva L. Couch, Tufts University
The core of this paper is a file distribution tool written by Couch
called DISTR. Using DISTR, administrators of unrelated networks can use
the same file distribution system, yet retain control of their own
systems. DISTR can "export" and "import" files to and from systems
managed by other people. Frank discussion is given to the existing
limitations and potential. DISTR is available at
An Analysis of UNIX System Configuration
Remy Evard, Argonne National Laboratory
This paper is an attempt to step back and take a look at what is
available for use in UNIX configuration and file management, examine a
few case studies, and make some observations concerning the current
configuration process. Finally, Evard argues for a "stronger
abstraction" model in systems management, and makes some suggestions on
how this can be accomplished.
Summaries by Mark K. Mellis
Tuning Sendmail for Large Mailing Lists
Rob Kolstad, BSDI
Kolstad delivered a paper that described the efforts to reduce delivery
latency in the <firstname.lastname@example.org> mailing list. This mailing
list bursts to up to 400,000 message deliveries per day. As a result of
the tuning process, latency was reduced to less than five minutes from
previous levels that reached five days.
Kolstad described himself as a member of Optimizers Anonymous, and he
shared his obsession with us. He described the process by which he and
his team analyzed the problem, gathered data on the specifics, and
iterated on solutions. He took us through several rounds of data
analysis and experimentation, and illustrated how establishing
realistic bounds on performance and pursuing those bounds can lead to
insights on the problem at hand.
Kolstad and his team eventually homed in on the approach of increasing
the parallelism to the extreme of using hundreds of concurrent sendmail
processes to deliver the list. They also reduced timeouts for
nonresponsive hosts. This, of course, required the creation of a number
of scripts to automate the parallel queue creation. These scripts are
available upon request from Kolstad, <email@example.com>.
Kolstad closed by noting that after the optimizations were made, the
biggest remaining problem was unavailability of recipients. He
expressed his amazement that in a mailing list dedicated to Internet
service providers, some one to three per cent of recipients were
unreachable at any point in time. Also, even with these improvements,
the mailing list traffic of mostly small messages doesn't tax even a
single T-1 to its limits.
Selectively Rejecting SPAM Using Sendmail
Robert Harker, Harker Systems
Harker offered a presentation that addressed one of the hottest topics
on the Internet today unsolicited commercial email, otherwise
known as spam. He characterizes spam, examines the different
requirements for antispam processing at different classes of sites, and
offers concrete examples of sendmail configurations that address these
After his initial discussion of the nature of spam, Harker outlined the
different criteria that can be used for accepting and rejecting email.
His approach differs from others in that he spends sendmail CPU cycles
to get finer granularity in the decision to reject a message. He goes
on to treat the problem of spammers sending their wares to internal
aliases and mailing lists.
The remainder of the presentation was devoted to detailed development
of the sendmail rulesets necessary to implement these policies. He
discussed the specific rulesets and databases needed, and how to test
the results. His discussion and code are available at
A Better E-mail Bouncer
Richard J. Holland, Rockwell Collins
Holland presented work that was motivated by corporate reorganization:
how to handle email address namespace collisions in a constructive way.
As email usage becomes more accessible to a wider spectrum of our
society, fewer and fewer email users are able to parse the headers in a
bounced message. Holland talked about his bouncer, implemented as a
mail delivery agent, which provides a clearly written explanation of
what happened and why when an email message bounces due to an address
change. This helps the sender understand how to get a message through,
helps the recipient get a message, and helps the postmaster by
automating another portion of her workload.
The bouncer was originally implemented as a simple filter. Because of
the diversity in headers and issues related to envelope vs. header
addresses, especially in the case of bcc: addresses, the bouncer was
reimplemented as a delivery agent. The bouncer, written in Perl,
relinquishes its privilege and runs as "nobody." Many of the aspects of
bouncer operation are configurable, including the text of the
explanatory text to be returned. A particularly nice feature is the
ability to send a reminder message to the recipient's new address when
mail of bulk, list, or junk precedence is received, reminding them to
update their mailing list subscriptions with the new address.
Holland concluded by discussing alternatives to the chosen
implementation and future directions. Those interested in obtaining the
bouncer should contact Holland at <firstname.lastname@example.org>.
INVITED TALKS TRACK
So Now You Are the Project Manager
William E. Howell, Glaxo Wellcome Inc.
Summary by Bruce Alan Wynn
Many technical experts find themselves gaining responsibility for
planning and implementing successively larger projects until one day
they realize that they have become a project manager.
In this presentation, Howell offered helpful advice on how you can
succeed in this new role without the benefit of formal training in
Howell's first suggestion is to find a mentor, someone who has
successfully managed projects for some time. Learn from that mentor not
only what the steps are in managing a project, but also the reasons why
those are the right steps.
But, as Howell points out, a mentor is not always available. What do
you do then? Howell presented a few tips on what you can do if you
can't find a mentor.
For copies of the presentation slides, contact Marie Sands at
<email@example.com>; please include both your email and
When UNIX Met Air Traffic Control
Jim Reid, RTFM Ltd.
Summary by Mike Wei
Every once in a while we see reports of mishaps of the rapidly aging
air traffic control (ATC) system in the United States. We have also
seen reports that some developing countries have ATC systems "several
generations newer" than the US system. For most of the flying public,
the ATC system is something near a total mystery on which our lives
depend. As a pilot and a system administrator, I hope I can lift the
shroud of mystery a little bit and help explain the ATC system Reid
talked about, how UNIX handles such a mission-critical system, and how
this system helps air traffic control.
The primary purpose of air traffic control is traffic separation,
although it occasionally helps pilots navigate out of trouble.
Government aviation authorities publish extensive and comprehensive
regulations on how aircraft should operate in the air and on the
ground. Air traffic control is a massively complex system of computers,
radar, controllers, and pilots that ensures proper traffic separation
and flow. Human participants (i.e., controllers and pilots) are as
essential as the computer and radar systems.
Naturally, air traffic congestion happens near major airports, called
"terminal areas." In busy terminal areas, computer-connected radar
systems provide controllers with realtime traffic situations in the
sky. Each aircraft has a device called a transponder that encodes its
identity in its radar replies, so the controllers know which aircraft
is which on the computer screen. Computer software along with
traffic controllers ensure proper separation and traffic flow by
vectoring planes within the airspace to their destinations.
Outside terminal areas, large planes usually don't fly anywhere they
want. They follow certain routes, like highways in the sky. On-route
traffic control centers control traffic along those routes. Traffic
separation is usually ensured by altitude separation or fairly large
horizontal separation. Some on-route centers have radar to help track
the traffic. For areas without radar coverage, on-route centers rely on
pilot position reports to track the traffic and usually give very large
This system worked fairly well for many years, until air travel reached
record levels. Two things happened. First, some terminal areas became
so congested that, during some parts of the day, the airspace just
couldn't hold any more traffic. Second, traffic among some terminal
areas reached such a level that these on-route airspaces became almost
as congested as terminal areas.
A new kind of system was developed to address the new problems. This
"slot allocation system" tries to predict what the sky will look like
in the future, based on the flight plan filed by airliners. Based on
the computer prediction, we can allocate "slots" in the sky for a
particular flight, from one terminal area to another, including the
on-route airspace in between. Every airline flight is required a flight
plan, including departure time, estimated time on-route, cruising
airspeed, planned route, destination, and alternate destination. With
the flight plan, an airplane's position in the sky is fairly
This slot allocation system is very much like TCP congestion control in
computer networking: when the network is congested, the best way to
operate is to stop feeding new packets into it for a while. For the
same reason it's much better to delay some departures than to let
planes take off and wait in the sky if the computer system predicts
congestion sometime in the future.
The Western European airspace, according to Reid, is the busiest
airspace in the world. Instead of a single controlling authority, like
the US Federal Aviation Authority, each country has its own aviation
authorities. Before "Eurocontrol," the agency Reid worked at last year,
each country managed its airspace separately, and an airliner had to
file a flight plan for each country it had to fly over along its route.
This led to a chaotic situation when traffic volume increased.
According to Reid, there was also a problem of ATC nepotism (i.e., a
country favoring its own airliners when congestion occurred).
The Eurocontrol agency has three UNIX-based systems that serve Western
Europe. IFPS is a centralized flight plan submission and distribution
system, TACT is the realtime slot allocation system, and RPL is the
repeat flight plan system.
IFPS provides a single point of contact for all the flight plans in
Western Europe. It eliminates the inconvenience of filing multiple
flight plans. This is basically a mission-critical data entry/retrieval
The TACT system provides slot allocation based on the flight plan
information in the IFPS system. It provides slots that satisfy
separation standards in the airspace above Western Europe. It controls
when an airplane can take off and which slots in the sky it can fly
through to its destination. It keeps a "mental picture" of all the air
traffic in the sky for all the
moments into some future. RPL is the repeat flight plan system.
Airlines tend to have the same flights repeatedly, and this system
simplifies filing those flight plans. The RPL system is connected with
the IFPS system and feeds it with those repeat flight plans.
This must be an awesomely impressive system with equally impressive
complexity. According to Reid, it actually works. Ever since the
adoption of the system, it has never failed. Furthermore, the increase
in traffic delay is much less than the increase in traffic volume.
Kudos for our European computer professionals!
The slot allocation system does not provide the actual traffic
separation. Realtime traffic separation must be based on actual
position data obtained from radar or pilot position report, rather
than projected position data based on flight plan. However, this
slot allocation system is an invaluable tool to help the realtime
traffic separation by avoiding congestion in the first place.
Using UNIX in such a mission-critical system is quite pioneering in an
ATC system. Most ATC systems in the US are still mainframe-based. The
system is built on multiprocessor HP T90 servers, and the code is
written in Ada.
Like most of the mission-critical systems, operation of those UNIX
systems has its idiosyncrasies. According to Reid, the system operation
suffers organizational and procedural inefficiencies. However, some of
them may well be the necessary price to pay for such a mission-critical
system. The whole system is highly redundant; almost all equipment has
a spare. The maintenance downtime is limited to one hour a month.
Change control on the system is the strictest I've ever heard of. For
new code releases, it has a test environment fed with real data, and
there's a dedicated test group that does nothing
but the testing. Any change to the production systems must be
documented as a change request and approved by a change board, which
meets once a week. Any kind of change, including fixing the sticky bit
on /tmp, needs change board approval. Reid said that it took
SA six weeks to fix the /tmp permissions on six machines
because each one needed a change request and only one change a week is
allowed on the production system. To minimize the chance of system
failure, all nonessential service on the system is turned off,
including NFS, NIS, and all other SA life-saving tools. This does add
pain to SA's daily life.
This kind of process sounds bureaucratic, and it's a far cry from a
common UNIX SA's habit. However, for this kind of system, it might be
right to be overly conservative. At least when Reid flew to the LISA
conference this year, he knew nothing bad would likely happen to
Eurocontrol due to a system administrator's mistake.
Enterprise Backup and Recovery: Do You Need a Commercial Utility?
W. Curtis Preston, Collective Technologies
Summary by Bruce Alan Wynn
Nearly every system administrator has been asked to back up
filesystems. Even those who haven't have probably been asked to recover
a missing file that was inadvertently deleted or corrupted. How can a
system administrator determine the best solution for a backup strategy?
In this presentation, Preston presented an overview of standard
utilities available on UNIX operating systems: which ones are common,
which ones are OS-specific. He then explained the capabilities and
limitations of each. In many cases, claims Preston, these "standard"
utilities are sufficient for a site's backup needs.
For sites where these tools are insufficient, Preston discussed many of
the features available in commercial backup products. Because some
features require special hardware, Preston described some of the
current tape robots and media available. Once again, he iterated the
capabilities and limitations of each.
Copies of Preston' presentation are available upon request; Preston can
be reached at <firstname.lastname@example.org>.
A Technologist Looks at Management
Steve Johnson, Transmeta Corp.
Summary by Bruce Alan Wynn
Employees often view their management structure as a bad yet necessary
thing. Johnson has worked in the technical arena for years, but has
also had the opportunity to manage research and development teams in a
number of companies. In this presentation, he offered his insight into
methods that will smooth the relationship between employees and
Johnson began by postulating that both employees and managers have a
picture of what the manager-employee relationship should look like, but
it is seldom a shared picture. He further postulated that a great deal
of the disconnect is a result of personality and communication styles
rather than job title.
Johnson loosely categorized people as either thinkers, feelers, or
act-ers. A thinker focuses on analyzing and understanding; a feeler
focuses on meeting the needs of others; an act-er focuses on activity
These differences in values, combined with our tendency to presume
others think as we do, cause a breakdown in communication that leads to
many of the traditional employee-manager relationship problems.
After making this point, Johnson suggested that technical people who
are given the opportunity to move into management first examine closely
what the job entails: it's not about power and authority; it's about
meeting business needs. He suggested a number of publications for
additional information on this topic.
Steve Johnson can be reached at <email@example.com>.
IPv6 Deployment on the 6bone
Bob Fink, Lawrence Berkeley National Laboratory
Summary by Mike Wei
We all know that IPv6 is the future of the Internet; there's simply no
alternative to support the explosive growth of the Internet. However,
despite years of talking, we see little IPv6 deployment. According to
Fink, the adaptation and deployment of IPv6 is currently well under
way, and it's heading in the right direction.
An experimental IPv6 network, named 6bone, was created to link up early
IPv6 adopters. It also serves as a test bed to gain operational
experiences with IPv6. Because most of the machines on the 6bone also
run regular IPv4, it provides an environment to gain experience in IPv4
to v6 transition.
The 6bone is truly a global network that links up 29 countries. Most of
the long haul links are actually IPv6 traffic tunnelled through the
existing IPv4 Internet. This strategy allows 6bone to expand
anywhere that has an Internet connection for almost no cost. On the
6bone network, there are some "islands" of network that run IPv6
natively on top of the physical network.
An important milestone was achieved in IPv6 deployment when Cisco,
along with other major router companies, committed to IPv6. According
to Fink, IPv6 will be supported by routers in the very near future, if
it's not already supported. In addition, we will start to see IPv6
support in major OS releases.
A typical host on the 6bone runs two IP stacks, the traditional v4
stack and the IPv6 stack. The IPv6 stack can run natively on top of the
MAC layer if the local network supports v6, or it can tunnel through
IPv4. The v6 stack will be automatically used if the machine talks to
another v6 host. An important component of the 6bone network will be
the new DNS that supports IPv6 addresses. The new DNS supports AAAA
record (quad-A record, because a v6 address is four times the length of
a v4 address). If a v6 host queries the new DNS server for another v6
host, an AAAA record will be returned. Because the new DNS simply maps
a fully qualified domain name to an IP address (v4 or v6), the DNS
server itself doesn't have to sit on a v6 network. It will be perfectly
normal for a dual-stack v6/v4 host to query a DNS server on the v4
network, getting a v6 address, and talk to the v6 host in IPv6.
The key to the success of IPv6 deployment is smooth transition. The
transition should be so smooth that a regular user should never know
when the IPv6 has arrived. Given the fact that the IPv4 network is so
far reaching throughout the
world, IPv6 and v4 will coexist for a very long time; the transition to
IPv6 from v4 will be gradual. Routers will be the first ones that have
IPv6 capabilities. Just like the 6bone, an IPv6 backbone can be built
by tunnelling v6 traffic through the existing v4 network or run v6
natively on the physical network when two neighboring routers both
support v6. Because v6 is just another network layer protocol, it can
run side by side with IPv4 on the same physical wire without conflict,
like IP and IPX can run together on the same Ethernet. This means that
we do not have to make a choice between v6 and v4; we can simply run
both of them during the transition period. IP hosts will gradually
become IPv6 capable when the new OS versions support it. During the
transition, those IPv6 hosts will have dual IP stacks so they can talk
to both v4 and v6 hosts. Nobody knows how long this "coexist" will
last, but it will surely last for years. When the majority of the hosts
on the Internet are doing v6, some of the hosts might choose to be v6
only. One by one, the v4 hosts will fade away from the Internet.
Will that ever happen? The answer is yes. In the next decade, the IPv4
address will be so hard to obtain that IPv6 will be a very viable and
attractive choice. We haven't seen that yet, but based on the current
Internet growth, it will happen.
The IPv6 addressing scheme is another topic Fink talked about in the
seminar. IPv6 has a 128 bit address space, which allows thousands of
addresses per square
foot if evenly spread on the earth's surface. How to make use of this
address space in a highly scalable way is a big challenge. IPv4 suffers
the problem of an explosive number of routing entries, and this problem
arises years before the exhaustion of IPv4 addresses. To address this
problem and to allow decades of future expansions, IPv6 uses an
aggregation-based addressing scheme.
3bits 13bits 32bits 16bits 64 bits
001 TLA NLA SLA Interface ID
public topology site local machine topology
The best analogy of this aggregation-based addressing is the telephone
number system. We have ten-digit phone numbers in US and Canada, with a
three-digit area code, three-digit exchange code, and the last four
digits for individual telephone lines.
The first three bits are 001. In the great tradition of TCP/IP, other
combinations are reserved for future use, in case one day we have an
interplanetary communication need that requires a different addressing
scheme. The 13-bit TLAs are top-level aggregators, designed to be given
to long-haul providers and big telcos that run backbone service. The
32-bit NLAs are next-level aggregators for various levels of ISPs. It
can be further subdivided to several levels of NLAs. The 16-bit SLAs
are for site topologies. (It's like getting a class A IPv4 address and
use 16-bit for network address.) The machine interface ID is 64 bits.
An important feature of IPv6 is autoconfiguration, in which a host can
figure out its own IPv6 address automatically. The 64-bit interface ID
is designed so that the host can use its data link layer interface
address as the host portion of the IPv6 address. Ethernet uses a 48-bit
address, and it seems adequate for globally unique addresses. Reserving
64 bits for the local machine shall accommodate any future address
method used by future physical networks.
Aggregation-based addressing is a big departure from the current IPv4
addressing. Although IPv4 has three classes of addresses, it's not a
hierarchical addressing scheme. In IPv4 (at least before the CIDR
days), all the network's addresses were created equal, which means they
could all be independently routed to any locations they chose to be.
This caused the routing entry explosion problem when the Internet grew.
Classless Inter Domain Routing (CIDR) was introduced as a stopgap
measure to address this urgent problem by introducing some hierarchy in
the IPv4 address space. IPv6 is designed at the beginning with a
hierarchical scheme. By limiting the number of bits for each
aggregator, there is an upper limit to the number of routing entries
that a router needs to handle. For example, a router at a long-haul
provider needs only to look at the 13-bit TLA portion of the address,
limiting the possible number of routing entries to 213.
Another advantage of a hierarchical-based addressing system is that
address allocation can be delegated in a hierarchical manner. The
success of DNS teaches us the important lesson that delegation of
address allocation authority is a key to scalability.
There's a price to pay to use a hierarchical addressing system. When a
site changes its providers, all the IP addresses need to
be changed. We already experience the same kind of issue in IPv4 when
we use CIDR address blocks. IPv6 tries to make address changes as
painless as possible, to have a host autoconfigure itself. The host
will use its MAC layer address as the lower portion of its IPv6 address
and use a Network Discovery protocol to find out the upper portion of
the address (routing prefixes). The whole site can be renumbered by
simply rebooting all the hosts without any human intervention.
There are still lots of problems to be discovered and addressed in
IPv6. That's exactly what the 6bone is built for. IPv6 is the future of
the Internet, and the transition to IPv6 will start in the near future.
More information on 6bone can be found on <https://www.6bone.net>
Panel: Is System Administration a Dead-End Career?
Moderator: Celeste Stokely, Stokely Consulting
Panelists: Ruth Milner, NRAO; Hal Pomeranz, Deer Run Associates; Wendy
Nather, Swiss Bank Warburg; Bill Howell, Glaxo Wellcome Inc.
Summary by Carolyn M. Hennings
Ruth Milner opened the discussion by responding to the question with,
"It depends." She went on to explain that it is necessary for everyone
to define "system administration" and "dead-end career" to answer this
question for themselves. In some organizations, "system administration"
leaves no room for growth. However, Ruth pointed out that if people
enjoy what they do, then maybe it should not be considered a
Hal Pomeranz outlined the typical career progression for system
administrators. He described the first three years in the
career field as a time of learning while receiving significant
direction from more senior administrators. During the third through
fifth years of practicing system administration, Hal suggested that
even more learning takes place as the individual works with a greater
degree of autonomy. Hal observed that people with more than five years
of experience are not learning as much as they were, but are more
focused on producing results as well as mentoring and directing others.
Hal commented that many organizations move these senior people into
management positions and wondered how technical tracks might work.
Wendy Nather discussed the question from the angle of recruiting. Those
hiring system administrators are looking for people who have dealt with
a large number of problems as well as a variety of problems. She
pointed out that being a system administrator is a good springboard to
other career paths. Wendy outlined some of the characteristics of good
system administrators that are beneficial in other career areas: a
positive attitude, social skills, open-mindedness, and flexibility.
Bill Howell examined the financial prospects for system administrators.
He commented that there will always be a need for system
administrators. However, industry may be unable and unwilling to
continue to pay high salaries for them, and salary increases may begin
to be limited to "cost of living" increases. Bill suggested that growth
in personal income and increases in standard of living are the results
of career advancement. If salaries
do become more restricted in the future, system administration may
become a dead-end career.
Celeste then opened up the floor for questions and discussion. One
participant asked about other career options if one was not interested
in pursuing the managerial or consultant path. The panel suggested that
specializing in an area such as network or security administration
would be appropriate. Discussion ranged among topics such as motivation
for changing positions, how the size of the managed environment affects
opportunities and working relationships, the impact of Windows NT on
UNIX administrator's careers, how an administrator's relationship with
management changes with career advancement, and the importance of
promoting system administration as a profession.
Summaries by Carolyn M. Hennings
Keeping a Local SAGE Group Active
This BOF at LISA '96 and SANS '97 inspired me to start a group in
Chicago. Chigrp's initial meeting was in early October, and I was
anxious to announce our existence. General suggestions for getting a
group started and keeping one alive were shared by attendees. If you
want more information on how to start a group, see
As system administrators, we all know how important documentation is,
but we hate to write it. This BOF explored some of the reasons we don't
like to write documentation, elements of good documentation, and what
we can do personally to improve our efforts in this area. About 50
people attended the BOF. Some professional technical writers
participated in the BOF and were interested in the approach sys admins
were taking in their struggle to write documentation.