|Pp. 77-83 of the Proceedings|
Pelendur: Steward of the Sysadmin
Matt Curtin - Interhack Corporation
Sandy Farrar & Tami King - The Ohio State University
Here we describe Pelendur, a system for the management of
common system operation tasks. Specifically, Pelendur focuses
on the management of user accounts and related information (such as
groups) across platforms and even for particular software packages
(like databases) that require user authentication. Pelendur
has reduced the massive process of deleting expired accounts and
creating new accounts between terms from a week-long operation by
several part-time operators (with subsequent cleanup by a
collaboration of instructors and staff) into a completely automated
process that requires less than 15 minutes of staff work and
completely eliminates the need for instructor intervention.
In 1998, our department was in the midst of a massive migration of
our computing facilities wherein we moved from an architecture of many
HP-UX clusters to an architecture using Solaris-based function-specific servers with thin clients in offices and labs. Some of the
software used in the old environment needed to be replaced .
Because of severe limitations in the functionality and correctness of
the largely ad-hoc scripts run by operators for the creation of
accounts, it was determined that an account management system capable
of managing our evolving multiplatform environment was needed.
Yet Another Account Management System?
The idea of implementing a software system for the management of
user accounts is not new; past years' LISA conferences have seen many
such systems. Even if we briefly ignore the issue of availability,
some systems described were unsuitable because of extreme differences
in the way that accounts are created and managed [1, 5] and
incompatible means of handling account data [7, 11, 9, 10, 4, 8].
In the end, the most compelling reason for us to build our own
software was that a grander vision existed: a single data repository
for our department, which would include such things as data needed for
user accounts, course-specific computing requirements, and access to
various limited-access department resources. We could not find any
available account management system that would work easily with any
sort of database that we would construct.
The Academic Environment
Account management, though a fairly straightforward task, is quite
demanding in an academic environment. Each term, we receive course
rosters from the university registrar. Although students who major in
computer and information science have ``permanent'' accounts (those
that will remain until after they graduate or change majors), we have
thousands of other accounts that are created specifically for the
duration of the term. We have one week between most terms, which means
that during this time, we need to delete potentially more than 2,500
accounts and then create another 3,000 accounts for the next term.
In this paper, and in our environment, we use a term that will be
important to understand: ``section''. This is a specific ``class''
that meets together. This term is introduced because many sections of
a given class can be scheduled for the same term and we need to know
the most granular level of grouping available from the registrar.
Specific requirements for this system were identified. In its full
design, our scope is actually much more broad than the rather specific
task of account management. The reason for this is that parts of
account management (such as determining how much quota to associate
with an account) are dependent on other criteria like the requirements
for courses that the user of that account is scheduled to take. For
example, a course that deals with particularly large data sets might
have a requirement for more than the default amount of disk quota.
Initial requirements focused on the actual management of user
accounts and dependencies.
- Account Management:
This is the management of individual account profiles. Adding,
editing, expiring, and purging them.
- Course Management
Courses have particular requirements (such as the need for one
platform or another and the use of a software system like
Sybase) that need to be configured and managed. Some of these
configuration options are simple matters of preference. Others are
matters of policy, which the course coordinator does not have the
authority to change. (Though we refer specifically to courses, there
is nothing that prevents these entities to be managed from being
project groups, departments, or any other sort of group that might
have particular requirements in other environments.)
- Resource Management:
Anything that exists in our environment (such as ``Unix machines'',
``NT machines'', ``Sybase database'', ``disk quota'', ``print quota'',
and ``color printers'') might also need to be managed. As these are
neither accounts nor courses, but share basic properties, we classify
all of these as ``resources''. The basic point here is that users who
are administratively responsible for these resources manage them
through Pelendur instead of having us perform all of the
management tasks for them.
Additional requirements were slated for future development. Some
of these features are now implemented and others are still on our
- Unix groups:
As we use groups to manage sets of users on the Unix systems,
Pelendur should be aware of groups and know how to use them.
(This interface is now partially implemented; new groups are created,
but old groups are not garbage collected.)
- Mailing lists:
Some of our course instructors and students prefer to use mailing
lists to stay in touch. We presently use both faculty-maintained
aliases files and Majordomo , but are now investigating the
possibility of replacing both of these mechanisms with Mailman
- Course directories:
Because some courses have group projects or software that is specific
to the course, we need to be able to associate directories with a
given course. Handling of filesystem permissions should enforce the
policy for read and write access established by the instructor.
- Multidomain management:
Currently, everything that is a part of the instructional environment
is considered ``the system''. Accounts that belong to individual
research labs are not managed by Pelendur but it could be
convenient for us to have that option available. Should we do this in
the future, it would be nice to have the option of having a single
Pelendur installation be able to manage multiple ``systems'',
rather than having to make a new installation of the software for each
domain that needs to be managed. This feature would also be useful to
include access to limited-access machines, such as those that are set
aside for long-term computationally-intensive jobs.
- Electronic lock systems:
A relatively new addition to our department is an electronic lock
system, whereby ID cards are used for access control instead of
physical keys. Presently, this is managed by a standalone DOS-based
- Course newsgroups:
Most courses in our environment is assigned a newsgroup on our local
news server. This is the default means of providing an ``out of
class'' communication channel. Presently, we just create new groups as
new courses are added and cancel the messages in the groups at the end
of each term by hand. Though this is not a big time-sink, we would
like for these processes to be automated.
- Course-specific environment:
Some courses have particular environmental needs, such as a course-specific $PATH setting, for example. Pelendur can
- ``Any computing resource'':
As we continue to move forward, other resources are identified and
incorporated into the system's functionality. At a very high level,
the goal of the system is to manage the systems' configurations so
that system administrators can do other things that computers can't do
very well, like planning.
Design and Implementation
Pelendur is a large system, made up of several programs.
We'll first describe the philosophies that influenced the system's
design and then consider the programs and major library modules that
these programs use.
The entire system sits atop a Sybase relational database.
Rather than creating code that would depend upon very specific data,
working with entities that make sense for our environment, we opted to
put as much of the system in data as possible and to make those data
be as generic and flexible as possible. Thus, rather than dealing with
courses, sections, and instructors, other users of Pelendur
will be able to work with teams, departments, and project
coordinators. Each deployment of Pelendur will define its own
terms and the relationships among them that make sense.
We believe it important to emphasize that this makes the integrity
of the database especially critical. In such a highly dynamic system,
we're not dealing with simple cases of the Wrong Thing failing to
achieve the desired result. Data that have been compromised by a
moderately clever attacker can be used to attack the system itself,
creating accounts for attackers, granting them privileges to the
entire system, and possibly even running commands with superuser
The schema is represented in Figure 1. Here we describe each of
these tables in some detail.
contains information about a person. When a person is added to the
table they are assigned a database identifier and a user login. Those
two values will be used to tie a person to their resources.
is a table that contains information about individual accounts.
contains information on thingies in the system. A thing can be
anything with the exception of a person or an individual account. Most
classifications define a resource or a membership group. There are
also templates and defaults in Classification that are used to create
resources and membership groups or define certain values in the system
(e.g., what the current quarter is). Each classification is assigned a
unique number (SID) when it is added to the database.
contains the memberships for the membership groups from
Classification. It maps a user login to a SID and is used to
determine what resources that login should have.
defines owners of classifications and grants access to users for a
classification. This allows a user to manage resources for their
defines what resources a classifications has. It also indicates if the
resource can be edited by the owner or proxy of the classification.
links classifications together.
1: Schema of Pelendur's Database
We made an effort to avoid ``hardwiring'' anything in the system.
This was largely accomplished by taking a very dynamic view of the
data. That is, instead of having an ``account'' with a ``disk quota''
field that would be assigned a value based on the state of the system
at the time of the account's creation, all information about an
account must go through a process of resolution, where its
dependencies are determined and the values for each of the account's
properties are resolved at run-time. This view was taken for
everything in the system, not just accounts.
Something else that we incorporated in order to allow maximal
flexibility is a system of property inheritance. Although dealing with
a relational database system at the core, we were able to provide the
ability to inherit properties from a parent by specifying a
relationship between various records in the database by defining a
``parent ID'' (Called ``PID'') as a means of determining which ``SID''
is one step closer to the root than the current.
This gives us the ability to specify object-oriented ``is-a''
relationships between records in the database. Thus, the ``tree'' of
elements to be resolved can be of an arbitrary depth, allowing each
site (and each particular type of resource being managed) to have an
appropriate number of levels to support the sort of abstraction
desired, without forcing some high level of overhead on those who do
not need to deal with such abstractions.
A good example of how we use the ability to inherit properties is
in the case of a series of courses that a student will take in
sequence. There will be some Classification table entry that will
identify the series. Each course in the series will have its own
Classification entry whose PID identifies the Classification entry of
the series as the parent. Each section in a course will have its own
Classification entry whose PID identifies the Classification entry of
the parent course. Thus, configuration changes are made at the
appropriate level: those that affect all sections in the series will
be made in the Classification entry for the series, those that affect
all sections in a specific course in the series will have those
changes made in the course's Classification entry, and those that are
specific to a particular section will be made in that section's
Classification entry. When we need to determine how much quota, for
example, an account has, we'll determine which sections in which the
account has membership. Those will be resolved by walking up the tree
until we get to the root object (as determined by having a PID of 0),
populating the fields in memory with the values in the database and
returning. By the time the initial Classification entry returns, we
will have queried each level in the tree, populating the object with
the levels specified at the highest level first, and overriding those
with whatever (if anything) was specified in the lower levels.
Additionally, where values are numeric, they need not be absolute;
signed numeric values indicate relative values. Consequently, a
property of a Classification might be to increase disk quota by 30MB,
instead of specifying some absolute value.
Rather than requiring code changes in many different places in
order to create new interfaces in the system, we designed the system
to be made up of a basic core which includes the database and the
resource resolution logic. The rest of the system interfaces to that
core. The core understands when an account needs to be created, or
what the properties of an account at any given time should be, but it
knows nothing about creating accounts. Instead, it can tell what the
account's properties are and what system(s) need to reflect those
properties. If the account needs access to both Unix and
Sybase, the core will pass the appropriate information to the
modules for Unix and Sybase through a standardized interface.
As described in section on implementation language, this design
makes it possible for even the database itself to be replaced with
another database that has the same schema.
Perl was chosen as the implementation language. Its freely
available DBI (database interface) and DBD (database driver) package
for talking to a wide variety of databases makes it an excellent
option for building atop a database:
Very little investment is made in an interface to any particular
database (since we code to DBI), thus allowing us (theoretically) to
use any database for which a DBD module exists. In practice, we did
use one Sybase-specific feature, which we can and will in the future
We can spend our time focusing on the problem at hand, instead of how
to write to the database.
Being free, the price is right.
Since source code is available, if there is a problem with it for
which a fix is not available, we can fix the bug ourselves.
Because parts of this system will need to run with superuser
privileges, we're concerned about safety of our code. Perl has some
excellent safety features: namely, the ability to identify ``tainted''
data and support for arrays and buffers that grow dynamically.
Finally, Perl is available on a huge number of platforms. As long
as we keep portability in mind, Perl will provide us all of the
language support necessary to allow our code to run unmodified on
essentially any system we could use in the forseeable future.
is the program that loads rosters into the database. In our
environment, rosters are course rosters that identify which students
and instructors should be associated with a given section. For the
most part, rosterload just calls
Roster::rosterload, which does all of the work.
Our plans are for rosterload and Roster.pm to be modified to
be able to load roster information directly out of the Data Warehouse
or from email.
is the program responsible for creating things, that is, resources and
accounts. First it will initialize the resource methods, then it
creates any resources that don't already exist. Then it creates any
accounts that don't exist. We have a cron job run
makeaccounts runs twice per day.
is our garbage collector; it's responsible for removing things from
the system. Expired resources are removed from both the database and
from the systems that were used to support the account. After this has
been accomplished, it will remove entries in the ``MembershipIn''
table that have expired and then any user resources that are specified
for the user in the database. cron runs expireaccounts
once per day.
sends account removal notifications. All accounts scheduled to be
removed will receive a seven day notification. Account that are
`persistent' will also receive a 30 and 14 day notification of
expiration. [Note 1]
Note that this includes not only accounts for operating systems, but
this also includes ``accounts'' in software systems like
Sybase. As Pelendur continues to help us blur the
distinctions among different systems that comprise the ``computing
environment'', we'll move away from providing notification that
specific systems' accounts will be unavailable and provide
notification only on the user's ``meta-account'' in the department.
The Sybase account will be created as soon as the student
shows up on the roster for a class that has this resource. The account
will always [Note 2]
match the Unix login name and will have the typical default password.
is the course configuration interface for instructors. This provides a
convenient means for them to manipulate the database in a controlled
manner, allowing them to change only that which is under their
administrative authority without creating artificial restrictions that
require staff to perform their administrative tasks for them.
Instructors and their ``proxies'' (those whom they designate) to add
and to remove resources from a particular section and a course
default. Project and course directories are also configured through
is the account management administration tool. Essentially, this is
the interface that is used to manipulate the state of the database on
anything that is in the database. Where crconfig deals with
abstractions and has a ``course-specific'' view of the world,
cradmin allows manipulation of non-course-related resources.
The user interface itself still works with many of these abstractions,
but it is in this program where we can make changes to the user
interface to allow administration of new resources.
Here we describe the major modules of the Pelendur system.
Many of these modules are shared by various standalone programs in the
is the module contains the generic routines for interacting with the
account management database. There is typically a routine for each
table for searching, adding, and removing. They follow the naming
convention get_<table-name>, add_<table-name>, remove_<table-name>. Table
names are converted to lowercase and underscores are used where there
would be white space (``MembershipIn'' becomes membership_in]).
For some tables an update function exists also. The
get_ routines are all polymorphic and will do different
searches depending on what data is passed to the routine. This module
also contains the routines to walk the database recursively
(resolve_pids) in order to resolve all dependencies and
provide an up-to-the-second view of an account's properties, as
determined by what the database knows about the account.
is our logging mechanism. Presently, this basically accepts messages
and puts them in the ``right place'', but the intention is that it
will be a general-purpose log gatherer for all applications in our
contains the routines that deals with logins in our environment. It
contains the routines that generates new usernames and default
contains the routines to do all things on the NT systems. It is
currently not implemented; an older standalone account creation and
deletion system was developed locally for NT accounts. Instead of
implementing this part of the system initially, we opted to focus on
the Unix, Sybase, and core database portions of the system,
building an interface between Pelendur and the old NT account
management scripts. This decision has turned out to work relatively
well for us, and has saved us from what could have become a large
amount of redundant work as Microsoft seems to think that making major
interface changes from version to version of its operating systems is
an appropriate thing to do. Thus, we can avoid writing most of NT.pm
until the NT based systems are on a newer version of the operating
system than the current programs support.
interfaces with the printing system's quota handling. We use LPRng 
for our printing system throughout the department.
is a module that contains all of the methods for the resources in the
account management system. Each resource has its method defined for it
in the METHOD field in its ``Classification'' entry.
When called, this method will `do the right thing' for the resource.
For example, a resource like ``mailing list'' might have a method that
specifies a program to be run in order to create or to delete the
contains the routines for processing the roster.
contains all of the routines for dealing with Sybase
contains all of the routines for Unix resources. It adds and removes
accounts in the password file, manipulates the group file, and
generates the Quotas file for disk quotas.
is used to create and to remove directories on the Unix systems. It
handles several types of directories: users' home directories, group
project directories, and the directories used by Submit, our
program for electronic laboratory submissions. Where possible, this
module will run as the owner of the directory instead of root.
The Effect of Pelendur
Our environment has benefited tremendously from Pelendur
in many ways. What used to be a painful experience for all is now
essentially a non-event.
The total staff labor expenditure for processing accounts between
terms is greatly reduced.
Accounts to be removed are no longer processed manually. When an
account has no more references (managed through the
MembershipIn table), the account is garbage collected.
Instructors would manually add and remove students from their sections
using hardcopy provided by the university registrar. We now get these
data from the registrar directly and automatically add and remove
Historically, this has been a problem. The old software had to run
by someone who knew its idiosyncrasies and limitations. Mistakes were
frequent and could easily require several hours to fix. Errors are now
much less frequent, because we get data directly from the registrar
and do not require any manual intervention before account
configuration. Because the entire state of the system is driven by
the database, errors can now be fixed by making appropriate changes in
the database and waiting for Pelendur to propagate them.
Since managing our systems with Pelendur, we have been
able to identify accounts that have expired long ago but were never
removed, to identify problems that arise because of an account being
misclassified, and generally to free ourselves of the kinds of
concerns that come about when the administrators need to manage things
In section ``Programs'', we identified which parts of the system
run on a regular basis in our environment. These are configurable to a
site's requirements. In our environment specifically, this means that
changes made take no more than one day to take effect. This is a huge
difference from the days, weeks, or even more that it took under the
Quite a lot can still be done with Pelendur. Specifically,
we need to increase the number of systems against which we can
interface, including native NT account management, more intelligent
Account management can be greatly simplified by taking a more
abstract view and thinking of system access as a property that results
from the state of the account. Pelendur has proven to be a
highly effective means of managing a very large number of highly
Although the system has been designed and implemented in a way
that emphasizes flexibility and freedom from very a very site-specific
view of the world, it will still take quite a bit of effort for
another site to bring Pelendur into production. We're
currently working on finishing the functionality and hope that we will
be able to revisit some of the areas of the system that work for us
but would make it difficult or impossible for other sites to use the
system as-is. This work is geared toward making a general release of
the system. We have no idea when this could possibly take place.
Matt Curtin is founder of Interhack Corporation, which helps
developers and system managers build and run systems they can trust.
He is also a Lecturer at The Ohio State University's Department of
Computer and Information Science. His current interests include Lisp
programming, secure systems development, and Internet privacy. Reach
him electronically at
Tami King is a senior software specialist for the Computer and
Information Science Department at The Ohio State University. Since
graduating from Utah State University with a B.S. in computer science,
she has worked as programmer, UNIX systems administrator, and database
administrator. She also managed a group of UNIX systems administrators
before returning to programming, what she enjoys most. Her free time
is taken up by her husband, her new baby boy, and pedaling out of
sync. Reach her electronically at
 Arnold, Bob, ``Accountworks: Users create accounts on SQL, notes,
NT, and UNIX,'' Twelfth Systems Administration Conference (LISA
'98), page 49, USENIX, Boston, Massachusetts, December 6-11 1998.
 Chapman, D. Brent, ``Majordomo: How Imanage 17 mailing lists
without answering ``-request'' mail,'' Systems Administration (LISA
VI) Conference, pages 135-143, Long Beach, CA, USENIX, October
 Curtin, Matt, Creating an Environment for Reusable
Software Research: A Case Study in Reusability, Technical Report
OSU-CISRC-8/99-TR21, The Ohio State University, Department of Computer
and Information Science, August 1999.
 Geer, Daniel E. Jr., ``Service Management at Project
Athena,'' Large Installation Systems Administration Workshop
Proceedings, page 71, Monterey, CA, USENIX, November 17-18 1988.
 Harris, J. Archer and Gregory Gingerich, ``The Design and
Implementation of a Network Account Management System,'' 10th
Systems Administration Conference (LISA'96), pages 33-41, Chicago,
IL, USENIX, September 29-October 4 1996.
 Powell, Patrick and Justin Mason, ``Lprng - An Enhanced
Printer Spooler System'' Ninth Systems Administration Conference
(LISA '95), pages 13-24, Monterey, CA, USENIX, September 17-22
 Riddle, Paul, Paul Danckaert, and Matt Metaferia, ``AGUS: An
Automatic Multi-platform Account Generation System,'' In Ninth
Systems Administration Conference (LISA '95), pages 171-180,
Monterey, CA, USENIX, September 17-22 1995.
 Rosenstein, Mark A., Daniel E. Geer, Jr., and Peter J.
Levine, ``The Athena Service Management System,'' USENIX Conference
Proceedings, pages 203-211, Dallas, TX, USENIX, Winter 1988.
 Spencer, Henry, ``Shuse: Multi-host Account Administration,''
10th Systems Administration Conference (LISA'96), pages 25-32,
Chicago, IL, USENIX, September 29-October 4 1996.
 Spencer, Spencer, ``Shuse at two: Multi-host Account
Administration,'' Eleventh Systems Administration Conference (LISA
'97), page 65, San Diego, USENIX, California, October 26-31 1997.
 Tomas, Gregory S., James O. Schroeder, Merrilee E. Orcutt,
Desiree C. Johnson, Jeffrey T. Simmelink, and John P. Moore,
``UNIXhost Administration in a Heterogeneous Distributed Computing
Environment,'' 10th Systems Administration Conference
(LISA'96), pages 43-50, Chicago, IL, USENIX, September 29-October
 John Viega, Barry Warsaw, and Ken Manheimer, ``Mailman: The
GNUmailing List Manager,'' Twelfth Systems Administration
Conference (LISA '98), page 309, Boston, Massachusetts, USENIX,
December 6-11 1998.
Note 1: The amounts of time on
the notification are configurable parameters.
Note 2: ``Always'' is a pretty
strong word. There are a few exceptions, as the result of ancient
accounts that predate Pelendur that still contain dashes
(-) so they can't be used as logins to Sybase. If
the username has a dash in it, the dash will be converted to an
underscore (_) for the Sybase account. New
accounts are always created with names that are portable across
our systems so that these kinds of conversions will not be