Highly Automated Low Personnel System Administration in a Wall Street Environment

Administering a small system running intensive financial applications with the demand for twenty four hour coverage is nearly impossible for a single person. Moreover, most solutions to automated system administration are not clever enough to pinpoint the problem and deliver the required action to the responsible person in a timely manner. Our system evolved by integrating small utilities which function as information gathering tools, notifying tools and action tools. Vital information to us means: the users on each system, load on each server, temperature in the computer room, remaining swap and UNIX file system space on each machine in the company, health of our market data ticker plant, and state of backup tapes. Notifying tools include electronic mail, electronic postit notes, a digitized dial-up voice messaging system, and alphanumeric pagers. Action tools include fairly standard UNIX utilities such as killing runaway application processes, removing core dumps, and, as such, require little further description.