Check out the new USENIX Web site.
18th Large Installation System Administration Conference, November 14-19, Atlanta, GA
LISA '04 Home            USENIX Home            Events            Publications            Membership



Overview | By Day (Sunday, Monday, Tuesday, Wednesday, Thursday, Friday) | By Instructor | All in One File

Sunday, November 14, 2004
S1 Hands-on Linux Security Class: Learn How to Defend Linux/UNIX Systems by Learning to Think Like a Hacker (Day 1 of 2) NEW!
Rik Farrow, Security Consultant
9:00 a.m.–5:00 p.m.

Who should attend: System administrators of Linux and other UNIX systems; anyone who runs a public UNIX server.

Few people enjoy learning how to swim by being tossed into the ocean, but that's what happens if a system you manage gets hacked. You often have little choice other than to reload that system, patch it, and get it running again. This two-day class gives you a chance to work with systems that have been "hacked," letting you search for hidden files or services or other evidence of the intrusion. Examples are taken from real, recent attacks on Linux systems. You will perform hands-on exercises with dual-use tools to replicate what intruders do as well as with tools dedicated to security. The tools vary from the ordinary, such as find and strings, to less familiar but very important ones, such as lsof, scanners, sniffers, and the Sleuth Kit.

The lecture portion of this class covers the background you need to understand UNIX security principles, TCP/IP, scanning, and popular attack strategies.

Day Two will explore the defenses for networks and individual systems. The class will end with a discussion of the use of patching tools for Linux, including cfengine.

Class exercises will require that you have an x86-based laptop computer that can be booted from a KNOPPIX CD. Macintosh owners interested in taking this class should contact the instructor, as a bootable KNOPPIX CD for the PPC may be provided as well if there is sufficient interest. Students will receive a version of Linux on CD that includes the tools, files, and exercises used in the course. If you have a laptop but don't know whether it can run a bootable Linux CD (that will not have an impact on your installed hard drive or operating systems), please download a copy of KNOPPIX (, burn it, and try it out. KNOPPIX support for wireless is the same as common Linux kernels (not exciting), but KNOPPIX does a superb job of handling most other hardware found in laptops.

Exercises include:


  • Finding hidden files and evidence of intrusion
  • TCP/IP and its abuses
  • hping2 probes while using ethereal
  • nmap while watching with ethereal or tcpdump (connect and SYN scans)
  • Working with buffer-overflow exploit examples
  • Apache servers and finding bugs in scripts
  • John the Ripper, password cracking
  • Elevation of privilege and suid shells
  • Rootkits, and finding rootkits (chkrootkit)
  • Sleuth Kit (looking at intrusion timelines)
  • iptables and netfilter
  • cfengine configuration
Rik Farrow (S1, M1) provides UNIX and Internet security consulting and training. Rik Farrow He has been working with UNIX system security since 1984 and with TCP/IP networks since 1988. He has taught at the IRS, Department of Justice, NSA, NASA, US West, Canadian RCMP, Swedish Navy, and for many US and European user groups. He is the author of UNIX System Security, published by Addison-Wesley in 1991, and System Administrator's Guide to System V (Prentice Hall, 1989). Farrow writes a column for ;login: and a network security column for Network magazine. Rik lives with his family in the high desert of northern Arizona and enjoys hiking and mountain biking when time permits.

S2 System and Network Monitoring
John Sellens, Certainty Solutions
9:00 a.m.–5:00 p.m.

Who should attend: Network and system administrators interested in real-life, practical, host- and network-based monitoring of their systems and networks. Participants should have an understanding of the fundamentals of networking, basic familiarity with computing and network components, and some familiarity with UNIX and scripting languages.

Participants will leave this tutorial able to immediately start using a number of monitoring systems and techniques that will improve their ability to manage and maintain their systems and networks.

Topics include:

  • Monitoring: goals, techniques, reporting
  • SNMP: the protocol, reference materials, relevant RFCs
  • Introduction to SNMP MIBs (Management Information Bases)
  • SNMP tools and libraries
  • Other (non-SNMP) tools
  • Security concerns when using SNMP and other tools on the network
  • Monitoring applications: introductions, use, benefits and complications, installation and configuration (Big Brother, Nagios, SNIPS, MRTG, Cricket, etc.)
  • Special situations: remote locations, firewalls, etc.
  • Monitoring implementation roadmap: policies, practices, notifications, escalations, reporting
John Sellens (S2, M2) has been involved in system and network administration John Sellens since 1986 and is the author of several related USENIX papers, a number of ;login: articles, and the SAGE Short Topics in System Administration booklet #7, System and Network Administration for Higher Reliability. He holds an M.Math. in computer science from the University of Waterloo and is a chartered accountant. He is the proprietor of SYONEX, a systems and networks consultancy. From 1999 to 2004, he was the General Manager for Certainty Solutions in Toronto. Prior to joining Certainty, John was the Director of Network Engineering at UUNET Canada and was a staff member in computing and information technology at the University of Waterloo for 11 years.

S3 Seven Habits of the Highly Effective System Administrator
Mike Ciavarella, University of Melbourne, and Lee Damon, University of Washington
9:00 a.m.–5:00 p.m.

Who should attend: Junior system administrators with anywhere from little to 3+ years of experience in computer system administration. We will focus on enabling the junior system administrator to "do it right the first time." Some topics will use UNIX-specific tools as examples, but the class is applicable to any sysadmin and any OS. Most of the material covered is "the other 90%" of system administration—things every sysadmin needs to do and to know, but which aren't details of specific technical implementation.

We aim to accelerate the experience curve for junior system administrators by teaching them the time honored tricks (and effective coping strategies) that experienced administrators take for granted and which are necessary for successful growth of both the administrator and the site.

The class covers many of the best practices that senior administrators have long incorporated in their work. We will touch on tools you should use, as well as tools you should try to avoid. We will touch on things that come up frequently, as well as those which happen only once or twice a year. We will look at a basic security approach.

We will talk about issues such as why your computers should all agree on what time it is, why root passwords should not be the same on every computer, why backing up every filesystem on every computer is not always a good idea, policies - where you want them and where you might want to avoid them. Ethical issues, growth and success as a solo-sysadmin as well as in small, medium, and large teams. We will discuss training, mentoring and personal growth planning as well as site planning, budgeting and logistics. We will discuss books that can help you and your users.

Mike Ciavarella (S3, T7, T10) has been producing and editing technical documentation sinceMike Ciavarella he naively agreed to write application manuals for his first employer in the early 1980s. He has been a technical editor for MacMillan Press and has been teaching system administrators about documentation for the past eight years. Mike has an Honours Degree in Science from the University of Melbourne. After a number of years working as Senior Partner and head of the Security Practice for Cybersource Pty Ltd, Mike returned to his alma mater, the University of Melbourne. He now divides his time between teaching software engineering, providing expert testimony in computer security matters, and trying to complete a Doctorate. In his ever-diminishing spare time, Mike is a caffeine addict and photographer.

Lee Damon (S3) has a B.S. in Speech Communication from Oregon State University.Lee Damon He has been a UNIX system administrator since 1985 and has been active in SAGE since its inception. He assisted in developing a mixed AIX/SunOS environment at IBM Watson Research and has developed mixed environments for Gulfstream Aerospace and QUALCOMM. He is currently leading the development effort for the Nikola project at the University of Washington Electrical Engineering department. He chaired the SAGE Ethics Working Group and coordinated authorship of the initial draft of the current document. He has championed awareness of ethics in the system administration community, including writing it into policy documents.

S4 Solaris Kernel Performance, Observability, and Debugging NEW!
James Mauro and Richard McDougall, Sun Microsystems
9:00 a.m.–5:00 p.m.

Who should attend: System/database administrators and performance analysts wanting to obtain a deeper understanding of the key Solaris subsystems, as well as the tools and facilities that can be used to observe, trace, debug and optimize performance. Attendees should have some basic understanding of operating system principles and application performance analysis.

Applications are becoming more complex every day, and many of the new Solaris features significantly reduce the effort required to administer and anazlyze performance of the entire application and operating system stack. In this class we provide an architectual overview of the major Solaris subsystems, and methodologies for the end-to-end analysis and control.

Topics include:

  • Kernel debugging/monitoring tools
    • Introduction to core file analysis
    • Mastering Solaris DTrace
    • How to debug/monitor with 'mdb'
  • Performance monitoring and tuning
    • Using DTrace for performance optimization
    • Overview of Solaris perf tools
  • Process management & scheduling
    • Introduction to the Solaris process and thread model
    • Developing and tuning multi-threaded processes
    • Observing debugging processes with the ptools
    • Controlling processes with ptools
    • Introduction to scheduling
    • Controlling and observing scheduling behavior
  • File systems
    • Overview of Solaris file system architecture
    • Understanding caching
    • File systems in Solaris - UFS, NFS and the new S10 ZFS
    • Measurement and tuning
  • Memory
    • Overview of Solaris virtual memory
    • Observing and managing memory
    • Understanding memory utilization, optimizing and monitoring
  • Workload consolidation and resource management
    • Introduction to tools for workload and resource management
    • Workload measurement
    • Using Solaris resource manager to isolate and control workloads
    • Using zones to containerize applications

James Mauro (S4) is a Senior Staff Engineer in the Performance and AvailabilityJames Mauro Engineering group at Sun Microsystems. Jim's current projects are focused on quantifying and improving enterprise platform availability, including minimizing recovery times for data services and Solaris. Jim co-developed a framework for system availability measurement and benchmarking and is working on implementing this framework within Sun.

Richard McDougall (S4) is a Sun Microsystems Distinguished Engineer who specializes in Richard McDougalloperating systems technology and system performance. He is based at the Menlo Park Performance and Availability Engineering group, where he drives development of performance and behavior enhancements to the Solaris operating system and Sun's hardware architectures. He has led the development of resource management principles, has contributed to the development of virtual memory and file systems within the Solaris operating system, and has architected many tools for analysis, monitoring, and capacity planning. He is the lead author of Resource Management (Prentice Hall). He has written numerous articles and papers on measurement, monitoring, and capacity planning of Solaris systems and frequently speaks at industry and customer technical conferences on the topics of system performance and resource management.

Richard and Jim authored Solaris Internals: Architecture Tips and Techniques (Sun Microsystems Press/Prentice Hall, Feb 2000, ISBN 0-13-022496-0) and are currently collaborating on an update of the book for Solaris 8, as well as volume II.

S5 Bridges, Routers, Switches, and Internetworking Protocols
Radia Perlman, Sun Microsystems
9:00 a.m.–5:00 p.m.

Who should attend: Anyone who might need to design a protocol, implement a protocol, write network-based applications, or plan or manage a network. Anyone who is just curious about what is really going on under the covers in a network, and how things got the way they are. Anyone with the courage to see things from different angles, and not just parrot orthodoxy. Paradoxically, this tutorial is good as an introduction to people who are incredibly confused by all the terms and don't know where to start, as well as people who have been using this stuff for years, assumed they understood it, and want to see how all the pieces fit.

The concepts of IP addresses, masks, MAC addresses, routing algorithms, domains, switches, bridges, are pervasive when dealing with networks. We all use these terms, and configure these things, but what is really going on? What are the implications of choosing a switch vs a router? What kinds of things can go wrong in a protocol that is misdesigned, misimplemented, or mismanaged? This tutorial describes the major protocols involved in the network infrastructure. It describes conceptually what goes on in the packet switches (both layer 2/bridges and layer 3/routers), as well as the implications on endnodes. It contrasts connection-oriented approaches such as ATM and MPLS with connectionless approaches such as IPv4 and IPv6. It covers the endnode-visible pieces of layer 3, such as neighbor-discovery and address autoconfiguration. It covers intradomain routing algorithms (distance vector such as RIP and link state such as OSPF or IS-IS) and interdomain (BGP). It describes the spanning tree algorithm used by bridges/switches.

Topics include:

  • Layer 2 (MAC) addresses
    • Why 6 bytes?
    • Relation to layer 3 addresses (IP)
  • Bridges
    • Basic idea
    • Why it's more powerful than a repeater
    • Station address learning and forwarding
    • Spanning tree
  • What are switches? "switched Ethernet"
  • Connection-oriented networks: ATM, MPLS
  • Connectionless protocols: IPv4, IPv6, and comparison with others
  • Neighbor discovery (ARP, DHCP)
  • Routing (distance vector vs link state, interdomain vs intradomain)
  • IP Multicast
  • NAT

Radia Perlman (S5, M5) is a Distinguished Engineer at Sun Microsystems. She is knownRadia Perlman for her contributions to bridging (spanning tree algorithm) and routing (link state routing), as well as security (sabotage-proof networks). She is the author of Interconnections: Bridges, Routers, Switches, and Internetworking Protocols and co-author of Network Security: Private Communication in a Public World, two of the top ten networking reference books, according to Network Magazine. She is one of the twenty-five people whose work has most influenced the networking industry, according to Data Communications Magazine. She has about fifty issued patents, an S.B. and S.M. in mathematics and a Ph.D. in computer science from MIT, and an honorary doctorate from KTH, the Royal Institute of Technology in Sweden.

S6 Essential Topics in System Administration NEW!
Trent Hein and Ned McClain, Applied Trust Engineering
9:00 a.m.–5:00 p.m.

Who should attend: System and network administrators who are interested in picking up several new technologies quickly.

Topics include:

  • BIND9 Tips and Tricks: A Better DNS
    Most sites have migrated to BIND9, but are you really getting the most out of this major rewrite of the Internet's most popular nameserver? Learn about powerful new functionality such as split views, remote management, and even DNSSEC. This topic is a must for every modern administrator.
  • Rapid Linux Disaster Recovery
    Tape backups are essential, but they are not an efficient way to restore a server in an emergency. We evaluate the ins and outs of Mondo, an open source disaster recovery tool that can create bootable recovery CDs from any Linux server. When used in tandem with a solid tape backup system, Mondo recovery CDs can reduce "bare metal" recovery time from hours to minutes.
  • Linux Kernel Tuning
    As Linux's popularity in production environments grows, so does your need to know how to tune the Linux kernel, whether performance, security, or functionality is your goal. We'll give you the what-tos, the how-tos," and even the what-you-can'ts of this rare art.
  • Practical Integration of UNIX and Active Directory
    With Active Directory, Microsoft introduced an open LDAP directory that has become the de facto authentication store at many organizations. UNIX/Linux administrators are often tasked with the unthinkable: to integrate UNIX authentication with Active Directory. We'll not only explore the standard integration tools, such as OpenLDAP, PAM, and NSS, but will show you how to create custom scripts to manage Active Directory from UNIX.
  • Performance Crises Case Studies
    Don't miss the latest episode of this incredibly popular segment! We've taken a new set of real-life system administration performance crises and dissected them, providing insight on how to diagnose and remedy situations that you may someday face.
  • Custom Open Source Performance Monitoring
    Most organizations have monitoring systems that provide real-time problem alerts, but few can produce graphs of resource utilization over time. We provide practical examples of extending a monitoring system to collect historical performance trends. We'll use examples specific to Nagios and RRDtool, but the lessons and gotchas discussed here will prove useful to anyone looking to implement any new monitoring system.
Trent Hein (S6, M6) is co-founder of Applied Trust Engineering, a leader in holistic  Trent Hein infrastructure and security. Trent worked on the 4.4 BSD port to the MIPS architecture at Berkeley, is co-author of both the UNIX Systems Administration Handbook and the Linux Administration Handbook, and holds a B.S. in Computer Science from the University of Colorado.

Ned McClain (S6, M6), co-founder and CTO of Applied Trust Engineering, lectures around the globe Ned McClain on applying cutting-edge technology in production computing environments. Ned holds a B.S. in Computer Science from Cornell University and is a contributing author of both the UNIX Systems Administration Handbook and the Linux Administration Handbook.

S7 An Introduction to OpenAFS and Its Administration NEW!
Esther Filderman, Pittsburgh Supercomputing Center, and Alf Wachsmann, Stanford Linear Accelerator Center
9:00 a.m.–5:00 p.m.

Who should attend: Anyone looking to learn more about OpenAFS and how to set up and administer an OpenAFS cell.

AFS is a global distributed file system which works on many different operating systems (UNIX, Windows, Mac OS). It is ideal for sharing data and software in a heterogeneous distributed computing environment. Now that AFS has become available through an open source license, it is available to sites and IT groups of all sizes. Although the use of AFS is simple, setting up your own AFS servers can be a rather daunting task.

Topics include:

  • Overview of AFS concepts and semantics
  • Setting up and managing the AFS client (even without your own servers)
  • A working outline of the AFS server processes and how they play together
  • How to set up a new AFS cell: design decisions, initial setup, planning for the future
  • Authentication issues: Native KAS vs. Kerberos5
  • Backups: How and what to choose to use
  • AFS tools to make everything from maintenance to monitoring easier

Esther Filderman (S7) has been working with AFS since its infancy at CMU, before it Esther Filderman was called AFS, and is currently Senior Operations Specialist and AFS administrator for the Pittsburgh Supercomputing Center. She has been working to bring AFS content to LISA conferences since 1999. She is also coordinating documentation efforts for the OpenAFS project.

Alf Wachsmann (S7) works at the Stanford Linear Accelerator Center (SLAC) in Alf Wachsmann the Computing Services' High-Performance Computing Group, where he is an infrastructure designer and automation specialist. He has a doctor's degree in natural sciences obtained in computer science at the University of Paderborn (Germany). He worked as a post-doc in the computing center of DESY Zeuthen (Germany) before he came to SLAC in 1999.

S8 Network Security Profiles: Protocol Threats, Intrusion Classes, and How Hackers Find Exploits NEW!
Brad C. Johnson, SystemExperts Corporation
9:00 a.m.–5:00 p.m.

Who should attend: Administrators, managers, auditors, those being audited, those responsible for responding to intrusions or responsible for network resources that might be targets for crackers, hackers, or determined intruders.

Participants should understand the basics of TCP/IP networking. Examples will use actual tools and will include small amounts of HTML, JavaScript, and Tcl code and show command-line arguments and GUI-based applications.

This tutorial is focused on helping you understand how people profile your network to identify resources that might be vulnerable to attack. Simply put, the more information somebody can generate about your site (by profiling it), the more likely it is that they will be able to exploit something on it. This course will also help you recognize common protocol threats and intrusion classes.

Topics include:

  • Profiling your network and system
    • Methods and tools
    • An example of a profile
  • Intrusions
    • Awareness and statistics
    • Examples of intrusions
    • Common intrusion areas
      • Web servers
      • Web applications
      • Wireless infrastructure
      • Modems
  • Discovery/profiling tools
    • Tools: nmap, ntop, nessus, nikto, Satan/Saint/Sara, curl, dsniff, whisker, netstumbler, Websleuth
    • Understanding protocol tunneling
  • Protocol profiling threats
    • DNS
    • SNMP
    • Issues with handhelds
    • Web infrastructure

Brad C. Johnson (S8, M8) is vice president of SystemExperts Corporation. He has Brad C. Johnsonparticipated in seminal industry initiatives such as the Open Software Foundation, X/Open, and the IETF, and has been published in such journals as Digital Technical Journal, IEEE Computer Society Press, Information Security Magazine, Boston Business Journal, Mass High Tech Journal, ISSA Password Magazine, and Wall Street & Technology. Brad is a regular tutorial instructor and conference speaker on topics related to practical network security, penetration analysis, middleware, and distributed systems. He holds a B.A. in computer science from Rutgers University and an M.S. in applied management from Lesley University.

S9 Advanced Perl Programming NEW!
Tom Christiansen, Consultant
9:00 a.m.–5:00 p.m.

Who should attend: Anyone with a journeyman-level knowledge of Perl programming who wants to hone Perl skills. This class will cover a wide variety of advanced topics in Perl, including many insights and tricks for using these features effectively. After completing this class, attendees will have a much richer understanding of Perl and will be better able to make it part of their daily routine.

Topics include:

  • Symbol tables and typeglobs
    • Symbolic references
    • Useful typeglob tricks (aliasing)
  • Modules
    • Autoloading
    • Overriding built-ins
    • Mechanics of exporting
    • Function prototypes
  • References
    • Implications of reference counting
    • Using weak references for self-referential data structures
    • Autovivification
    • Data structure management, including serialization and persistence
    • Closures
  • Fancy object-oriented programming
    • Using closures and other peculiar referents as objects
    • Overloading of operators, literals, and more
    • Tied objects
  • Managing exceptions and warnings
    • When die and eval are too primitive for your taste
    • The use warnings pragma
    • Creating your own warnings classes for modules and objects
  • Regular expressions
    • Debugging regexes
    • qr// operator
    • Backtracking avoidance
    • Interpolation subtleties
    • Embedding code in regexes
  • Programming with multiple processes or threads
    • The thread model
    • The fork model
    • Shared memory controls
  • Unicode and I/O layers
    • Named Unicode characters
    • Accessing Unicode properties
    • Unicode combined characters
    • I/O layers for encoding translation
    • Upgrading legacy text files to Unicode
    • Unicode display tips
  • What's new in Perl lately
    • Switch statement
    • Defined-or operators
    • Pre-compiled modules
    • Dynamic handles
    • Virtual I/O through strings

Tom Christiansen (S9) has been involved with Perl since day zero of its initial public release Tom Christiansen in 1987. Author of several books on Perl, including The Perl Cookbook and Programming Perl from O'Reilly, Tom is also a major contributor to Perl's online documentation. He holds undergraduate degrees in computer science and Spanish and a Master's in computer science. He now lives in Boulder, Colorado.

?Need help? Use our Contacts page.

Last changed: 17 Aug. 2004 aw
Events Calendar