Managing Network Security with cfengine, Part 2

by Mark Burgess
<Mark.Burgess@iu.hioslo.no>

Mark is associate professor at Oslo College and is the author of cfengine and winner of the best paper award at LISA 1998.


Also, see Part 1 (August issue) and Part 3 (December issue) of this article.

You can read more about cfengine and obtain it from <http://www.iu.hioslo.no/cfengine>. (Please note that the following Web sites are not associated with cfengine, and readers are advised to steer clear of them: www.cfengine.com, www.cfengine.org, www.cfengine.net)

The point of cfengine is normally to have only one global configuration for every host. This needs to be distributed somehow, which means that hosts must collect this file from a remote server. This in turn means that you must trust the host, which has the master copy of the cfengine configuration file.

The beginning of security is correct host configuration. Even if you have a firewall shielding you from outside intrusion, an incorrectly configured host is a security risk. Host configuration is what cfengine is about, so we could easily write a book on this. Rather than reiterating the extensive documentation, let's just consider a few examples that address actual problems.

A cfengine configuration file is composed of objects with the following syntax (see the cfengine documentation):

rule-type:
 classes-of-host-this-applies-to::
  Actual rule 1
  Actual rule 2 ...

The rule-types include checking file permissions, editing text files, disabling (renaming and removing permissions to) files, controlled execution of scripts, and a variety of other things relating to host configuration. Some of the "control" rules are simply flags that switch on complex ("smart") behavior. Every cfengine program needs an actionsequence that tells it the order in which bulk configuration operations should be evaluated. For example:

control:
 actionsequence = ( netconfig copy processes editfiles )

Now, using Solaris and GNU/Linux as example operating systems, I'll step through some basic idioms that can repeated in different contexts.

Disabling and Replacing Software

One of the simplest things we are constantly asked to do is to disable dangerous programs as bugs are discovered. CERT security warnings frequently warn about programs with flaws that can compromise a system. In cfengine, disabling a file means renaming it to *.cf-disabled and setting its permission to 400.

disable:

 #
 # CERT security patches
 #

 solaris::

  /usr/openwin/bin/kcms_calibrate
  /usr/openwin/bin/kcms_configure
  /usr/bin/admintool
  /etc/rc2.d/S99dtlogin
  /usr/lib/expreserve

 linux::

  /sbin/dip-3.3.7n
  /etc/sudoers
  /usr/bin/sudoers

Although this is a trivial matter, the fact that it is automated means that cfengine is checking for this all the time. As long as a host is up and running (connected to the network or not), cfengine will be ensuring that the named file is not present.

Another issue is replacing standard vendor programs with drop-in replacements. For example, most sysadmins like to replace vendor sendmail with the latest update from Eric Allman's site. One way to do this is to compile the new sendmail into a special directory, separate from vendor files, and then to symbolically link the new program into place.

links:

 solaris||linux::

  /usr/lib/sendmail ->! /usr/local/lib/mail/bin/sendmail-8.9.3
  /usr/sbin/sendmail ->! /usr/local/lib/mail/bin/sendmail-8.9.3
  /etc/mail/sendmail.cf ->! /usr/local/lib/mail/etc/sendmail.cf

The exclamation marks mean (by analogy with the csh) that existing file objects should be replaced by links to the named files. Again, the integrity of these links is tested every time cfengine runs. If the object /usr/lib/sendmail is not a link to the named file, the old file is moved and a link is made. If the link is okay, nothing happens. After putting the new sendmail in place, you will need to make sure that the restricted shell configuration is in order.

#
# Sendmail, restricted shell needs these links
#
solaris::

 # Most of these will only be run on the MailHost
 # but flist (procmail) is run during sending...
 /usr/adm/sm.bin/vacation -> /usr/ucb/vacation
 /usr/adm/sm.bin/flist -> /home/listmgr/.bin/flist

linux::

 /usr/adm/sm.bin/vacation -> /usr/bin/vacation

Link management is a particularly useful feature of cfengine. By putting links (actually all system modifications) into the cfengine configuration and never doing anything by hand, you build up a system that is robust to reinstallation. If you lose your host, you just have to run cfengine once or twice to reconstruct it.

Of course, the fundamental tenet of security is to be able to restrict privilege to resources. We therefore need to check the permissions on files. For instance, a recent CERT advisory warned of problems with some free UNIX mount commands that were setuid root. If we suppose there is a group of hosts called "securehosts" that we don't need to worry about, then we could remove the setuid bits on all other hosts as follows:

files:

 !securehosts.linux::

  /bin/mount mode=555 owner=root action=fixall
  /bin/umount mode=555 owner=root action=fixall

 securehosts.linux::

  /bin/mount m=6555 o=root action=fixall
  /bin/umount m=6555 o=root action=fixall

One area where cfengine excels over other tools is its ASCII-file-editing abilities. Editing text files in a nondestructive way is such an important operation that once you've used it you will wonder how you ever managed without it! Here are some simple but real examples of how file editing can be used.

editfiles:

 # sun4, who are they kidding?
 { /etc/hosts.equiv
 HashCommentLinesContaining "+"
 }
#
# CERT security patch for vold vulnerability
#

sunos_5_4::

 { /etc/rmmount.conf
 HashCommentLinesContaining "action cdrom"
 HashCommentLinesContaining "action floppy"
 }

TCP wrapper configuration can be managed easily by maintaining a pair of master files on a trusted host. Files of the form

# /etc/hosts.allow (exceptions)
#
# Public services

sendmail: ALL
in.ftpd: ALL
sshd: ALL

# Private services

in.fingerd: .mydomain.country LOCAL
in.cfingerd: .mydomain.country LOCAL
cfd: .mydomain.country LOCAL
sshdfwd-X11: .mydomain.country LOCAL

# Portmapper has to use IP series

portmap: 128.39.89. 128.39.74. 128.39.75.

and

# /etc/hosts.deny (default)

ALL: ALL

may be distributed to each host by cfengine:

copy:

/masterfiles/hosts.deny dest=/etc/hosts.deny
   mode=644
   server=trusted
/masterfiles/hosts.allow dest=/etc/hosts.allow
   mode=644
   server=trusted

and installed as follows:

editfiles:

 { /etc/inet/inetd.conf

 # Make sure we're using tcp wrappers

 ReplaceAll "/usr/sbin/in.ftpd" With "/local/sbin/tcpd"
 ReplaceAll "/usr/sbin/in.telnetd" With "/local/sbin/tcpd"
 ReplaceAll "/usr/sbin/in.rshd" With "/local/sbin/tcpd"
 ReplaceAll "/usr/sbin/in.rlogind" With "/local/sbin/tcpd"

processes:
 "inetd" signal=hup

The services that we do not need should be removed altogether; there's no sense in tempting fate:

editfiles:

 { /etc/inetd.conf

 # Eliminate unwanted services

 HashCommentLinesContaining "rwall"
 HashCommentLinesContaining "/usr/sbin/in.fingerd"
 HashCommentLinesContaining "comsat"
 HashCommentLinesContaining "exec"
 HashCommentLinesContaining "talk"
 HashCommentLinesContaining "echo"
 HashCommentLinesContaining "discard"
 HashCommentLinesContaining "charge"
 HashCommentLinesContaining "quotas"
 HashCommentLinesContaining "users"
 HashCommentLinesContaining "spray"
 HashCommentLinesContaining "sadmin"
 HashCommentLinesContaining "rstat"
 HashCommentLinesContaining "kcms"
 HashCommentLinesContaining "comsat"
 HashCommentLinesContaining "xaudio"
 HashCommentLinesContaining "uucp"
 }

Process Monitoring

When it comes to process management, we are usually interested in three things:

• making sure certain processes are running

• making sure some processes are not running

• sending HUP signals to force configuration updates

To HUP a daemon and make sure that it is running, we write:

processes:

 linux::

  "inetd" signal=hup restart "/usr/sbin/inetd" useshell=false
  "xntp" restart "/local/sbin/xntpd" useshell=false

The useshell option tells cfengine that it should not use a shell to start the program. The idea here is to protect against IFS attacks. Unfortunately, some programs require a shell in order to be started, but most do not. This is an extra precaution. When the cron daemon crashes, restarting it can be a problem since it does not close its filed descriptors properly when forking. The dumb option helps here:

"cron" matches=>1 restart "/etc/init.d/cron start" useshell=dumb

To kill processes that should not be running, we write:

processes:

 solaris::

 #
 # Don't want CDE stuff or SNMP peepholes...
 #

 "ttdbserverd" signal=kill
 "snmpd" signal=kill
 "mibiisa" signal=kill

A couple of years ago, a broken cracked account was revealed at Oslo College by the following test in the cfengine configuration:

processes:

 # Ping attack ?

 "ping" signal=kill inform=true

There are few legitimate reasons to run the ping command more than a few times. The chance of cfengine detecting single pings is quite small. But coordinated ping attacks are another story. When it was revealed that a user had twenty ping processes attempting to send large ping packets to hosts in the United States, it was obvious that the account had been compromised. Fortunately for the recipient, the ping command was incorrectly phrased and would probably not have been noticed.

processes:

 "sshd"
   restart "/local/sbin/sshd"
   useshell=false

 "snmp" signal=kill
 "mibiisa" signal=kill

 "named" matches=>1
   restart "/local/bind/bin/named"
   useshell=false

 # Do the network community a service and run this

 "identd" restart "/local/sbin/identd" inform=true

Process management also includes garbage collection, which we shall turn to later.

Monitoring Files

Almost all security programs available monitor file integrity. Cfengine also incorporates tools for monitoring files. Here are some of the elements in the fairly complex files command:

files:

 classes::

 /file-object
   mode=mode
   owner=uid-list
   group=gid-list
   action=fixall/warnall..
   ignore=pattern
   include=pattern
   exclude=pattern
   checksum=md5
   syslog=true/on/false/off

In addition to these, there are extra flags for BSD filesystems and ways of managing file ACLs for systems like NT. Here are some examples of basic checks on file permissions:

classes:

 have_shadow = ( `/bin/test -f /etc/shadow' )
 NFSservers = ( server1 server2 )

files:

 any::
  /etc/passwd mode=0644 o=root g=other action=fixplain

 have_shadow::
  /etc/shadow mode=0400 o=root g=other action=fixplain

 # Takes a while so do this at midnight and only on servers

 NFSservers.Hr00::

  /usr/local
   mode=-0002 # Check no files are writeable!
   recurse=inf
   owner=root,bin
   group=0,1,2,3,4,5,6,7,staff
   action=fixall

In the last example we parse through a whole file system (recurse=inf) and as a result get a number of checks for free. Any previously unknown setuid programs are reported, as well as any suspicious filenames. Let's elaborate.

The Setuid Log

Cfengine is always on the lookout for files that are setuid or setgid root. It doesn't go actively looking for them uninvited, but whenever you get cfengine to check a file or directory with the files feature, it will make a note of setuid programs it finds there. These are recorded in the file cfengine.host.log, which is stored under /etc/cfengine or /var/log/cfengine. When new setuid programs are discovered, a warning is printed, but only if you are root. If you ever want a complete list, delete the log file, and cfengine will think that all of the setuid programs it finds are new. The log file is not readable by normal users.

Suspicious Filenames

Whenever cfengine opens a directory and scans through files and directories (recursively: files, tidy, copy), it is also on the lookout for suspicious filenames, for example files like ".. ." containing only space and/or dots. Such files are seldom created by sensible sources, but are often used by crackers to try to hide dangerous programs. Cfengine warns about such files. Although not necessarily a security issue, cfengine can also warn about filenames that contain nonprintable characters and directories that are made to look like plain files by giving them filename extensions:

control:

 #
 # Security checks
 #

 NonAlphaNumFiles = ( on )
 FileExtensions = ( o a c gif jpg html ) # etc
 SuspiciousNames = ( .mo lrk3 lkr3 )

The file-extension list may be used to detect concealed directories during these searches and warn if users create directories that look like common files. Additional suspicious filenames can be checked for automatically.

The mail spool directory is a common place for users to try to hide downloaded files. These options inform about files that do not have the name of a user or are not owned by a valid user:

control:

 WarnNonOwnerMail = ( true )
 WarnNonUserMail = ( true ) # Warn about mail which is not owned by a user

Corresponding commands exist to delete these files without further ado. This can be a useful way of cleaning up after users whose accounts have been removed.

Checksums and Tripwire Functionality

Cfengine can be used to check for changes in files that only something as exacting as an MD5 checksum/digest can detect. If you specify a checksum database and activate checksum verification,

control:

 ChecksumDatabase = ( /etc/cfengine/cache.db )
 ChecksumUpdates = ( false )

files:
 /filename checksum=md5 ....
 /dirname checksum=md5 recurse=inf....

 # If the database isn't secure, nothing is secure...

 /etc/cfengine/cache.db mode=600 owner=root fixall

then cfengine will build a database of file checksums and warn you when files' checksums change. This makes cfengine act like Tripwire (currently only with MD5 checksums). It can be used to show up Trojan-horse versions of programs. It should be used sparingly, though, since database management and MD5 checksum computation are resource-intensive operations that could add significant time to a cfengine run. The ChecksumUpdates variable (normally false) can be set to true to update the checksum database when programs change for valid reasons.

Warnings are all well and good, but the spirit of cfengine is not to bother us with warnings, but rather to fix things automatically. If you are worried about the integrity of the system, then don't just warn about checksum mismatches here; make an md5 copy comparison against a read-only medium that has a correct, trusted version of the file on it. That way if a binary is compromised you will not only warn about it but also repair the damage immediately!

The control variable ChecksumUpdates may be switched to on in order to force cfengine to update its checksum database after warning of a change.

FileExtensions

This list may be used to define a number of extensions that are regarded as plain files by the system. As part of general security checking, cfengine will warn about any directories having names that use these extensions (possibly in order to conceal them):

FileExtensions = ( c o gif jpg html )

NonAlphaNumFiles

If enabled, this option causes cfengine to detect and disable files that have purely non-
alphanumeric filenames, that is, files that might be accidentally or deliberately concealed. The files are then marked with a suffix .cf-nonalpha and are rendered visible:

NonAlphaNumFiles = ( on )

These files can then be tidied (deleted) or disabled by searching for the suffix pattern. Note that nonalphanumeric means ASCII codes lower than 32 and higher than 126.

Defensive Garbage Collection

We tend to be worried about the fact that crackers will destroy our systems and make them unusable, but many operating systems are programmed to do this to themselves! Few systems can survive a full system disk, yet many logging agents go on filling up disks without ever checking to see how full they are getting. In short, they choke themselves in a self-styled denial-of-service attack. Cfengine can help here by rotating logs frequently and by tidying temporary file directories:

disable:

 Tuesday.Hr00::

 #
 # Disabling these log files weekly prevents them from
 # growing so enormous that they fill the disk!
 #

/local/iu/httpd/logs/access_log rotate=2
/local/iu/httpd/logs/agent_log rotate=2
/local/iu/httpd/logs/error_log rotate=2
/local/iu/httpd/logs/referer_log rotate=2

FTPserver.Sunday::

/local/iu/logs/xferlog rotate=3

tidy:

/tmp pattern=* age=1

Process garbage collection is just as important. There are lots of reasons why process tables fill up with unterminated processes. One example is faulty X terminal software that does not kill its children at logout. Another is that programs like Netscape and pine tend to go into loops from which they never return, gradually loading the system with an ever-increasing glacial burden. If the host concerned has important duties, this lack of responsiveness can compromise key services. It also gives local users a way of carrying out denial-of-service attacks on the system. Just killing old processes can cause your system to spring back from its ice-age blues (hopefully without littering the system with too many dead mammoths or bronze-age ax bearers). If the host concerned has important duties, then this lack of responsiveness can compromise key services. It also gives local users a way of carrying out denial-of-service attacks on the system.

If users always log out at the end of the day and log in again the next day, then this is easy to address with cfengine. Here is some code to kill commonly hanging processes. Note that on BSD-like systems process options "aux" are required to see the relevant processes:

processes:

linux|freebsd|sun4::

SetOptionString "aux"

any::

"Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec"

signal=kill

include=ftpd
include=tcsh
include=xterm
include=netscape
include=ftp
include=pine
include=perl
include=irc
include=java
include=/bin/ls
include=emacs
include=passwd

This pattern works like this: As processes become more than a day old, the name of the month appears in the date of the process start time. These are matched by the regular expression. The include lines then filter the list of the processes further, picking out lines that include the specified strings. On some BSD-like systems the default ps option string is "-ax" and you might need to reset it to something that adds the start date in order to make this work.

Another job for process management is to clean up processes that have hung, gone amok, or are left over from old logins. Here is a regular expression that detects non-root processes that have clocked more than 100 hours of CPU time. This is a depressingly common phenomenon when a program goes into an infinite loop. It can starve other processes of resources in a very efficient denial-of-service attack.

any::

#
# Kill processes which have run on for too long e.g., 999:99 cpu  # time
# Careful a pattern to match 99:99 will kill everything!
#

"[0-9][0-9][0-9][0-9]:[0-9][0-9]" signal=term exclude=root
"[0-9][0-9][0-9]:[0-9][0-9]" signal=term exclude=root

Under NT this is not so simple, since the process table for the cygwin library applies only to processes that have been started by programs working under the UNIX process emulation. Hopefully this shortcoming can be worked around at some point in the future.

The next and final thrilling installment discusses how to secure a Web and ftp site and details some of the ways that cfengine can protect you from attacks.