|
|
ypmake
A Tool for NIS
by Albert Chin-A-Young
Albert has been a sys admin for the past 5+ years. He recently
started a company to compile and configure freely available software.
He enjoys reading the comic books Akiko, Usagi Yojimbo,
and Dragonball.
A typical environment using NIS has one master where map edits and distribution occur, several NIS slaves to receive updates from the master and to reduce load on query requests to the master, and many NIS clients that do not receive map updates but query NIS slaves and the NIS master for information in the above-mentioned maps. As an environment grows, the number of slaves increases to distribute the load for the additional clients. This approach assumes one NIS domain where all hosts share the same map information. Such a setup is simple for centrally managed sites. However, sites with a decentralized environment where multiple domains exist do not fit well into the typical NIS master/slave/client model. In addition, as the number of hosts at a site grows, the scalability of NIS begins to break down. As the number of NIS slaves grows to meet the demand of the increase in clients, the NIS master requires more time to distribute map updates to those additional NIS slaves. Furthermore, an adequate number of NIS slaves must exist on each subnet to allow for failover should an NIS slave fail, or NIS clients must be hard-coded with a list of NIS slaves to bind to (a list which must always be kept up to date). Scalability Solutions Using ingenuity, administrators have come up with a few solutions to the scalability problem with NIS. Because NIS provides quick lookups for information in a map, it serves a very useful purpose. However, to address problems with large environments, several variations on the NIS master/slave/client model have been devised. Multiple Domains To anwer the problem of an increase in slaves causing slower updates from the master, the single NIS domain may be broken into multiple NIS domains. The host currently acting as the NIS master remains the NIS master but is in a separate domain. It no longer uses yppush to update its slaves, since it now has no slaves. Rather, it uses rcp, scp, rdist, or rsync to update the masters for the new domains that have been created. These new NIS domains, while separate from the old single NIS domain, continue to share the same map information as they receive the same map updates. The first complication with this is maintaining the top-level NIS master — the host distributing maps to the NIS masters in the new domains — as the master for the passwd map, so password changes are propagated up the food chain. This is easily accomplished with a kludge to the ypxfr program or by having slaves pull information. The second complication is the need to update the vendor-provided utility used to distribute maps, using a makefile or shell script. For the top-level master, flat files need to be copied to second-tier masters, and for second-tier masters (masters in the new domains), a yppush needs to occur. All Clients Are Slaves To address the problem of failover if not enough slaves exist on a subnet for clients to bind to, all NIS clients may be made NIS slaves. While this increases the number of hosts to distribute to, it lessens the dependency a host has on another, assuming hosts are now forced to bind to themselves. Each Host Is Its Own Master The NIS master/slave/client model may be thrown out the window in favor of many domains with each host acting as its own NIS master. This requires the development of home-grown tools but allows the hierarchy of the NIS environment to be defined in any way. Rather than three levels with the NIS master/slave/client, it is trivial to construct an environment with two, four, or five levels. The top-level master is responsible for distributing flat files to second-level masters, which then distributes flat files to third-level masters, and so on. This environment requires the most administrative overhead if it is not implemented correctly using appropriate tools. Minimizing Overhead With each of these variations comes management overhead. However, the appropriate tools can help minimize the overhead and make the new environment more manageable. The vendor-provided utility used to distribute maps to slaves is usually via a makefile or shell script. For the typical NIS master/slave/client model, this is sufficient. However, as the environment grows to one of the variations above or as the number of administrators increases, forcing the use of revision control, the vendor-provided tools break down. Standardization Project While working on a project to define standards for the UNIX environment at a client site, we decided that all hosts would be NIS masters, following an idea from the Aurora project at Morgan Stanley. This decision, a departure from the client's existing NIS environment, was designed to solve the problems with their existing NIS setup and to address the requirements of the new environment being created. At the client site, the UNIX environment was decentralized. Each community of users had a separate administrator or administrators dedicated to its group. This led to differences in how NIS was deployed and managed in each group. While some of the NIS maps were shared and distributed centrally from one server, most maps were maintained separately and distributed differently. The following problems existed with their NIS configuration:
/usr/etc/yp/Makefile
The process to update a file was:
% cd /usr/etc/yp
The makefile, provided by the OS vendor, would distribute maps from /usr/etc/yp. As RCS was employed, the flat file used to build the NIS maps from and distribute from should be the last version checked in to RCS. A New Tool As a result of the project to define new standards, a new tool was created to solve the existing NIS problems and address other common NIS problems. The tool also had to work in the new NIS environment, where all hosts would be NIS masters. Following are the requirements for the new tool, ypmake: The first attempt at meeting these requirements was by modifying an existing makefile and augmenting it with features from GNU make. As more complexity was added to the makefile to satisfy the requirements, it became obvious that a separate program was needed to manage the task. Coding ypmake in Perl provided some advantages: The end result was a tool, designed for a specific need, that worked well enough to be used with other NIS configurations. One of the main benefits of ypmake was centralized administration of all NIS maps for all NIS hosts, regardless of whether or not custom modifications needed to be made to a map. This was controlled through the configuration file, /etc/ypmake.conf. Because all hosts were NIS masters at the client site, a local customization was done by adding a second configuration file, /etc/ypmasters.conf, which defined the group a host belonged to and who its master was (i.e., which master updated its flat files). While ypmake does not need /etc/ypmasters.conf for a typical NIS master/slave/client configuration, the configuration file can be used with other variations on the NIS model given earlier. Before providing examples of how ypmake can be configured to work in the typical NIS master/slave/client model and with the alternate models described earlier, let us take a brief overview of the configuration file. An understanding of the configuration file is vital to understanding how modification of this file is all that is necessary to adapt ypmake to your NIS environment and to the configurations to be discussed later. Ypmake Configuration File The default location of the configuration file is /etc/ypmake.conf. This can be modified with the -c option. The configuration file is divided into two logical sections, variable assignments and instructions for building NIS maps. Variable assignments are of the form VAR = VALUE. Variables are used by ypmake and any of its modules. (This includes custom modules written for a site.) A simple configuration file looks like:
# Variables
# Where sendmail is located
# Max number of groups a user can be a member of
# Additional shells allowed in 'login-shell' field of password
# Build ethers map the same on all hosts
The configuration syntax looks similar to that used by the Berkeley AMD automounter. Variables alter defaults used in the program and provide information used when maps are built. By convention, variables with a :: are used by Perl modules. They are also used by site-specific modules. Ypmake contains no "special" knowledge about how to build a map. It is controlled entirely through the configuration file. The second section contains the instructions needed to build maps for all hosts. Each map entry in this section is called a location list, with each entry in the location list called a location selection. Location selections are split into selectors (var==value) and options (var:=value). Only one configuration file is needed for an environment regardless of how many domains exist. Should different actions need to be taken on different hosts, a host selector is available in the location list defining what action to take for the given host. The options in a location selection instruct ypmake on how a map should be built. If a selector is not present in a location selection, it is considered the default location selection. In the example above, all location selections are defaults, since no selectors exist. Briefly, the buildmap option instructs ypmake on how to construct the data for the NIS map that will be used to build the DBM files. The value flat indicates that ypmake should read a flat file, while rcs indicates ypmake should perform an RCS checkout of the last checked-in revision of the NIS map being built. A timestamp file is used to determine whether a map should be rebuilt. For flat files, the timestamp of this file is used. For rcs files, the timestamp file contains the last revision that was built. The builddbm option indicates which Perl module is used to build the DBM representation of the map, and the pushmap option indicates how the map should be distributed. Ypmake with the Typical NIS Master/Slave/Client Model In this environment, ypmake runs only on the NIS master. The NIS slave receives DBM map updates via yppush on the NIS master. The NIS client receives no updates. A sample section of the configuration file for such an environment would look like:
group buildmap:=flat;builddbm:=group;p
ushmap:=yppush
Using the snippet above, the instructions for the group map indicate that flat files should be used as the source and the group Perl module should be used to build the DBM files. The file-repository variable specifies the location of flat files. A value of rcs would cause ypmake to "check out" from RCS the last checked-in revision of the group map and use that as the source for generating the DBM files. The rcs-repository variable specifies the location of the RCS repository. Except for the passwd map, the remaining maps are handled in a similar fashion. The passwd map is treated differently because rpc.yppasswdd modifies the NIS passwd file. Because of this, a special buildmap handler was created to perform file locking on the passwd file prior to using it for generating the DBM map. This ensures synchronization between ypmake and rpc.yppasswdd. Ypmake with the Multiple NIS Domains Sharing the Same Map Information In this environment, ypmake runs on all NIS masters. To properly keep the masters in sync, it makes sense to employ locking during the distribution of a map. This ensures that while the DBM representation of a map is being built, the source is not being modified during the build. In addition to control of map distribution by the master, it might also be desirable to provide a mechanism whereby a specific host or hosts in certain domains can be updated quickly. It would be undesirable to distribute maps from the master only to have to wait for a cron job to kick off to update the individual NIS domains. Ypmake provides a solution to this problem.
A sample section of the configuration file for this environment would
look like:
Because one of the NIS masters in this environment will be canonical for the NIS maps, ypmake will behave differently on this host from the other NIS masters. On this host, mapmaster, ypmake will need to transfer the flat files used to build the DBM files on the remaining NIS masters. It does this with the pushmap:=rdist option. In the previous example, we used yppush to distribute maps to slaves. As the top-level NIS master will distribute to other NIS masters, yppush cannot be used. The default location selection, applicable for all hosts except mapmaster, uses yppush to distribute maps to the remaining hosts, which are slaves. To enable synchronization of map updates between mapmaster and the other NIS masters, rdist uses the procmail lockfile program. An rdist special is used to copy the new flat file to /tmp on the remote server, lock the map in the build directory, copy the new flat file to the build directory, and then unlock the map. The rdist module uses the file /etc/ypmasters.conf to determine which hosts to push to. The format of this file is: [host]:[master]:[domain] [host] is the name of the host running ypmake, [master] the name of the host that distributes to it, and [domain] is a logical name assigned to groups of machines. While it could be the NIS domain name of [host], it need not be. For the example above, the contents would be:
master1:mapmaster:domain1
NIS slaves do not appear in this file, since the NIS ypservers map controls which hosts are distributed to via yppush. NIS clients are also absent, as they do not receive updates. Only hosts distributed to via rdist appear in this file. The [domain] portion is used as an argument to the -A command-line option. However, if ypmake is to provide the ability to update any host (via the -l and -A or -m switches) in the environment, NIS masters must appear in /etc/ypmasters.conf. For an update to propagate to a client or slave, ypmake must first run on the host mapmaster, then on the NIS master serving the domain of the client or slave. With the use of the -A and -m switch and /etc/ypmasters.conf, ypmake can achieve both of these steps using the following invocation of ypmake and the sample /etc/ypmasters.conf file: % ypmake -l -A domain1 group
(/etc/ypmasters.conf)
This causes ypmake to perform an NIS update to all hosts in the NIS domain domain1. Updates cannot be propagated directly to a slave or client because of the use of yppush as the distribution mechanism, and because the hostnames of slaves and clients are not listed in /etc/ypmasters.conf. Ypmake accomplishes the update of the NIS domain domain1 by searching /etc/ypmasters.conf for the domain domain1 to determine who the NIS master is. Ypmake then performs a local build on mapmaster (local builds do not update the timestamp file), rdists the group file to master1, then performs an ssh onto master1 and reruns ypmake -l -A domain1 group. This second invocation of ypmake will perform a yppush forcing the update of the all slaves and clients in domain1. On the master host mapmaster, the source used to build the DBM maps is under RCS control (buildmap:=rcs). Because of this, the most recent "checked in" version of the map will be used as the source. If someone is editing the group file, the flat file being edited will not be used as the source. Rather, the version in RCS will be used. This eliminates the problem of distributing partially edited maps. Ypmake with All Hosts as NIS Masters In this environment, ypmake runs on all hosts, since all hosts are NIS masters. While this allows fine-grained control over how NIS is architected, it does lead to administrative overhead. Nevertheless, it's cool. Also, ypmake was originally designed to work in such an environment (though examples have already been given describing how ypmake can work with other NIS configurations). The following map of this NIS configuration will be used in the discussion to follow:
master
The /etc/ypmasters.conf file looks like:
master:master:domain0
All hosts are listed in /etc/ypmasters.conf because all are NIS masters and because the rdist pushmap module uses this list for hosts to distribute to. Rdist is used rather than yppush because no slaves exist in this environment. Because each host is a master, for a NIS update to propagate to all hosts, the host master runs ypmake, which invokes an rdist of the flat files to master1 and master2. The hosts master1 and master2 then perform a ypmake at a time scheduled in cron to update master1a, master1b, master2a, and master2b. Finally, these four hosts invoke a ypmake from cron to update their NIS maps. Because updates occur automatically via cron, this sequence of steps is automated in the background. At this number of layers, manually updating a host becomes tedious. Because of this, if a map change is needed to propagate to all hosts in domain2, the following would need to occur:
master% ypmake
The -l or -A and -m options, as utilized in the previous section, are also used here to force propagations to a set of machines in a domain or to any number of machines. We go into more detail here on what ypmake is doing to accomplish this. With these options, the following one-line invocation of ypmake will accomplish the same as the previous five manual entries:
master% ypmake -l -A domain2 group # Update group map
for
master% ypmake -l -m master2a group # Update group map
for
The -l option causes a local update to occur. It is used only if the host running ypmake needs to be updated. If the host is a master that is responsible for distribution to other hosts in its domain, the -l option negates the push. Coupled with the -A <domain> and -m <host> options, this causes ypmake to reexecute itself on the masters of hosts that distribute to hosts in <domain> or to <host>. The complete execution tree for the above two lines follows. While only the first line is entered manually, the remaining are run "behind the scenes" by ypmake.
master% ypmake -l -A domain2 group
master% ypmake -l -m master2a
This stream of ssh invocations occurs regardless of the number of tiers in the NIS hierarchy. Ypmake uses the /etc/ypmasters.conf file to determine the hosts to distribute to. Regardless of how deep the number of levels in your NIS hierarchy, it determines the hosts it can distribute to and trusts these hosts to do the same until, eventually, the destination hosts are reached. As indicated in the expanded list, ypmake is run on all hosts. A separate version of ypmake is not needed depending on the role the hosts plays. Note that when distributing to host master2a, the host master performed an ssh to master2, since it was the master for master2a. This information is contained in the configuration file /etc/ypmasters.conf. Moreover, ypmake, from reading this configuration file, knew that master could not distribute to master2a directly. However, as master2 could, and master could distribute to master2, ypmake performed an ssh to master2 to accomplish the job. Ypmake Execution Trail The general output of ypmake will look like the following:
Creating lockfile for [map] map ... done
The verbosity of the output indicates the progress made as ypmake builds a map. Errors are displayed during the run. Where possible, a log file is created giving details of any errors that occur. Ypmake creates a lockfile via procmail's lockfile or fcntl() for the map it is working on. Therefore, while many ypmake processes can be running at the same time, more than one ypmake process cannot build the same map. This locking is also synchronized when maps (i.e., flat files) are being distributed. In this cause, procmail's lockfile must be used because the rdist special-executed after the map is transferred performs a lock on the remote file repository before transferring the new file from the temporary location to the repository. Following the lock, a build of the map begins. The creating [map] output indicates that the flat-file representation of the map was built. The flat-file representation is created by the buildmap option in a location selection. Values currently supported are flat, passwd, and rcs. Additional values can be added via dynamically loadable Perl modules. Thus, flat files could be extracted from an Oracle database if a BuildMap::Oracle Perl module were created and the buildmap option was specified as buildmap:=oracle. Flat files are created in a temporary directory under /tmp/ypmake/[pid] (where ypmake does all its work). Outside of the main ypmake code, almost all Perl code is dynamically loaded. This is the case for bundled modules and site-specific modules. A custom buildmap module is needed for the password map because it needs to be locked to disable updates via rpc.yppasswdd. The checkmap option specifies syntax checking. The value for the checkmap option contains the Perl module used to perform the syntax check of the map. As with the buildmap option, custom syntax checkers can be added via Perl modules. To augment the list included with ypmake, additional values can be added to the checkmap option, separated by commas. Thus, to perform the ypmake group check and a site-specific group check, the checkmap option would appear as checkmap:=group,site-group. The building [map] DBM files output indicates that DBM representations of flat files are being built. Ypmake goes to great lengths to ensure that sufficient disk space exists before it replaces the existing DBM files. It replaces the current files as follows: 1. Build DBM maps in /tmp/ypmake/[pid] 2. Move to /var/yp/[domain]/[map].tmp 3. Backup existing NIS DBM maps to /var/yp/[domain]/[map].orig 4. Rename /var/yp/[domain]/[map].tmp to /var/yp/[domain]/[map] 5. Remove /var/yp/[domain]/[map].orig Until step 3, ypmake can deal with error recovery properly. If the process succeeds with step 2, sufficient disk space exists to replace the map. The updating local NIS [map] map output is for hosts that need a ypxfr -f -h localhost [map] to use the new map information immediately. Digital UNIX is an example of such a host. Nevertheless, it is run on all hosts. Signal Handling Ypmake catches the HUP, INT, TERM, and QUIT signals. At startup, it registers a subroutine to handle each of these signals. During program execution, cleanup subroutines are registered during specific areas of the code to deal with backout conditions should an interrupt occur. As an example, when the passwd buildmap module is executed, it registers a function to unlock the password file should an interrupt occur. The same signal-handling routines also remove temporary files created during execution. Future Directions Ypmake only supports NIS. Support for NIS+ would be a benefit. Distributing a map to a host without building it is done with the -A or -m option. However, distributing the flat file does not follow the mechanism of forcing updates on clients by trickling down until the ssh/ypmake pair arrives at the final host with the -l option. Instead, ypmake copies the flat file directly to the master.
Support for rsync as a distribution mechanism would allow
quicker updates (difficult, though, since it doesn't support the
rdist "special" operation). If an API for rsync were
available, such support would be possible.
|
|
 Need help? Use our Contacts page.
Last changed: 22 Nov. 1999 mc |
|
Issue index