################################################ # # # ## ## ###### ####### ## ## ## ## ## # # ## ## ## ## ## ### ## ## ## ## # # ## ## ## ## #### ## ## ## ## # # ## ## ###### ###### ## ## ## ## ### # # ## ## ## ## ## #### ## ## ## # # ## ## ## ## ## ## ### ## ## ## # # ####### ###### ####### ## ## ## ## ## # # # ################################################ The following paper was originally presented at the Ninth System Administration Conference (LISA '95) Monterey, California, September 18-22, 1995 It was published by USENIX Association in the Conference Proceedings of the Ninth System Administration Conference For more information about USENIX Association contact: 1. Phone: 510 528-8649 2. FAX: 510 548-5738 3. Email: office@usenix.org 4. WWW URL: https://www.usenix.org ^L Patch Control Mechanism for Large Scale Software Atsushi Futakata - Central Research Institute of Electric Power Industry (CRIEPI) ABSTRACT Applying patches to large scale software is often difficult because unofficial patches and user modifications conflict with any ``official'' patches. Version control systems such as RCS[1], CVS[2], and configuration management[3,4,5] are useful solutions for this problem when the baseline of the software is fixed. However, an official patch that is developed externally changes the baseline and any local changes based on this become obsolete. Thus we must re-apply various unofficial patches and modifications, identify the causes of conflict, change or remove patches, and repeat the patch and unpatch operations. This paper presents a mechanism for (1) managing versions of a software package based on patches, (2) automating the application of unofficial patches and modifications by the user, and (3) rebuilding the package using file versions instead of timestamps. Using this mechanism, it becomes easy to apply patches and re-build software. Introduction We have spent a lot of time installing and patching large scale software packages such as the X11 Window System, TeX, etc. Installation of new software involves checking storage space, reading documentation and setting various configuration files correctly. This can be a non-trivial task even if the platform is officially supported. If the platform is not supported, installation becomes more complicated because changes to the source code may be required and tools such as Configure are not applicable. Thus software porting systems represented by the FreeBSD ports system[6] appear and become to support the installation task. Applying patches poses another difficult problem: If only official patches are applied to an officially supported platform, the task is usually easy because the patches are well managed and cause no conflict. However an unsupported platform requires source code changes which often conflict with an official patch. Furthermore, the user may require many useful, unofficial patches. These may be patches for emergency security, localization (e.g., japanization), machine/OS-dependencies or various extensions, such as Tcl/Tk has. Those patches may also conflict with official ones. The reason for the conflict is a lack of version management facilities for distributed development. This conflict usually necessitates the following operation: + Remove all unofficial patches and apply the official one, + Re-apply the unofficial patches and user modifications. If reject files are generated, the unofficial patch must be fixed or removed, + Rebuild the software. This can take a long time because the above operations may cause unnecessary changes to timestamps. Configuration management systems such as Aegis[7], CMS/MMS[8] are useful for version control and building software for multi-user development. They target the continuous development of the software and manage products based on a current baseline, that is a reference version of software on which each member of developing team fixes bugs and develops new functions. After each task is complete, all modifications are integrated and the modified source code becomes a new baseline for succeeding development. This baseline approach is useful for inhouse development teams. However, an official patch is delivered outside of a user's control and it only changes the baseline. All modifications based on the previous version of the software then become obsolete. Thus if the user wants to apply a new official patch, all other patches and modifications must be rearranged and re-applied after the official one is applied. In future, self-adaptive software agents or automatic programming from very high level specifications may solve the problem but, for the present, we have no silver bullet. Thus, in order to solve the above problem and support patch application, this paper proposes a patch control mechanism which has the following features: + version management of the whole package, including individual files and patches, + management and control of patch application order, + assistance with patch/unpatch operations and patch modification, + software rebuilding according to an individual file version rather than a timestamp. Classification of patches Unofficial patches may be generated by different people based on different baselines. Unified management of these patches can be difficult and confusing. In this paper, we classify patches into the following three types and treat each differently. official patch A patch that is authorized by and distributed from the software developer/maintainer. The latest official patch number is the official software version and we call it the patchlevel. unofficial patch A patch/extension that is widely distributed but is not an official patch. An unofficial patch may be applied in various directories with various patch(1) options. modification A change made by the local user, which includes editing files, fixing bugs, changing configuration files, etc. System Overview This section presents an overview of the system which is an implementation of the patch control mechanism. This includes; (1) management of the three types of patch, (2) control of patch application, and (3) rebuilding of the software. The components VM (Version Manager), PM (Patch Manager), and BM (Build Manager) implement the three functions respectively. Figure 1 shows the components and the relation among them. [figure 1 is not available] Figure 1: The components of this system The VM manages information in the VDB (Version Database) and the version tree, which records information about version control for updating and rebuilding of the software. The VDB records the location of the patches and the versions of individual files. The version tree records the application order for unofficial patches and modifications at each patchlevel. When a new patch arrives, the user adds the patch to the VDB using the VM. If the user wants to apply this, and the result is successful, the VM registers the sequence of patches actually applied, to the version tree. The PM (Patch Manager) controls the patch/unpatch operations and the building of the software according to the version tree. In this system, all operations, including editing a patch file, applying a patch, and building a package, are performed via the PM, and the result of the operation is reflected in the VDB and the version tree. When a user applies a new official patch, the PM tries to apply unofficial patches which were applied to the last version of software. If one of the patches is rejected, the PM notifies the user. The user may then remove or edit this patch and continue the job. After the job finished, the PM returns the result of the patch application and the VM revises the VDB and the version tree. The BM (Building Manager) is an extended make command, invoked from the PM to build a target according to the version of file instead of its timestamp. Because a new official patch forces patch/unpatch operations of unofficial patches and modifications, the timestamp of a file may change even if the contents is not altered. The VM registers the version of the newly generated target with the VDB and the version tree. This system manages several packages at once by referring to the pcm file. An entry of the pcm file has the following form: application_name:top_directory:patch_option Application_name is an identifier to be used for selecting a package. Top_directory is the directory where official patches are applied. Patch_option species an option to the patch(1) command. For example, the entry for X11R5 (X11 Window System, Version 11, Release 5) becomes*: [[FOOTNOTE: Because of the limitation of line width, we denote the location of the X11R5 package as /X11R5 in the following examples. The actually location is /staff/src/X11R5 in our site. ]] X11R5:/X11R5:-p -s This means that the following command is needed to apply the patch. % cd /X11R5 % patch -p -s ...... Figure 6: A head of e-mail which notifies the rejection After receiving the e-mail, the user can choose from the following three actions: + replace: replaces a part of patch/modification by this file, + delete: deletes this patch from the version tree entry, + abort: aborts the job, recovers the software and terminates the PM. If the user chooses replace, the user must reply the e-mail like Figure 7 to the PM after editing the file in the original e-mail. The first line of the mail body is the action that the user chosen, and the rest of the body is the file to be replaced. If the user chooses delete or abort, only the action is needed in the body. After receiving all replies from the user, the PM extracts a new patch from the replies by the same manner described in the explanation of .unofficial, and applies the new patch instead of the original patch and revises the version of the patch. To: pcm@denken.or.jp Subject: RE: REJECTION $1.1: sun.cf From: futakata@denken.or.jp replace ...... Figure 7: A head of reply mail with the action replace Make Command The PM often causes needless changes to timestamps and confuses timestamp-based traditional make(1) commands. Thus we developed the BM, which is based on the GNU make command, to perform actions according to the historical file version, i.e., when the historical versions of source files are not included in the version of the object, the BM rebuilds the object. After that, the BM revises the version of object by the merged version of sources because the object is made under the effect of all patches applied to the sources. To compare versions, the BM treats the version as an ordered sequence of patches. Thus a version A includes a version B if and only if all patches in the B is appeared in the A, and an unified version of A and B does not causes inconsistency of the application order. For example, the version #1:$1.1,$2.1,$3.1 does not include the #1:$3.1,$1.1. After building objects, the BM sends the result to the VM and the VM updates the historical versions of the objects. Experiment We have tested this system for applying patches to the X11R5. In this experiment, the number of official patches is 26 and the number of unofficial ones is 12 including internationalization patches for libraries Xaw, Xsi and Xwchar. We also modify configuration files in the /X11R5/mit /config directory and several files in /X11R5/mit/lib/Xt. Figure 8 is a head part of the version tree of this experiment. After the official patch #3 is applied, 3 unofficial patches are issued and we apply them and rebuild the X11R5. There is no problem in this phase. When the #6 is applied, the internationalization mechanism of X11R5 is changed and @1.1 and @2.1 become obsolete. The PM then sends e-mail telling us that reject files are generated. We remove the patches and do not rebuild X11R5. After #8, a new unofficial patch @4.1 is issued and we rebuild after applying @4.1. This experiment shows that it is easy to control the patch/unpatch operations and to find conflicts between official patches and unofficial ones. #0::$1.1,$2.1,$3.1,$4.1 #1::$1.1,$2.1,$3.1.$4.1 #2::$1.1,$2.1,$3.2.$4.1 #3:@1.1.@2.1,@3.1:$1.2,$2.1,$3.3,$4.1 #4:@1.1,@2.1,@3.1:$1.2,$2.1,$3.3,$4.1 #5:@1.1,@2.1,@3.1:$1.2,$2.1,$3.3,$4.1 #6:@3.1:$1.2,$2.1,$3.3,$4.1 #7:@3.1:$1.2,$2.1,$3.3,$4.1 #8:@3.1,@4.1:$1.3,$2.1,$3.3,$4.1 Figure 8: A part of the version tree for X11R5 Conclusion In this paper, we presented a control mechanism for applying patches and confirmed that this system works well. However, even if the patches are successfully applied, rebuilding the software still takes a long time. In order to reduce the rebuilding time, we have tried to develop a mechanism for analyzing the dependency between makefiles and a distributed make mechanism based on it. Availability The system which we presents in this paper will be available via WWW from https://www .denken.or.jp/people/cirl/futakata. Acknowledgements I would like to thank Paul Anderson at the Univ. of Edinburgh for his useful advice on drafts of this paper. My thanks also go to Yasusi Sinohara and Shouichi Matsui at the CRIEPI for their suggestion in development of this system. Author Information Atsushi Futakata graduated from the Tokyo Institute of Technology at Tokyo in 1987. He has made studies of automatic programming and management of distributed information system in the Central Research Institute of Electric Power Industry. He is now working on the CSCW environment for researchers. Reach him via mail at: Information Science Department, Communication and Information Research Laboratory, CRIEPI, 11-1 Iwado-kita 2-chome Komae-shi Tokyo 201, JAPAN His e-mail address is futakata@denken.or.jp. References [1] Tichy, W. F.: ``RCS - A System for Version Control'', Software-Practice & Experience, Vol. 15, No. 7, 1985. [2] Berliner, B.: ``CVS II: Parallelizing Software Development'', included in the CVS ver.1.3 package, 1995. [3] Bersoff, E.: ``Elements of Software Configuration Management'', IEEE Transaction of Software Engineering, Vol. SE-10, No. 1, 1984. [4] Whitgift, D.: Methods and Tools for Software Configuration Management, John Wiley and Sons, 1991. [5] Eaton, D.: ``comp.software.config-mgmt FAQ version 1.12'', posted to the comp. software.config-mgmt, 1995. [6] Palmer, G. and Hubbard, J. K.: ``The FreeBSD Ports FAQ file version 1.1'', included in the FreeBSD rel.2.0.5 package, 1995. [7] Miller, P.: ``Aegis: A project Change Supervisor User Guide'', included in the aegis ver.2.1 package, 1993. [8] DEC: CASE Environment of DEC: COHESION - COHESION handbook rev.1, DEC, 1991.