BSDCon '03 Paper
[BSDCon '03 Technical Program]
NetBSD has a cross build infrastructure which allows cross-building of an entire NetBSD release including bootable distribution media. The build process does not require root privileges or writable source directories. The build process works on many POSIX compatible operating systems. This paper explains the changes made to NetBSD to enable this build process, enumerates benefits of the work, and introduces future work enabling cross building of any software for NetBSD.
NetBSD  is the most portable Unix operating system in common use. It is freely available and redistributable, and runs on a broad variety of platforms from modern desktop systems and high end servers that can build an entire release in less than an hour, to embedded systems and older machines that may take several days to build a release.
In late 2001, work began on changes to improve the ability of NetBSD to be cross built, especially an entire release. This system is referred to as " build.sh ", because that is the name of the script that is the user-visible front-end to the infrastructure.
NetBSD 1.6 was the first release to be shipped with build.sh , and the Release Engineering group of the NetBSD Project took advantage of it to cross-build binary releases for 39 platforms on a near daily basis during the release cycle for NetBSD 1.6 . Previous releases required access to each of the various platforms by release engineers, or co-ordination with developers with that hardware. While that method works for a moderate number of platforms (NetBSD 1.5 released binaries for 20 platforms), it does not scale, especially as the number of platforms in NetBSD is growing (54 as of June 2003).
Since the NetBSD project was started in 1993 it has had a goal of being portable  to many target platforms. There has been significant effort in designing, implementing, and improving NetBSD to make it easier to "port" to a new target platform . Device drivers are written in a way that permits easy sharing between platforms without unnecessary code replication .
Prior to build.sh a NetBSD release for a given platform was built "natively" on that platform on a version of the operating system that was "close" to the target release. There were exceptions, but these alternate processes were not simple to use nor easily automated, and had a variety of other limitations which build.sh addresses.
Cross-compilation is the technique of running programs on a "host" system to generate object code for a different "target" system. This has not been an easy task for most system builders and has generally not been integrated into operating system build processes as used by open source operating systems.
Freely available software projects such as GCC  have supported being cross-compiled for a long time, and GCC is part of the GNU toolchain which NetBSD uses and is heavily dependent upon for cross-compiling.
There are many parts to a full cross compiler environment. Besides the compiler itself, many others tools and files are required to create functional programs. Everything that a normal compiler needs must be present. For the GNU toolchain, this includes:
In addition to creating executables, archive and shared libraries are built. Archive libraries are usually created with the ar binutils program from object files. Shared libraries are created by calling gcc with the -shared option, which calls ld with various options to create a shared library.
what-time-is-love ~> /tools/bin/sparc--netbsdelf-gcc -I/dest/usr/include -L/dest/usr/lib -B/dest/usr/lib/ -v -save-temps -o hello.x hello.c
what-time-is-love ~> ls -l hello.?
For a cross compiler the assembler, linker and binutils must also be "cross" tools. For the GNU toolchain these are normally called "$target-$tool". For example powerpc-eabi-ld is the linker for the "powerpc-eabi" target.
Each cross compile environment needs to provide header files and libraries appropriate to the target platform. These may be provided either as a static set of files or the build process may produce them as part of its bootstrap. The NetBSD process uses both of these techniques for header files and builds libraries from source.
This strange beast is called a "Canadian cross compiler", because when a name was needed there were three political parties in Canada. It is most often found when dealing with pre-built toolchains. The above example would produce a toolchain that will run on a i686-pc-cygwin host (i.e., a Windows system with Cygwin installed) and will target embedded PowerPC platforms. The toolchain would be built on a NetBSD/i386 machine.
The source tree can be read-only and can be shared between builds for different target NetBSD platforms. This permits building directly from read-only media or "mirrors" of source code without causing unnecessary mirroring conflicts. In this case the build output is written to a separate writable section of the file system.
No root or other special privileges are required in order to build distribution media. Previous build processes required privileges to create tar files or file systems that contained devices, files owned by other users, set-ID files, etc.
Most other build systems require special privileges. For example, the Debian fakeroot  mechanism requires the host operating system to support shared object libraries with $LD_PRELOAD functionality (where a shared object is loaded after the main program but before the program's shared object dependencies), and this breaks the next requirement.
Prior to build.sh , the creation of NetBSD boot media required root privileges to create a "loopback vnode disk device" ( vnd ), install boot blocks, create a file system on the vnd, mount the file system, and create file system entries owned by the non-building user. Most other systems still build their distribution media in this manner.
build.sh builds "host" tools to $TOOLDIR, which is a location separate to the host system's native tools. build.sh uses the tools in $TOOLDIR to build the NetBSD system installing to a separate $DESTDIR location. This removes the "chicken & egg" problem that can occur when the host platform's in-tree tools would need to be upgraded to build a newer source tree.
During the NetBSD 1.6 release engineering process, 39 platforms had near daily builds of the base operating system release, ensuring that at least all build problems were well known every day, and that up to date builds were available for testing by anyone. The majority (37) of these builds were performed on a dual processor AMD MP2000+ machine generously donated by AMD and Wasabi Systems, and the alpha and sparc64 platforms were built on Matt Thomas' CS20 alpha.
This contrasts with the NetBSD 1.5 release where 20 platforms had a binary release; all of them were built natively on the particularly port, requiring each port to have a dedicated person to build it, with 12 days time from when the first port was finished and available to the last port.
It has identified every tool in NetBSD used for building. We know what tools we need and what features these tools must support. We had to make sure these tools were portable programs that would build and run correctly on other platforms.
We no longer require running a particular version of NetBSD in order to be able to build it. Building NetBSD 1.6 from a NetBSD 1.5 machine is now supported, as is building from Solaris, Linux and more. Gone are the days of source build bootstrap problems that has plagued NetBSD since day one.
Much software is written without consideration to cross compiling and thus makes it very difficult to cross compile. Not only the way the software itself is written, but the way the software is actually built. Many software build processes build helper programs to generate real code, and so these programs must not only work correctly for the target system rather than the native system, they must be built by the "host" compiler. The automounter amd was affected by this, as the build process assumed the local host architecture was the same as the program being built, causing a sparc binary built on i386 to incorrectly byte swap data into little endian for processing.
Endianness and CPU word size issues within the toolchain can cause problems. Due to bugs in the version of GCC that we use (2.95.3), NetBSD/alpha can not be cross compiled from a 32-bit host, and NetBSD/i386 can not be cross compiled from a 64-bit host.
Not every platform has been converted to build.sh yet. In some cases (e.g., pc532, playstation2) the in-tree toolchain does not support those platforms in the build.sh framework, whilst in others the platforms may not have had a complete build infrastructure before build.sh was integrated, and there were limited resources prior to the NetBSD 1.6 release.
There is a time cost in compiling the host tools, but it is not considered significant enough to pose a problem. For example, on an AMD XP2500+, the times for the various methods of building an i386 release are:
build.sh is a Bourne shell script designed to build the entire NetBSD system on any host with a POSIX compliant Bourne shell in /bin/sh . It creates a directory for various host tools uses to cross build the system (referred to as $TOOLDIR), creates a "wrapper" to make to pass in various settings, builds the host tools with the make wrapper, and uses the host tools to build the rest of the system, into a staging area referred to as $DESTDIR.
Certain $MACHINE platforms support more than one $MACHINE_ARCH CPU architectures (a bi-endian processor or are processor with more than one word size are considered to be separate CPU machine architectures), and build.sh supports building the different target CPU architectures as separate releases.
To simplify the build process, NetBSD uses a library of make Makefile include files (with the suffix ".mk") in src/share/mk . This was inherited from the 4.3BSD Networking/2 and 4.4BSD Lite releases, and has been significantly enhanced in the decade since.
Traditionally, BSD systems are built using in-tree tools, compilers, include files and libraries. This isn't usable for cross-compilation, and has many other problems which this new infrastructure fixes, as documented in See Feature set..
src/tools contains the make infrastructure required to build various host tools used during the build. These tools are built using an autoconf-built compatibility frame-work, and use "reach over" Makefiles into the rest of the NetBSD source tree to minimize replication of code. The source code for these host tools have been slightly modified to allow them to be built in as a normal NetBSD (target) program, as well as a host tool.
The host tools are installed into $TOOLDIR, and the BSD make ".include" infrastructure in src/share/mk selects the various host tools in preference to the "standard" versions. For example, for an i386 target $CC will be set to $TOOLDIR/bin/i386--netbsdelf-gcc instead of cc .
as asn1_compile binutils cap_mkdb cat cksum compile_et config crunchgen ctags db dbsym file gcc gencat groff hexdump install installboot ld lex lint lorder m4 makefs makewhatis mdsetimage menuc mkcsmapper mkdep mkdep mkesdb mklocale mktemp msgc mtree pax pwd_mkdb rpcgen sunlabel texinfo tsort uudecode yacc zic
The majority of the host tools are installed with an " nb " prefix, to differentiate them from similarly named commands on the host. The exceptions are the GNU toolchain programs, which already have a name such as i386--netbsdelf-gcc .
To solve this issue, we enhanced the specification file for the existing mtree tool to support a full path name (versus a context-sensitive relative path name), and referred to the result as a "metalog" entry. An example entry is:
For a given build, a METALOG file is created, with a "metalog" line for each installed path name. The METALOG file is manipulated and parsed by various tools as necessary throughout the full build process.
install was modified to optionally install the paths as the current user and without any special permissions, and instead log the requested permissions to the METALOG . The ".mk" files in src/share/mk and a small number of special case Makefiles were all that needed to be modified to take advantage of this support in install .
pax was modified to support parsing a METALOG for the list of paths to add to an archive, and even add "fake" entries for devices which may not have been created in $DESTDIR. This is used to build tar.gz files which contain the correct ownership and permissions from a METALOG and $DESTDIR populated by a build by an unprivileged user. The scripts that create the "installation sets" were enhanced to parse the METALOG and invoke pax appropriately.
Various platforms use distribution media which require a ffs file system to boot from. Previously, NetBSD built these using a "loopback vnode disk driver" ( vnd ), which is not available on many other systems, and requires root privileges to mount and write to in any case.
While makefs has been written in a manner that easily supports the addition of different file system types, it currently only supports creating ffs file systems, in either little or big endian (since NetBSD's ffs code supports opposite endian ffs file systems, c.f. the "FFS_EI" kernel option, and support in userland tools such as newfs and fsck_ffs ). Support for other file systems such as iso9660, ext2fs, and FAT has been considered, but is not a high priority at this time.
The implementation of the ffs back-end re-uses a reasonable amount of the existing source code in the current NetBSD kernel implementation of ffs (in src/sys/ufs/ffs ), but for simplicity, functions such as the block allocation code were re-implemented in a simpler manner. Various assumptions were made as part of this process, including that block reallocation would not be necessary, since the size of all files and directories (as files) is known at file system creation time.
While ffs was originally implemented as a user process before its integration into the 4.2BSD kernel, after two decades of kernel hacking it is heavily tied to the buffer cache and other kernel sub-systems, which makes it more difficult to use in a stand-alone program.
Most platforms need boot blocks on their distribution media, and these are installed with installboot . Previously, each platform had its own version of installboot in /usr/mdec/installboot , which was compiled as part of, and heavily dependent upon, the kernel sources for that platform. They also required root privileges and generally required kernel support for specific disk-label ioctl() 's and only worked on actual disk devices.
/usr/sbin/installboot is a replacement for the machine-dependent versions of installboot . It can function on file system images, and doesn't need root privileges or kernel support for specific ioctl() 's to do so. The design of the boot blocks between the various platforms has been standardized as well, even if the actual implementation is different due to obvious differences in platform hardware.
All platforms that shipped with binary releases for NetBSD 1.6 (except i386) were converted to this new infrastructure. i386 was a special case due to the baroqueness of the implementation of installboot for that platform, but as the "cross build" host for the i386 release was a dual processor i386 box, it could run the tool "natively". This has been resolved in NetBSD-current.
On a related note, it is possible to make CD-ROMs that boot NetBSD on multiple platforms. For example, i386, sparc64, and macppc can boot off the same disk, and there's ample room for other platforms. NetBSD 1.6, with the 39 platforms that shipped with a binary release and associated source, fit on 4 CD-ROMs and booted on 9 of them.
NetBSD does not currently support a cross-build of "xsrc" (our copy of X11R6 / XFree86 4.x), but this will be fixed eventually. XFree86 4.3 added its own support for cross-compiling. We may implement our own method of cross compiling X11 for better integration with our build system. XFree86's cross-build solution does not address all of our requirements, especially the removal of the need for root privileges.
"pkgsrc" (the NetBSD packages collection) is not cross buildable. A smaller number of particular, probably smaller packages would probably fairly easy to get cross buildable, but the vast majority would each require significant effort.
This proposal would be acceptable for solving the "xsrc" problem in See xsrc..
build.sh has been an extremely useful solution to a variety of problems. It's now much simpler to build NetBSD releases from systems running earlier NetBSD releases, and even build on other systems such as Darwin / MacOS X, FreeBSD, Linux, and Solaris.
Generally, the support issues in NetBSD using build.sh have been less than for previous releases, especially considering the number of platforms now supported. Previously, it was extremely important to upgrade various in-tree tools, includes, and libraries in a specific order (that often changed) before completing the rest of the build. Now, a full build can be made without impacting the running system, and then an upgrade easily performed once a successful build is available.
The authors regularly use build.sh to cross-build entire releases on our (faster) alpha, i386, macppc, and sparc64 systems for platforms such as alpha, i386, macppc, pmax, shark, sparc, sparc64, and vax, as an unprivileged user using read-only source.
Erik Berls wrote and maintains the autobuild script that is used on the NetBSD Release Engineering machines to automatically build multiple release branches for multiple target platforms on multiple build machines.
This paper was originally published in the
Proceedings of BSDCon '03,
September 812, 2003,
San Mateo, CA, USA
Last changed: 21 Aug. 2003 ch