################################################ # # # ## ## ###### ####### ## ## ## ## ## # # ## ## ## ## ## ### ## ## ## ## # # ## ## ## ## #### ## ## ## ## # # ## ## ###### ###### ## ## ## ## ### # # ## ## ## ## ## #### ## ## ## # # ## ## ## ## ## ## ### ## ## ## # # ####### ###### ####### ## ## ## ## ## # # # ################################################ The following paper was originally published in the Proceedings of the 1997 USENIX Annual Technical Conference Anaheim, California, January 6-10 1997. For more information about USENIX Association contact: 1. Phone: 510 528-8649 2. FAX: 510 548-5738 3. Email: office@usenix.org 4. WWW URL: https://www.usenix.org Porting UNIX* to Windows NT David G. Korn (dgk@research.att.com) AT&T Laboratories Murray Hill, N. J. 07974 Abstract program to the native operating system directly. The openness meant that the complete The Software Engineering facilities of the native Research department at Murray operating system were Hill writes and distributes accessible through this several widely used development environment. tools and reusable libraries that are portable across The result of this effort is a virtually all UNIX set of libraries, headers, and platforms.[1] To enhance reuse utilities that we collectively of these tools and libraries, refer to as UWIN. UWIN we want to make them available contains nearly all the X/Open on systems running Windows Release 4[4] headers, NT[2] and/or Windows 95[3]. We interfaces and commands. We did not want to support discuss alternative porting multiple versions of these strategies, commercial libraries, and we wanted to products, design goals, minimize the amount of problems that had to be conditionally compiled code. overcome, and the current status. Some performance This paper describes an effort measurements of the current of trying to build a UNIX system are presented here. interface layer on top of the Windows NT and Windows 95 operating system. The goal was 1. INTRODUCTION to build an open environment rich enough to be both a good The marketplace has dictated development environment and a the need for software suitable execution environment. applications to work on a This meant that the overhead variety of operating system needed to be small enough so platforms. Yet, maintaining that there was no incentive to separate source code versions and development environments __________ * UNIX is a registered trademark, licensed exclusively through X/Open, Limited. creates additional expense and to the WIN32 Application requires more programmer Programming Interface (API)[7]. training. The API for the WIN32 subsystem is also provided with Windows One way to lower this cost is 95, although not all of the to use a middleware layer that functions are implemented. In hides the differences among the most instances binaries operating systems. The problem compiled for Windows NT that with this approach is that it use the WIN32 API will also run forces you to program to a on Windows 95. non-standard, and often proprietary, interface. In The POSIX subsystem allows addition, it often limits you applications that are strictly to the least common denominator conforming to the IEEE POSIX of features of the different 1003.1 operating system operating systems. standard[8] to run on Windows NT. Since the POSIX standard An alternative is to build a contains most of the standard middleware layer based on UNIX system call interface, existing standards. This has many UNIX utilities are simple been the approach followed by to port to any POSIX system. IBM with the introduction of Because most of our tools OpenEdition[5] for the MVS require only the POSIX operating system, URL interface, we thought that it https://www.s390.ibm.com/products/owould be sufficient to port OpenEdition is X/Open compliant them to the POSIX subsystem of so that a large collection of Windows NT. We were wrong for existing software can be the reasons described in the transported at little cost. next section. Windows NT is an operating We investigated alternative system developed by Microsoft strategies that would allow us to fill the needs of the high- to run programs on both UNIX end market. It is a layered and Windows NT based systems. architecture, designed from the After looking at all the ground up, built around a alternatives, we decided to microkernel that is similar to write our own library that Mach.[6] One or more subsystems would make porting to Windows can reside on top of the NT and Windows 95 easy. We microkernel which gives Windows spent three months putting NT the ability to run different together the basic framework logical operating systems and getting some tools working. simultaneously. For example, Realizing that the task was the OS/2 subsystem allows OS/2 larger than a one person applications to run on Windows project, we contracted a small NT. The most important development team of 2 or 3 to subsystem that runs on Windows do portions of the library, NT is the WIN32 subsystem. The packaging, and documentation. WIN32 subsystem runs all This paper will discuss porting applications that are written alternatives, the goals for our library, the issues that need work to overcome some subtle to be addressed, and the differences. In addition, the implementation of our POSIX resulting code may have a large library. Finally, we present amount of conditionally some performance results and compiled code that is hard to future directions. test and maintain. A third strategy would be to 2. ALTERNATIVE_STRATEGIES rewrite the code using a framework which provides a Six basic strategies can be virtual system interface. employed to port existing UNIX There are several vendors that based applications to Windows offer object-oriented NT. The first strategy is to application layer interfaces rewrite the code using the that encapsulate the operating WIN32 API. This strategy makes system and therefore enable sense if there are no applications to work on requirements to continue to run multiple systems. There are on a UNIX system. Otherwise, three drawbacks to this this strategy will either approach. First of all, it require two sets of source requires a large up front (which will most likely be too investment. Secondly, you will expensive to maintain) or the be locked into the vendors' use of a WIN32 emulation libraries and not able to take library that runs on UNIX advantage of savings that platforms. There are at least result from competition. two vendors that have WIN32 API Finally, you will likely be libraries for UNIX systems. We restricted to the intersection ruled out this approach because of features available on the of the effort to rewrite the underlying platforms. code to the WIN32 API and because the WIN32 API is more A fourth strategy is to port complex than the X/Open API. the application to the POSIX subsystem of Windows NT. The The second strategy is to use POSIX subsystem can run any the Microsoft C library. strictly conforming IEEE POSIX Microsoft supplies a library of application program. This routines that are similar to strategy should not require their UNIX counterparts. You major investment, and any could then make modifications investment that you make should to your application as increase the portability of necessary where the calls your application to other POSIX differ from the UNIX call. conforming systems. This strategy has been used by Unfortunately, this is not a at least one commercial UNIX viable alternative for most tools vendor to port GNU based applications. Microsoft has tools to Windows NT. While made the POSIX subsystem as this strategy is appropriate useless as possible by making for some applications, other it a closed system. There is applications may require much no way to access functionality outside of the 1990 POSIX A sixth and final strategy 1003.1 standard from within the would be to write your own POSIX subsystem, either at the POSIX library using the WIN32 library level or at the command API. After investigating the level. Thus, you cannot even other alternatives, this is invoke the Microsoft C compiler what we decided to do. We are from within the POSIX convinced that this was the subsystem. However, since you best strategy for us, since we can invoke POSIX commands from believe that it resulted in a the WIN32 subsystem, it is better implementation than the possible to port some stand two commercial products alone programs to the POSIX described later, and because it subsystem. For example, we eliminates the need to pay ported the pax utility, the licensing fees for each copy of POSIX 1003.2[9] replacement for each product that uses the cpio and tar, to Windows NT, library. The availability of and it can be invoked from any source code makes it possible WIN32 program. Softway System, to provide adequate support. Inc., URL https://www.softway.com, has an agreement with Microsoft to 3. GOALS enhance the POSIX subsystem so that they can achieve POSIX We wanted our software to work 1003.2 conformance. Softway with Windows 3.1, Windows 95, claims that they will open up and Windows NT. A summer the POSIX subsystem so that it student wrote a POSIX library can access WIN32 applications. for Windows 3.1 and we were Even if the POSIX subsystem on able to port a number of our Windows NT is opened up, the tools. However, the limited POSIX subsystem is not capabilities of Windows 3.1 available for Windows 95. made it a less than desirable platform. We instead focused The fifth strategy is to use an our goals on Windows NT and existing POSIX or X/Open Windows 95. We decided to use library that runs in the WIN32 only the WIN32 API for our subsystem. At the time that we library so that the library began this effort, we were would work on Windows 95 and so aware of two vendors that sell that all WIN32 interfaces would such libraries but as discussed be available to applications. later, these products were less than satisfactory. In Initially, our goal was to addition, Steve Chamberlain at provide the IEEE POSIX.1 Cygnus has started writing a interface with a library. This POSIX interface for Windows NT would be sufficient to run ksh and Windows 95, but it appears and about eighty utilities that as if his goals are less we had written. It soon became ambitious than ours, URL obvious that this wasn't enough https://www.cygnus.com/misc/gnu- for many applications. Most win32/. real programs use facilities that are not part of this standard such as sockets or wanted to be able to compile IPC. and execute UNIX programs without having to understand We needed to provide a their semantics. In addition character based terminal we wanted to limit the number interface so that curses based of new interfaces functions and applications such as vi could environments variables that we run. After the initial set of had to add to use our library. utilities was running, we It is difficult to manage more wanted to get several socket than one or two environment based tools working. Several variables when installing a new projects at AT&T that became package. interested in using our libraries, required the System Another goal that we had was to V IPC facilities. The S provide a robust set of graphics system[10] and ksh- utilities with minimal 93[11] required runtime dynamic overhead. If utilities written linking. As the project to the X/Open API were progressed, the need for noticeably slower than the same privileged users, such as root utilities written to the native on UNIX systems, surfaced. We WIN32 API, then they were decided that it was important likely to be rewritten making to have setuid and setgid our library unnecessary in the capabilities. It soon became long run. clear that we needed full UNIX functionality and we set our A final and important goal was goal on X/Open Release 4 interoperatability with the conformance. native Windows NT system. Integration with the native We needed to have a complete system not only meant that we set of UNIX development tools could use headers and libraries since we didn't want to get from the native system, but into the business of rewriting that we could pass environment makefiles or changing build variables and open file scripts. Most code written at descriptors to commands written AT&T, including our own, uses with the native system. There nmake[12], (no relation to the couldn't be two unrelated sets Microsoft nmake), but we also of user ids and separate wanted to be able to support passwords. If write permission other make variants. We didn't were disabled from the UNIX want to do manual configuration system, then there should be no on tools that have automatic way to write the file using configuration scripts. facilities in the native system and vice versa. One important goal that we had from the beginning was to not We have not as yet achieved all require WIN32 specific changes of our goals, but we think that to the source to get it to we are close. We are in the compile and execute. The process of running the X/Open reason for this is that we conformance tests to verify compliance with the X/Open the path name, so that the API's. The rest of the paper pathname c:\home\dgk names the will discuss some of the issues file in directory \home\dgk on we needed to deal with and our drive c:. Many UNIX utilities solutions. expect only / separated names, and expect a leading / for absolute pathnames. They also 4. PROBLEMS_TO_SOLVE expect multiple /'s to be treated as a single separator. The following problems need to be understood and dealt with in Even though NTFS supports case porting applications to Windows sensitivity for file names, the NT. These are some of the WIN32 API has no support for issues that need to be case sensitivity for addressed by POSIX library directories and minimal support implementations. Section 6 for case sensitivity for files, describes how UWIN solved most limited to a of these problems. FILE_FLAG_POSIX_SEMANTICS creation flag for the 4.1 Windows_NT_File_Systems CreateFile() function. Certain characters such as *, ?, >, |, Windows NT supports three :, ", and \, cannot be used in different file systems, called filenames created or accessed FAT, HPFS, and NTFS. FAT, with the WIN32 API. The names, which stands for File Access aux, com1, com2, nul, and Table, is the Windows 95 file filenames consisting of these system. It is similar to the names followed by any suffix, DOS file system except that it cannot be created or accessed allows long file names. There in any directory through the is no distinction between upper WIN32 API. and lower case although the case is preserved. HPFS, which Because Windows 95 doesn't stands for High Performance support execute permission on File System, was designed for files, it uses the .exe suffix OS/2. NTFS, the native NT File to decide whether a file is an System, is similar to the executable. Windows NT doesn't Berkeley file system.[13] It require this suffix, but some allows long file names (up to NT utilities, such as the DOS 255 characters) and supports command interpreter, require both upper and lower case the .exe suffix. characters. It stores file names as 16 bit Unicode names. 4.2 Line_Delimiters The file system namespace in Windows NT uses the DOS Win32 is hierarchical as it is convention of a two character in UNIX and DOS. A pathname sequence to signify can be separated by either a / the end of each line in a text or a \. Like DOS, and unlike file. UNIX uses a single UNIX, disk drives are specified to signify end of line. The as a colon terminated prefix to result is that file processing is more complex than it is with of an array of handles. UNIX. There are separate modes Unfortunately, socket handles for opening a file as text and do not have this attribute. binary with the Microsoft C One of the few novel features library. Binary mode treats of WIN32 is the ability to the file as a sequence of create a handle for a directory bytes. Text mode strips off with the synchronize attribute. each in front of each This handle changes state when new-line as the file is read, any files under that directory and inserts a in front of change. This is how multiple each as the file is views of a directory can be written. Because the number of updated correctly in the characters read doesn't presence of change. indicate the physical position of the underlying file, 4.4 Inconsistent_Interfaces programs that keep track of characters read and use lseek() The WIN32 API handle interface are likely to not work in text is often inconsistent. mode. Fortunately, many Failures from functions that programs that run on Windows NT return handles return either 0 do not require the in or -1 depending on the front of each in order to function. The CloseHandle() work. This difference turned function does not work with out to be less of a problem directory handles. The WIN32 that we had originally API is also inconsistent with expected. respect to calls that take pathname arguments and calls 4.3 Handles vs. file that take handles. Some descriptors functions require the pathname and others require the handle. The WIN32 API uses handles for In some instances, both calls almost all objects such as exist, but they behave a little files, pipes, sockets, differently. processes, and events, and most handles can be duped within a 4.5 Chop_Sticks_Only process or across process boundaries. Handles can be The WIN32 subsystem does not inherited from parent have an equivalent for fork() processes. Handles are or an equivalent for the analogous to file descriptors exec*() family. There is a except that they are unordered, single primitive, named so that a per process table is CreateProcess() that takes 10 needed to maintain the arguments, yet still cannot ordering. perform the simple operation of overlaying the current process Many handles, such as pipe, with a new program as execve() process, and event handles, requires. have a synchronize attribute, and a process can wait for a change of state on any or all 4.6 Parent/Child_Relationships 4.8 Ids_and_Permissions The WIN32 subsystem does not Windows NT uses subject support parent/child identifiers to identify users relationships between and groups. A subject processes. The process that identifier consists of an array calls CreateProcess() can be of numbers that identify the thought of as the parent, but administrative authority and there is no way for a child to sub-authorities associated with determine its parent. Most a given user. A UNIX user or resources, such as files and group id is a single number processes, have handles that that uniquely identifies a user can be inherited by child or group only within a single processes and passed to system. Information about unrelated processes. Any users is kept in the a registry process can wait for another database which is accessible process to complete if it has via the WIN32 API and the LAN an open handle to that process. manager API. There is a limited concept of process group that affects the Windows NT uses an access distribution of keyboard control list, ACL, on each signals, and a process can be file or object to control the placed in a new group at access of the file or object startup or can inherit the for each user. UNIX uses a set group of the parent process. of permission bits associated There is no way to get or set with the three classes of the process group of an users; the owner of the object, existing process. the group that the object belongs to, and everyone else. 4.7 Signals While it is possible to construct an access control The WIN32 API provides a list that more or less structured mechanism for corresponds to a given UNIX exception handling. Also, permission, it is not always signals generated from within a possible to represent a given process are supported by the access control list with UNIX API. However, signals permissions. generated by another process have no direct method of Windows NT has separate implementation. In addition to permissions for writing a file, being able to interrupt deleting a file, and for processing at any point, a changing the permission on a signal handler might perform a file. The write bit on UNIX longjmp and never return. systems determines all three. Thus, it is possible to encounter files that have partial write capability. UNIX processes have real and effective user and group id's that control access to to the screen. resources. Windows NT assigns each process a security token 4.10 Special_Files that defines the set of privileges that it has. UNIX The WIN32 API supports unnamed systems use setuid/setgid to pipes with the UNIX semantics. delegate privileges to Named pipes are also supported processes. Windows NT uses a but have different semantics technique called impersonation than fifos and occupy a to carry out commands on behalf separate name space. There is of a given user. There is no no /dev directory to name user that has unlimited special files such as /dev/tty privileges as the root user and /dev/null. The WIN32 does does with UNIX. Instead the support special names of the special privileges of root have form \\.\PhysicalDrive for disk been broken apart into separate drives and tape drive devices. privileges that can be given to one or more users. One of the Windows NT supports hard links biggest challenges we faced was to files, but there is no WIN32 providing the UNIX model of API call to create these links. setuid/setgid on top of the WIN They do not support symbolic 32 interface. links in the file system directly, but on Windows 95 and The implementation of WIN32 for on Windows NT 4.0, the file Windows 95 does not support the browser does support short cuts NT security model and calls which are very similar to return a not implemented error. symbolic links. 4.9 Terminal_Interface 4.11 Shared_libraries Windows NT and Windows 95 allow The WIN32 API supports the each character based linking of shared libraries at application to be associated program invocation and at run with a console which is similar time. The libraries are called to an xterm window. Consoles dynamically linked libraries or support echo and no echo mode, DLL's and are represented by and line at a time or character two separate files. One file at a time input mode, but lack provides the interface and is many of the other features of needed at compile time to the POSIX termios interface. satisfy external references. There is no support for The second file contains the processing escape sequences implementation as is needed at that are sent to the console run time. window. In echo mode, characters are echoed to the There are some restrictions on console when a read call is DLL's that are not found on pending, not while they are UNIX system shared library typed. There are separate implementations. One console handles for reading restriction is that you cannot from the keyboard and writing override a function called by a DLL by providing your own variables, such as PATH are version of the function. Thus, used by both WIN32 and by UNIX, supplying your own malloc() and yet have different formats. free() functions will not UNIX uses a : separated list override the calls to malloc() of pathnames; WIN32 uses a ; and free() made by other DLL's. separated list. Secondly, the library can only contain pointers to data, not data itself. Thus, making a 5. COMMERCIAL POSIX LIBRARY symbol such as errno part of a INTERFACES DLL is impossible. Even making symbols such as stdin point to We purchased software from the data in a DLL invites trouble two commercial vendors that we since it is not possible to were aware of that sell POSIX compile code that uses libraries for Windows NT that static FILE *myfile = run under the WIN32 subsystem. stdin; Each offers a software development kit containing 4.12 Compilers_and_libraries include files and libraries, and each offers a set of UNIX Microsoft sells the Visual utilities. Both of these C/C++ compiler for Windows NT vendors require a license to and Windows 95. This compiler use their libraries in has both a graphical and products. We used earlier command line interface. versions of their products but Microsoft also sells a software based on their web pages at the developers kit (SDK) that time this paper was written, contains tools, including the the following description still Microsoft nmake. The compiler applies. Both of these vendors and linker use a different set supply cc commands that invoke of flags than standard UNIX the underlying Microsoft Visual compilers, and C files produce C/C++ compiler. Neither of .obj files by default, rather these products support symbolic than .o files. Fortunately, links, job control and fifos. the linker can handle both .obj Both appear to have implemented and .o files. The linker has the exec*() family incorrectly options to choose a starting in that the process that does address and to specify whether the exec does not terminate the application is a console until the child process application, a GUI application, completes. A process that a POSIX application, or a repeatedly execs itself will dynamically linked library. eventually cause the operating system to run out of processes. 4.13 Environment_Variables It is not clear from their home pages whether either of these The WIN32 API supports the products work with Windows 95. creation and export of environment variables in much the same way that UNIX systems do. Some environment 5.1 NuTCracker_from_DataFocus 5.2 Portage_from_Consensys NuTCracker, by Datafocus, URL The other product that we https://www.datafocus.com, makes purchased is named Portage and an attempt to support UNIX is sold by Consensys Systems, conventions. It maps Windows URL https://www.consensys.com. NT file names to and from UNIX The source is based on System file names, and adjusts the V, Release 4, which makes it PATH environment variable the more suitable for most AT&T accordingly. For example, it products. Their utilities were maps the Windows NT file name built from the System V source, d:\bin to the UNIX filename but it was clear that changes /d=/bin and handles the special were made in order to port them names /dev/null and /dev/tty. to Windows NT. The = is a poor choice because the POSIX.2 standard for the Portage Version 1.0 does not shell language leaves the map Windows NT file name into behavior of commands that have UNIX names. They have modified an = in their name unspecified. some tools such as ksh to recognize ; as the PATH NuTCracker ships the MKS delimiter in place of :. Toolkit as the utilities. The Version 1.0 did not support MKS Toolkit is a completely case distinction, but their independent implementation that home page indicates that they does not use the NuTCracker now do. libraries. We view this as a serious deficiency since the In terms of functionality, the behavior or the utilities is no NuTCracker suite is more guide as to the correctness or complete than Portage. functionality of the NuTCracker library. 6. UWIN DESIGN AND The NuTCracker library lacks IMPLEMENTATION some functions not defined by POSIX or ANSI C that are We started work on writing our available on UNIX systems such own POSIX library at the as hsearch() and cuserid(). beginning of 1995 after being frustrated with the existing In addition to the above commercial products. We were deficiencies, NuTCracker does able to put together a useful not support filename case subset of functions in about 3 distinction. months. However, to be successful, it was necessary to NuTCracker supports a Motif provide as complete a package library for porting X11 based as possible. The library applications including a needed to handle console and version that offers a Windows serial line support, sockets, look and feel. UNIX permissions, and other commonly used mechanisms such as memory mapping, IPC, and dynamic linking. In addition, shared by all the currently to be useful, the libraries had active UNIX processes in a to be documented and supported. memory mapped region. This This put the scope of the region is writable by all project outside of the reach of processes so that an ill- a small research department behaved process could affect such as ours. another process. Even though all processes have read and We subcontracted some of the write access to the shared development to AT&T GIS in segment, secure access to India to help complete this kernel objects in Windows NT is project. We jointly designed not compromised by this model the terminal interface and the because a process must have group in India implemented it. access rights to an object to They also worked on completing use it; knowing its address or the sockets library. They value doesn't give additional packaged the software for access rights. Some initial installation and are providing measurements indicated that the documentation. This section alternative of having a server describes the UWIN process update the shared implementation and how we memory region, would have had a solved many of the problems performance penalty that we did described in Section 4. not believe was worth the cost. However, this is an area for 6.1 UWIN_Architecture future investigation. The current implementation of The open file table is an array UWIN consists of two of structures of type Pfd_t as dynamically linked libraries illustrated in Table 1. named posix.dll and ast.dll that more or less implement the ______________________ functions documented |_____________________| respectively in section 2 and |long refcount | section 3 of UNIX manuals. In |_____________________| addition, a server process |_____________________| named UMS runs as Administrator |char type | (the closest thing to root). |_____________________| UMS generates security tokens |_____________________| for setuid/setgid programs as needed. It also is responsible TABLE 1. File Table Structure for keeping the /etc/passwd and /etc/group files consistent The refcount field is used to with the registry database. keep track of free entries in The Architecture for UWIN is this table. The Win32 illustrated in Figure 1. The InterlockedIncremenet() and UMS server does not exist for InterlockedDecremenet() Windows 95. functions are used to maintain this count so that concurrent The posix.dll library maintains access by different processes an open file table that is will work correctly. The oflag field stores the open flags for __________________________________________ the file. The type field |_________________________________________| indicates what type of file, |long refcount | regular, pipe, socket, or |_________________________________________| special file. The function |_________________________________________| that is used read from or to |HANDLE sigevent | write to the file depend on the |_________________________________________| value of type. For certain |_________________________________________| types, the extra field stores |HANDLE etok,rtok | an index into a type-specific |_________________________________________| table that stores additional |_________________________________________| information about this file. |pid_t pid,ppid,pgrp,sid | |_________________________________________| The posix.dll library also |_________________________________________| maintains a per process |Psig_t siginfo | structure, Pproc_t. The per |_________________________________________| process structure contains |_________________________________________| information required by UNIX |ulong alarmremain | processes that is not required |_________________________________________| by Win32 processes such as |_________________________________________| parent process id, process |time_t cutime,cstime | group id, signal masks, and |_________________________________________| process state as illustrated in |_________________________________________| Table 2. TABLE 2. Process Table Like the open file table, the Structure process table maintains a reference count so that process The process structure contains slots can be allocated without an array of up to OPEN_MAX creating a critical region. structures of type Pprocfd_t The meaning of most of the that is indexed by file fields in the process structure descriptor. The Pprocfd_t can be deduced by its name. structure contains the close- The Psig_t structure contains on-exec bit, the index of the the bit mask for ignored, file in the open file table, blocked and pending signals. and the corresponding handle or When the first child process is handles as illustrated in Table invoked by a process, a thread 3. is created that waits for this and subsequent processes to The posix.dll library complete. The waitevent field implements the malloc(), contains an event this thread realloc(), and free() interface also waits on so that using the Vmalloc library additional children can be written by Kiem-Phong Vo[14]. added to the list of children The Vmalloc library provides an to wait for. interface to walk over all memory segments that are allocated which is needed for the fork() implementation described later. __________________________ explicitly opened for reading |_________________________| as a text file, an Sfio |short index | discipline for read() and |_________________________| lseek() can be inserted on the |_________________________| stream to change all |HANDLE primary | sequences are to . The |_________________________| lseek() discipline uses logical |_________________________| offsets so that the removal of characters is transparent. TABLE 3. Process file We did not provide a discipline structure to change to since we discovered that most The ast.dll library provides a Windows 95 and Windows NT portable application utilities worked without the programming interface that is s. The s could be used by all of our utilities. inserted by a filter such as The interface to this library sed if required. is named libast.a, for compatibility with its name on 6.2 Files UNIX systems. libast.a provides C library functions The posix.dll library performs that are not present on all the mapping between handles and systems so that application file descriptors. Usually, code doesn't require #ifdefs to each file descriptor has one handle system dependencies. handle associated with it. In libast.a is built using the some cases, two handles may be iffe command [15] to feature associated with a file test the host system and descriptor. An example of this determine what interfaces do is a console that is open for not exist in the native system. reading and writing which uses separate handles for reading libast.a relies on the and writing. Microsoft C library for most of the ANSI-C functionality. The The posix.dll library handles most significant exception to the mapping between UNIX this, other than malloc() which pathnames and WIN32 pathnames. is provided by posix.dll, is Many UNIX programs assume that the stdio library. libast.a pathnames that do not begin provides its own version of the with a / are relative stdio library based on calls to pathnames. In addition, only / Sfio[16]. The Sfio library is recognized as a delimiter. makes calls to posix.dll rather There is only a single root than making direct calls to the directory; the operation of WIN32 API as the Microsoft C changing to another drive does library does so that pathnames not change the root directory. are correctly mapped. The posix.dll library maps all file names it encounters. If The use of Sfio also provides a the file name begins with a / simple solution to the and the first component is a problem. When a file is single letter, then this letter is taken as the drive letter. 6.3 fork/exec Thus, the UNIX filename /d/bin/date gets translated to The fork() system call was d:\bin\date. The file name implemented by creating a new mapping routine also recognizes process with the same startup special file names such as information as the current /dev/tty and /dev/null. A / process. Before executing not followed by a drive letter main(), it copies the data and is mapped to the drive that stack of the parent process UWIN has been installed on so into itself. Handles that were that programs that embed closed when the new process was absolute pathnames for files in created are duplicated into the /bin, /tmp, /dev, and /etc work new process. The exec*() without modification. family of functions was much harder to implement. The Finally, the path search problem is that there is no way algorithm was modified to look to overlay the calling process. for .exe and .bat suffices. Portage and NuTCracker have the current process wait for the One problem introduced by the child process to complete and pathname mapping is that then exit. There are two passing file name arguments to problems with this approach. native NT utilities is more First, a process that execs difficult since it understands repeatedly will fill up the DOS style names, not UNIX process table. More names. A library routine was importantly, resources from the added to return a DOS name parent process are not given a UNIX name. released. Our method causes the child process to be The posix.dll library pathname reparented to the grandparent mapping function also takes and the process that calls care of exact case matching on exec*() to exit. The process file systems that require it. id returned by the getpid() One of the most troublesome function will be the process id aspects of the WIN32 API is its of the process that invoked the lack of support for pathname exec*() function. In other case distinction. It is not cases, it will be the same uncommon to have files named process id as the WIN32 uses. Makefile and makefile in the To prevent that process id from same directory in UNIX. UWIN being used again by WIN32, a handles case distinction by handle to the process is kept calling the WIN32 CreateFile() by the grandparent process. function both with and without the FILE_FLAG_POSIX_SEMANTICS Even though we implemented function. If they compare fork() and the exec*() family equal, it executes the function of functions, our code rarely internally, otherwise it spawns uses them. Because the a POSIX subsystem process to CreateProcess() function carry out the task. doesn't have an overlay flag, two processes need to be created in order to do both 6.5 Terminals fork() and exec*(). libast provides a spawn*() family of The POSIX termios interface is functions that combines the implemented by creating two functionality of fork()/exec*() threads; one for processing on systems that don't have the keyboard input events, and the spawn*() family. All functions other for processing output in libast that create processes events and escape sequences. such as system() and popen() These threads are connected to are programmed with this the read and write file interface. On most UNIX descriptors of the process by systems, the spawn*() family is pipes. The same architecture written using fork() or vfork() is used for socket based and exec*(). We implemented terminals and serial I/O lines. spawn*() in our posix.dll Initially, these threads run in library to call CreateProcess() the process that created the directly. console and make it the controlling terminal. These 6.4 Signals threads service all processes that share the controlling Signals are handled by having terminal. New threads will be each process run a thread that created if the process that waits on an event. To send a owns the threads terminates and signal to a process, the bit another process is sharing the corresponding to the given console. When a process is signal number is set in the created, these threads are receiving process's process suspended and the console block, and then its signal handles are passed down to the thread event is set. The child. This enables a native signal thread then wakes up and application to run with its looks for signals. It is standard input and output as important for the signal console handles. If the handler to be executed in the application has been linked primary thread of the process, with the posix.dll, then these since the handler may contain a threads are resumed before longjmp() out of the handler main() is called so that UNIX function. Prior to calling style terminal processing takes main(), an exception filter is place. The result is that UNIX added to the primary thread processes will echo characters that checks for signals. The as they are typed and respond signal thread does this by to special keys specified by suspending the primary thread stty, whereas native WIN32 raising an exception that will applications will only echo active the exception filter of characters when they are read the primary thread, and then and will use Control-C as the resuming the primary thread. interrupt character. 6.6 Ids_and_Permissions permission bits. With the NT file system, this extra Permissions for files are only information has been stored by available on Windows NT. Calls using a poorly documented to get an set permissions feature called multiple data return not implemented errors streams that allows a file to on Windows 95. Creating a have multiple individually Windows NT ACL that closely named parts. A separate data corresponds to UNIX permissions stream is created to hold isn't very difficult. The ACL additional information about needs three entries; one for the file. The SYSTEM attribute owner, one for group, and one is put on any file or directory that represents the group that that has an additional data contains all users. Windows NT stream so that they can be allows separate permission to identified quickly with minimal delete a file and to change its overhead during pathname security attribute. These mapping. permissions are give to the owner of a file. The UNIX Using multiple data streams umask() command sets the requires the NT file system. default ACL so that native On other file systems, fifos applications that are run by and symbolic links are UWIN will create files with implemented by storing the UNIX type permissions. information in the file itself. The setuid, setgid Mapping of subject identifiers functionality is not supported to and from user and group ids on these file systems. is more complex. UWIN maintains a table of subject UWIN treats Windows 95 and identifier prefixes, and Windows NT 4.0 short cuts as if constructs the user id and they were symbolic links. group id by a combination of However, these links can be the index in this table and the created with any of the UWIN last component of the subject interfaces. This was done by identifier. The number of reverse engineering the format subject identifier prefixes of a short cut file and finding that are likely to be where the pathname of the file encountered on a given machine that it referred to was stored. is much smaller than the number of accounts so that this table Fifos are implemented by using is easier to maintain. WIN32 named pipes. A name is selected based on the creation 6.7 Special_files_and_Links date of the fifo file. Only the first reader and the first Special files such as fifos and writer on the fifo create and symbolic links require stat() connect to the named pipe. All information that is not kept by other instances duplicate the the NT or FAT file systems. handle of either the reader or Also, the file system does not the writer. This way all store the setuid and setgid writers to a fifo use the same handle as required by fifo it possible to use the UNIX semantics. implementation of tcl to port tksh[17] applications to A POSIX subsystem command is Windows NT. also invoked to create hard links since there is no WIN32 6.9 Invocation API function to do this. Hard links fail for files in the FAT When UWIN invokes a process, it file system. does not know whether the process is a UWIN process or a 6.8 Sockets native process. It modifies the PATH variable so that it Sockets are implemented as a uses the ; separated DOS layer on top of WINSOCK, the format. It also passes open Microsoft API for BSD sockets. files in the same manner that Most functions were straight the Microsoft C library does so forward to implement. The that programs that are compiled select() function proved more with this library should difficult than we had correctly inherit open files anticipated because socket from UWIN programs. The handles could not be used for initialization function also synchronization, and because sees whether a security token the Microsoft select() call has been placed in its address only worked with socket space by the UMS server, and if handles. The posix.dll so, it impersonates this token. select() function allows different types of file The POSIX library has an descriptors to be waited for. initialization routine that sets up file descriptors and Our first implementation of assigns the controlling select() created a separate terminal starting the terminal thread that used the Microsoft emulation threads as required. select() to wait for socket The posix.lib library also handles, and created an event supplies a WinMain() function for the main thread to add to that is called when the program the list of handles to wait begins. This function for. Our second implementation initializes the stdin, stdout, used a library routine to and stderr functions and then convert input/output events on calls a posix.dll function sockets to windows messages and passing the address of another then waited for both windows posix.lib function that messages and handle events actually invokes main(). The simultaneously. This method posix.dll function starts up up had the added advantages that the signal thread and sets the it was possible to implement exception filter for signal SIGIO and that it was easy to processing as described above. add a pseudo file device named The reason for this complexity /dev/windows that could be used is so that UNIX programs will to listen for windows messages. start with the correct Adding this pseudo device made environment, and so that argv[0] will have UNIX syntax programs work. Our compiler without the trailing .exe since wrapper can be invoked as cc many programs use argv[0]. for ANSI-C compilation, as CC Much of the complexity occurs for C++ compilation, and as pcc inside the posix.dll part to build POSIX subsystem because programs do no require applications. recompilation when changes are added there. Our compiler wrapper follows the normal UNIX defaults for suffixes rather than using the 7. CURRENT_STATUS Microsoft conventions; .o's rather than .obj's. The .exe At the time of this writing, suffix is not required for most interfaces required by the Windows NT since it uses X/Open Release 4 standard have permission bits to distinguish been written and work as executables. However, since we described in the standard. The also want binaries to run on X/Open standard requires full Windows 95, the .exe suffix is ANSI C functionality as well. added to the name of the output In addition, interfaces for the file if no suffix is supplied curses library, the sockets when the compiler is invoked as library, the dynamic linking cc or CC. library, are also working. The lastest version of ksh, A C/C++ compiler wrapper has ksh-93 was ported. The been written that calls either implementation supports all the Microsoft Visual C/C++ 2.x features of ksh-93 including or 4.x compiler. This compiler job control and dynamic linking supports the most commonly used of built-in commands at run UNIX conventions and implicitly time. While no changes to the sets default include files and code should have been libraries. In addition it has necessary, changes were made to an added hook for specifying ksh specifically for NT. The native compiler and linker hostname mapping attribute, options. Applications compiled typeset -H, which has no effect with our cc command can be on UNIX systems, was modified debugged with native debuggers to call the posix.dll function such as the Visual C/C++ that returns the WIN32 pathname debugger. Several auto corresponding to a given UNIX configuration programs use the pathname. The ability to do output of the C preprocessor to case insensitive matching for probe the features of the file expansion was also added. system. The output format of A compile time option to allow the Microsoft C compiler caused in place of was some of the configuration added to the shell grammar to programs to fail. To overcome avoid the overhead of text file this, a filter is inserted when processing. running the compiler to generate preprocessor output so About 150 UNIX tools have been that existing configuration ported to Windows NT, the vast majority required no changes. performance measures one of the Common software development costs involved in using UWIN as tools such as yacc, lex, make opposed to using an alternative and nmake have also been strategy such as rewriting to ported. Most of the utilities the WIN32 API. are versions that we have written at AT&T over the last The second is how Windows NT ten years and are easily performs relative to other UNIX portable to all UNIX platforms. systems. The performance of Other utilities, such as make, UNIX operating systems on the bc, and gzip we compiled from Pentium processor was the GNU source using autoconfig investigated by Keven Lai and to generate headers and Mary Baker[18], and showed that makefiles. The yacc and less except for networking, the utilities and the new vi Linux system, URL program were ported from freely https://www.linux.org, performs available BSD source code. In the best of the UNIX systems. most cases, no changes were The comparison to a UNIX system made to the original source may be important in deciding code. whether to choose a UNIX platform or a Windows NT The X Windows code has two platform, although other parts, the client and the considerations often dictate server. The server had already this choice. been ported to Windows NT and Windows 95 by commercial All performance comparisons vendors and there was no need were made on an Micron computer to build UWIN version for it. with 133MZ Pentium processor In addition, the server is and 32M-bytes of memory. The often running on a UNIX host. WIN32 measurements were made The most difficult part of using the NTFS file system on porting the X Windows client Windows NT 4.0 operating code was the fact that it had system. The UNIX measurements #ifdefs for WIN32 that selected were make on Linux version native WIN32 calls, bypassing 2.0.18 on the same hardware. the UWIN calls. Once this was straightened out, the There are four sets of tests. compilation was The first set of tests, shown straightforward. in Table 4, are the same ones used in the 1991 Usenix Sfio paper. The implementation of 8. PERFORMANCE stdio under UWIN uses Sfio rather than the Microsoft There are two issues to implementation since the consider with respect to Microsoft implementation makes performance. The first is how WIN32 calls directly rather UWIN performs compared to using than going through the the WIN32 API and/or Microsoft read()/write() UNIX interface. C library directly. The The Linux tests were run with comparison of UWIN to native an Sfio implementation rather than using the native due to the way UWIN deletes implementation to make it files to provide UNIX easier to compare results. The semantics. With UNIX it is results show that applications possible to delete a file while that are dominated by calls to it is open, and then create a Stdio or Sfio are likely to file of the same name. Clearly perform at least as well when a more efficient mechanism for run under UWIN. doing this is needed. The second set of tests, While reading small directories summarized in Table 5, measures is slower than with Linux, the the performance of certain tests show that large systems calls; for example the directories are actually faster time to open and close files, with NT. The system() test to read and write data, the shows that Linux is quite a bit time to create and delete faster in launching processes files, and the time to open and than NT. The NT native test, read directories. The tests unlike the UWIN test, executes are as follows: /bin/echo directly without running the shell so that the 1. Open and close a file results are better than they 10000 times. might otherwise be. 2. Create and delete a file The third set of benchmarks is 10000 times. called the Modified Andrew Benchmarks[19]. These 3. Open and read a directory benchmarks measure the elapsed containing two files time to perform a set of tasks 10000 times. such as copying files, doing recursive walks, and compiling 4. Open and read a directory code. The original set of containing 500 files Andrew Benchmarks used the 10000 times. native C compiler; the Modified Andrew Benchmarks come with 5. Run system("/bin/echo") source code for a stripped down 100 times. version of gcc so that the The tests were run five times differences in compilers can be and the middle three times were eliminated. It look little averaged. The first four tests effort to modify the makefiles report the sum of user+system and build the compiler. The time. The last test uses only time to run the Modified Andrew elapsed time because of the Benchmark was 110 seconds under difficulty of obtaining UWIN. The time was a mere 18 accumulated times for processes seconds under Linux. This using CreateProcess() call. benchmark shows the effect of the slower file and process These tests show that creating creation times. The UWIN times and deleting files in UWIN is could be improved by using the much slower than with Linux. spawn family of functions in Much of the time difference is place of fork/exec to execute the components of the compiler provide a transparent as the UWIN cc command does. interface. In both cases the test failed to complete near the final The current version of UWIN step; creating the archive does not handle may of the because the native archiver was internationalization issues unable to handle the format well. The current produced by the generated gcc implementation has been compiler. compiled for ASCII rather than UNICODE. We plan to use UFT8 The final set of benchmarks encoding of UNICODE for the that we tried to run was the system call interface, and to benchmark suite named lmbench, convert to UNICODE on the NT written by Larry McVoy and file system. This way we do presented at the 1996 USENIX not need to build separate conference[20]. We were able binaries for UNICODE. to run only a portion of these benchmarks because many of the The current version of UWIN benchmarks require the rpc does not support files larger library which hasn't been than two gigabytes because the ported to UWIN yet. In size of off_t is stored as a 32 addition, we omitted tests that bit integer. The underlying were covered by earlier NTFS file system supports 64 benchmarks. The results of bit file offsets. Since the this benchmark are presented in next version of Sfio supports Table 6. While it confirms 64 bit file offsets, we plan to that the I/O bandwidth under support large files in a future UWIN is quite good, it also version of UWIN. shows that other aspects such as pipe latency is quite large. Another issue worth We did not investigate this investigating is whether it is discrepancy. possible to run Linux binaries under UWIN. This would only make sense for dynamically 9. FUTURE linked programs. We are in the process of Finally there are some WIN32 running X/OPEN conformance interfaces that could be tests on UWIN and see how close handled through the file system we have come to being interface such as the Windows compliant. In addition, we are NT registry and the clipboard. trying to decide how to port the n dimensional file system, n-DFS[21], to Windows NT. n- 10. CONCLUSIONS DFS provides a mechanism to add file system services such as There appear to be few if any viewpathing and versioning. technical reasons to move from The difficulty in porting n-DFS UNIX to Windows NT. The is that it must also capture performance of Linux exceeds native WIN32 API calls to that of NT 4.0 and Linux appears to be more reliable. On three occasions NT 4.0 crashed when running the performance tests. There were no crashes with Linux. However, if you want to or need to move an application to Windows 95 or Windows NT, we believe the POSIX library we developed to be superior to any of the existing commercial libraries. While in many cases the performance loss using UWIN is minimal, the performance tests show that UWIN needs improvement. The code for the posix.dll library is fairly small, about 10K lines including the terminal emulator. This library runs in the WIN32 subsystem using the WIN32 API and runs under Windows 95 as well. We hope to be able to make version 1.1 of UWIN available in binary form on the internet. Check the internet web site https://www.research.att.com/sw/tools for details. We hope that this will encourage contributions of applications that have been built with UWIN. _________________________________________________________________________ |_________________||_________________||________________||________________| | test || size ||seconds | Kb/s ||seconds | Kb/s ||seconds | Kb/s | |________||_______||________|________||________|_______||________|_______| |fwrite ||10000K || 0.63 | 15923 || 0.49 | 20408 || 0.85 | 11709 | | fread ||10000K || 0.28 | 36231 || 0.27 | 36764 || 1.03 | 9671 | | revrd ||10000K || 0.31 | 32679 || 0.33 | 30303 || 0.50 | 19960 | | fw757 ||10000K || 0.76 | 13227 || 0.94 | 10593 || 1.06 | 9416 | | fr757 ||10000K || 0.41 | 24154 || 0.36 | 27472 || 1.09 | 9199 | |rev757 ||10000K || 0.91 | 10989 || 1.36 | 7731 || 0.66 | 15128 | |copy&rw ||10000K || 1.01 | 9940 || 1.00 | 9999 || 1.39 | 7173 | |seek+rw || 2000S || 0.93 | 34566 || 0.83 | 38672 || 1.78 | 8974 | | putc || 5000K || 0.87 | 5720 || 1.00 | 5020 || 2.09 | 2393 | | getc || 5000K || 0.49 | 10245 || 0.50 | 9960 || 1.57 | 3194 | | fputs ||50000L || 0.51 | 97656 || 0.59 | 84459 || 0.88 | 57102 | | fgets ||50000L || 0.40 | 126262 || 0.65 | 77399 || 0.54 | 93283 | |revgets ||50000L || 0.69 | 72254 || 2.37 | 21132 || 0.75 | 67114 | |fprintf ||50000L || 5.84 | 8567 || 7.31 | 6838 || 5.58 | 8968 | |________||_______||________|________||________|_______||________|_______| TABLE 4. Stdio timings ____________________________________________________________________________ |______________________||_______________||________________||________________| | test ||count ||seconds | #/s ||seconds | #/s ||seconds | #/s | |______________||______||________|______||________|_______||________|_______| | open/close ||10000 || 2.88 | 3472 || 1.89 | 5291 || 0.45 | 22222 | |create/delete ||10000 || 42.90 | 233 || 12.77 | 783 || 3.11 | 3215 | | readdir-2 || 1000 || 9.62 | 1040 || 4.07 | 2457 || 2.80 | 3571 | | readdir-500 || 1000 || 42.41 | 236 || 34.34 | 291 || 45.17 | 221 | |______________||______||________|______||________|_______||________|_______| TABLE 5. Syscall timings _________________________________________________ |________________________________________________| |Null syscall us 4 3 | |Pipe latency ms 295 35 | |Pipe bandwidth MB/sec 23.33 39.37 | |File write bandwidth KB/sec 1995 2824 | |File read bandwidth MB/sec 33.33 44.18 | |Mmap read bandwidth MB/sec 75.00 86.33 | |Memory read bandwidth MB/sec 90.91 82.28 | |Memory write bandwidth MB/sec 76.92 83.26 | |________________________________________________| TABLE 6. Selected lmbench results REFERENCES 10. Richard A. Becker, John M. Chambers, and Alan R. Wilks, The New S Language : A Programming Environment 1. Practical Reusable UNIX for Data Analysis and Software, Edited by Graphics, Wadsworth & Balanchander Krishnamurthy, Brooks/Cole, New Jersey, John Wiley & Sons, 1995. 1988. 2. Microsoft Win32 11. Morris Bolsky and David Programmer's Reference, Korn, The New KornShell Volume 2 Microsoft Press, Command and Programming 1993. Language, Prentice Hall, 1995. 3. Matt Pietrek, Windows 95 System Programming Secrets, 12. Glenn S. Fowler, A Case for IDG Books, 1995. Make, Software - Practice and Experience, Vol. 20, 4. The X/Open Release 4 CAE No. S1, pp. 30-46, 1990. Specification, System Interfaces and Headers, 13. M. McKusik, W. Joy, S. Issue 4, Vol. 2, X/Open Leffler, and R. Farbry, A Co., Ltd., 1994. Fast File System for UNIX, ACM Transactions on 5. The OpenEdition MVS Users Computer Systems, Vol. 2, Guide,IBM,1996. No. 3, August, 1984, 181- 197. 6. M. Accetta et al., Mach: A New Kernel Foundation for 14. Kiem-Phong Vo, Vmalloc - A Unix Development, Usenix General and Efficient Association Proceedings, Memory Allocator, Software Summer 1986. - Practice and Experience, Vol. 26, No. 3, pp 357-374, 7. Jeffrey Richter, Advanced March 1996. Windows - The Developers Guide to the Win32 API for 15. Glenn S. Fowler, David G. Windows NT 3.5 and Windows Korn, John J. Snyder, and 95, Microsoft Press, 1995. Kiem-Phong Vo, Feature Based Portability, 8. POSIX - Part 1: System Proceedings of the USENIX Application Program Symposium on Very High Interface, IEEE Std Level Languages, 1994. 1003.1-1990, ISO/IEC 9945- 1,1990. 16. David Korn and Kiem-Phong Vo, SFIO - A Safe/Fast 9. POSIX - Part 2: Shell and String/File I/O, Utilities, IEEE Std Proceedings of the Summer 1003.2-1992, ISO/IEC 9945- Usenix, pp. 235-255, 1991. 2, IEEE, 1993. 17. Jeffrey Korn, Tksh: A Tcl Library for KornShell, Fourth Annual Tcl/Tk Workshop, Monterey, CA, July 1996, pp 149-159. 18. Kevin Lai and Marry Baker, A Performance Comparison of UNIX Operating Systems on the Pentium, Proceedings of the San Diego Usenix, pp. 265-278, 1996. 19. John H. Howard, Michael L. Kazar, Sherri G. Menees, David A. Nichols, M. Satyanarayanan, Robert N. Sidebotham, and Michael J. West, Scale and Performance in a Distributed File System, ACM Transactions on Computer Systems, Vol. 6, No. 1, Feb 1988 pp 51-81. 20. Larry McVoy and Carl Staelin, lmbench: Portable Tools for Performance Analysis, Proceedings of the San Diego Usenix, pp. 279-294, 1996. 21. Glenn Fowler, David Korn and Herman Rao, "n-DFS The Multiple Dimensional File System", Trends in Software - Configuration Management, pp. 135-154, 1994.