Musings on Operating Systems

Unix, VMS, Windows NT, and the end of POSIX

March 5, 2023

Musings

Authors:

Article shepherded by:

Rik Farrow, Laura Nolan

Interviewing the founder of The Unix Heritage Society got me to thinking about Unix, as well as Linux. Why is Linux so popular in clouds and on servers, while Windows rules the office? The early days of both of these systems have much to do with how they are used today.

But first, the lineup of new articles.

The Lineup

Peter Sperl presents two system design patterns for implementing resilient and layered caching in distributed systems. Sperl's article helped me think differently about distributed systems design, with implications not only for resilience but also for security.

Michał Czapiński and Rainer Wolafka describe methods for securely and efficiently separating containers based on workload security rings and their level of sensitivity or hardening. This article nicely complements the layered caching one by Sperl.

Titus Winters suggests that developers include more alerting in early versions of software. While a lot of these log messages may turn out to be unnecessary, having them present early in the development process helps in choosing what should be alerts, and what doesn't need to be included.

Thai Wood considers the way that Australian rock fishers think about safety with how we handle incident analysis. I found the analogy between rock fishers and incident analysis right on.

Vartika Agarwal and Tracy Ferrell explain the SRE reliability maturity model, with examples that help you determine where your organization fits in this scheme, and why this is important.

Laura Nolan discuses the humble health check, why it is important to orchestration processes in complex modern systems, as well as its pitfalls:.

Eleven researchers write about their experiences uncovering fail-slow storage devices, describe their solutions, and make drive performance data available for other researchers. I particularly liked the way they discuss the various methods to discover fail-slow devices that failed, before describing the successful approach.

I interviewed Matt Welsh about using Rust in startups and the future of replacing programmers with AI/ML.

I also interviewed Warren Toomey, who was awarded the 2023 USENIX Flame Award for starting and maintaining The Unix Heritage Society, about how he became interested in Unix.

I reviewed the book Space Rogue: How Hackers Known as L0pht Changed the World, partly memoir and largely the story of the hacker scene in Boston in the 80s and 90s. I was working in security during the time that Space Rogue (Chris Thomas) covers, but his view of things was very different than mine was. An interesting and easy read.

Tom Limoncelli reviewed Trailblazer, the autobiography of Mary Ann Horton, member of the original BSD team and transgender activist.

Richard L. Sites tells about having Fred Brooks, best known for the Mythical Man-Month, as well as the design of System/360, as his PhD advisor.

Yongle Zhang et al describe a system for uncovering fault locations in distributed Java applications: .

The Unix System

I had heard that the Unix system was born so that Ken Thompson and Dennis Ritchies could play a game they had written: Space Travel. And it turns out that is true. Ritchie had published a paper in Australia [1] where he discussed the design decisions that resulted in the basics of systems still in use today.

Ritchie's paper doesn't include the specs for the first platform for Unix, a PDP-7. In a video interview with Brian Kernighan, Ken Thompson provides lots of Unix history, including that the PDP-7 had 8K of 18 bit words, and he ran Unix and programs in 4K memory each. Ritchie described the PDP-11 that was the second platform, and I am amazed. I built a computer from a kit in 1979 that was about as fast, although without the .5 megabyte hard drive, console, and two DEC tape drives. My system had 370 kilobytes of floppy disk and more memory, at 56 kilobytes compared to the 24 kilobytes in the PDP-11. Thompson and Ritchie used 16 kb for the operating system, all written in assembler, and 8 kb for the shell and applications. In order to justify having Bell Labs buy the PDP-11, a bargain at $65,000 (about $500,000 in 2023 US dollars), they wrote a version of runoff (groff is its descendent), typesetting software, for use in inputting, formatting and printing patent applications. Thompson's interview has more details about how they convinced Bell Labs to buy the mini-computer and allow them to work on an operating system.

I hope you are getting a feeling for the constraints the first Unix programmers had to work with. Ritchie's paper, written in a very conversational style and only about 15 pages long, is worthwhile reading for anyone interested in writing an operating system. The early versions didn't have exec() or wait(), and I found the side effect of adding fork() on the chdir (cd) command interesting, as well as having to overwrite the shell to load an application.

Thompson and Ritchie had been working on MULTICS (multicians.org), a mainframe operating system for a GE (later Honeywell) computer, and borrowed some of the features of that system, albeit in a very scaled-down form. For example, instead of access control lists, a permissions system first used in MULTICS, the Unix file system had read|write|execute permissions of owner|group|other. While current versions of Unix and Linux support ACLs, they are rarely used (in my experience). User and group ids were 16 bit integers, and the permission bits, which include the file type, also fit into 16 bits.

Windows

Windows started out as an interface for MS/DOS, a primitive operating system similar in size to early Unix. Like early Unix, Windows had no memory management and any program could overwrite the OS. Unlike Unix on the PDP-11, there were no users, as MS/DOS and Windows were strickly for the use of one user at-a-time.

The current version of Windows was created during the 1990s, and called Windows NT. Microsoft needed to solve several problems that had cropped up for Microsoft's attempt at world domination:

Unix workstations dominated the desktop graphics market
The majority of Internet services were provided by Unix systems, with DEC's systems providing a small minority of Internet servers
Novell Netware provided support for networking of offices, including shared storage and printing, as well as central management of user accounts

While Microsoft office applications dominated the desktop, having to share the office market with Novell was certainly annoying. And Microsoft wasn't even part of the Internet revolution, with no networking stack. Besides the office apps, Microsoft was best known as the superior games platform, something that made Windows more popular on the desktop but didn't make Microsoft much money.

Dave Cutler had been working on a new operating system for one of three RISC-based systems that DEC was working on [2] during this time. DEC management cancelled his project, and Cutler moved over to Microsoft to lead the design of Windows NT. During his career at DEC, Cutler had been a lead designer of several operating systems, including VMS, starting in 1975.

So it shouldn't surprise anyone that Cutler designed Windows NT as the successor to VMS. Where simplicity was the watchword for the Unix system, complexity and control were the guidelines for VMS.

As an example, consider how Windows NT handles user and group identification. Where Unix initially used 16 bit (currently 32 bit) integers, NT uses Security Identifiers (SIDs), minimally 96 bits, containing five different fields [3]. While there are some local SIDs that are the same on all systems, most SIDs are globally unique by design. SIDs get combined into an access token that provides the basis for a wide variety of user and system privileges.

From the perspective of Unix programmers, SIDs appeared to be massive overkill. But for a desktop OS vendor, especially one aiming to compete with Novell, SIDs make perfect sense. A universally unique user id means that there would never be user id collisions, unlike Unix systems, where not only was this a problem, but file sharing using NFS didn't even include user ids, but passed usernames as identification between systems. It wasn't until NFSv4 that support for stronger authentication appeared.

Active Directory appeared as part of Windows 2000, allowing Microsoft to compete directly against Novell with its support of user authentication using LDAP, an Internet standard for decades. Server Message Block (SMB) provided file and printer sharing. Novell lost the war for office desktops.

And Unix workstations were toppled from their domination in graphics. Workstations were very expensive, and the ones with superior graphics even more so. A 25 Mhz Sparcstation [4] IPC, with 24 MBs of DRAM and 10 MBs of disk sold for $6000 in 1990 — with the 40% developer's discount. I bought one. At the time, systems like these ran circles around PCs.

Within five years, Intel CPUs in desktop systems had an effective 66 Mhz clock speed, with noticeably better performance, and a comparable GPU, and cost around $2000. Around 1990, Silicon Graphics owned the 3D graphics market, with massively parallel systems, with each pipeline processor contained in a single chip. A high end graphics system would have five boards in it and cost hundreds of thousands of dollars. By 2000, desktop graphic cards with excellent gaming performance cost less than $500, and the era of Unix workstations was mostly over.

Servers

Windows NT servers soon became popular on the Internet for email and web services. But there were problems with NT as a server right from the start.

Ted Ts'o once remarked to me, during a Linux File System workshop, that a new file system, at that time BtrFS, would take five to ten years to mature enough to be used in production. Using that as a metric, using Windows NT, a new operating system with a new file system, for production servers in the early noughts was a terrible idea. That time period was full of amazing exploits against Windows servers, and even without being attacked, they generally either crashed or needed to be rebooted every two weeks.

But there's a much bigger issue than simple maturity. Several decades after its creation, Windows systems have become much more reliable, and less easily exploitable. Like Unix, Windows can run on relatively tiny hardware systems up to hefty servers. Unlike Linux, you don't find Windows on supercomputers, and most cloud servers run Linux.

After all, why do you need the complexity, things like SIDs, on servers that are running VMs, containers, and executing lambdas/functions? What you want instead is the smallest OS footprint possible. In Marc Brooker's presentation of the NSDI'20 paper about AWS Firecracker [5], the combined VMM and minimal Linux kernel take up only five MBs. AWS fits over 3000 instances of Firecracker VMMs with Linux in a single server running Linux and KVM.

POSIX

During the 1980s, with the birth of many different versions of Unix, there was a need to develop a standard for the programming interface, and this came to be known as POSIX. Microsoft built a POSIX subsystem that could run as part of Windows NT, so they could compete against Unix vendors for US government proposals that required the POSIX interface [6].

POSIX was developed to support non-graphical applications and during a time when memory sizes were small and CPU performance was still limited. Two decades later, I found a paper where the authors had written that POSIX was outdated. They analyzed system call usage and found that system calls relating to memory and interprocess communication were the most common [7]. In another paper at the same EuroSys conference, authors analyzed a large number of Linux applications in attempt to see how much of the system call interface was actually in use [8].

More recently, a group of CS researcher wrote the most popular ;login: article of 2022 in terms of downloads, Transcending POSIX [9], resulting in thousands of downloads a week, sometimes over one thousand downloads in a single day. With all of this interest, perhaps people will actually do something about replacing POSIX.

Or do we need to do that? POSIX was created so that porting Unix applications between different versions of Unix was relatively easy. Porting the graphics portion wasn't helped by POSIX, at all. Yet, given the most common current use of Linux, that of running in clouds, graphics is not important. Graphics is important to desktop users, and today there are just two popular versions of Unix on the desktop: Linux and the BSDs.

The Transcending POSIX article's authors cover other aspects of POSIX that are outdated. The POSIX model for threading includes assumptions from the 1990s, such as a single CPU core. And various coprocessors and accelerators, such as GPUs and NICs that can offload packet processing, didn't exist, and are not covered by POSIX. POSIX style file IO is terribly outdated, with workarounds like eBPF, DIO, and DPDK.

And we still have an application portability problem: cloud and serverless platforms have different, incompatible APIs. Part of this is from the creation of these new platforms, and part of the problem is vendor lock-in. We shouldn't be surprised by this, for as long as there have been computer vendors, those vendors have done their best to keep their customers captured, and make moving to a competitor difficult.

So will we ever have COSIX: Cloud Operating System Interface X? I do think that Linux will continue to dominate cloud providers for the present, for the very reason Marc Brooker mentioned in his Firecracker paper [5] presentation: it's an operating system that is familiar to many. The most common design pattern in large server platforms today includes distributed services, something that supports fast updates, scalability, security, and reliability. I don't see that changing any time soon. Yet, who among us who was working in CS in 2000 could have foreseen where we are today?

Ken Thompson began writing Unix because he had a game he liked to play, and a PDP-7 that would work but needed an operating system. That very minimal operating system soon supported a business application and multiple users. At the end of the '70s, virtual memory support was added to a version running on DEC's PDP11-780, and that version of Unix, BSD, soon included the version of the TCP/IP stack that other vendors had to interoperate with. By 1992, all the files that had started as being part of AT&T Unix had been replaced, and BSD Unix became the first open source version of Unix.

If we consider what Ted Ts'o said about file systems to be true, then an operating system created today will not be usable in production for five to ten years. Perhaps that system already exists and we just haven't heard about it. If so, I am guessing that it will need to follow in the steps of the Unix creators, in that the new operating system is simple, with just enough of a kernel to support distributed applications, and is shared from the start.

Appendix

References:

[1] Ritchie, D. The Evolution of the Unix Time-sharing System, Language Design and Programming Methodology conference at Sydney, Australia, September 1979: https://www.bell-labs.com/usr/dmr/www/hist.html

[2] Dave Cutler: https://en.wikipedia.org/wiki/Dave_Cutler

[3] Security Indentifiers: https://learn.microsoft.com/en-us/windows-server/identity/ad-ds/manage/u...

[4] Sparcstation IPC: https://en.wikipedia.org/wiki/SPARCstation

[5] Agache, A., Brooker, M., Florescu, A., Iordache, A., Liguori, A., Neugebauer, R., Piwonka, P. and Popa, D-M., Firecracker: Lightweight Virtualization for Serverless Applications, USENIX NSDI'20: https://www.usenix.org/conference/nsdi20/presentation/agache

[6] POSIX: https://en.wikipedia.org/wiki/POSIX

[7] Atlidakis, V., Andrus, J., Geambasu, R., Mitropoulos, D., Nieh, J., POSIX Has Become Outdated, ;login: Fall 2016, Vol. 41, No. 3: https://www.usenix.org/system/files/login/articles/login_fall16_02_atlid...

[8] Tsai, T., Jain, B., Abdul, N., and Porter, D., What to Support When You’re Supporting: A Study of Linux API Usage and Compatibility, ;login: Fall 2016, Vol 41, No 3: https://www.usenix.org/system/files/login/articles/login_fall16_03_tsai.pdf

[9] Enberg, P., Rao, A., Crowcroft, J, and Tarkoma, S., Transcending POSIX: The End of an Era?: https://www.usenix.org/publications/loginonline/transcending-posix-end-era

[10] Video interview by Brian Kernighan of Ken Thompson at VCF East in 2019: https://www.youtube.com/watch?v=EY6q5dv_B-o. Thompson starts talking 17 minutes in.

Article Categories:

Distributed systems

Operating Systems

Cloud

Linux

Culture

Last updated March 31, 2023

Authors:

Rik Farrow has been a consultant for 42 years. He has written two books, as well as worked as the technical editor for a UNIX magazine and for two editions of a popular operating system book. He also taught UNIX system administration and Internet security during the 90s internationally, and worked as a volunteer for USENIX program and steering committees. Rik has been the editor of ;login: since 2005.

[email protected]