musings on embedded systems

by Rik Farrow
<rik@spirit.com>

Rik Farrow provides UNIX and Internet security consulting and training. He is the author of UNIX System Security and System Administrator's Guide to System V.

I began this adventure into embedded systems when I started reading the proceedings for the 1999 Workshop on Embedded Systems (available from USENIX, of course). I thought I knew a lot about embedded systems, since that is where I began working as a nonstudent programmer, but boy, have things changed.

In this column, I will amuse you with my own experiences with embedded systems, and then venture into some of the new regions embraced by the embedded-systems movement. I think some of them will surprise you (they did me). The implications are staggering for those of you who manage networks, own homes, use PDAs, have a refrigerator, or drive cars.

Did I say refrigerator? The second paper in the proceedings postulates a future where embedded devices are truly ubiquitous. It describes a SmartCan that includes "a tiny computer, a small amount of memory, and a short-range radio transceiver." If that seems at all far-fetched, remember that Geode by National Semiconductor provides the first two (as well as a video controller and DSL) on a chip, and there are two existing transceivers-on-a-chip. The paper does omit the power supply, but a later paper describes schemes for broadcasting power . . .

In the SmartCan scenario, the embedded system remembers the make-up of the can, making it easy to recycle. It remembers the contents of the can and when those contents were stored (so that it can let us know when it is no longer safe to use). You can query your kitchen from the store and see if you have a can of stewed tomatoes or olives, and, if it is sitting in the refrigerator, how long it has been sitting there. Of course, I just can't wait until we have refrigerators that will be "intelligent" enough to start nagging us about the molding box of carry-out Thai sitting "on shelf 0, quadrant 4, and about to ooze over the vegetable tender." If the refrigerator is truly intelligent, it will try to discern our moods first (or, even better, hire someone to come in and clean it out without bothering us).

Welcome to the wonder of embedded systems. They already are everywhere. For every Pentium manufactured by Intel, there are somewhere between 20 and 100 embedded processors installed every year. You know that your VCR has one, as well as your microwave. Your new car or truck (er, SUV) likely has more than ten. Even a new blender has one, and of course your cell phone has a very zippy one. But this is nothing. In the future, the embedded systems will all talk to one another.

At least that is the plan. The big fight will be for the standard that unites all of the embedded systems. If you thought that the browser wars were something, just wait. Microsoft wants its software to be embedded in everything, and is making progress in that direction. (I can imagine the new excuse — I was late because my car crashed and had to be rebooted.) Sun's Java was always targeted, not at the Internet (that was an accident), but at embedded systems. I'll have more to say about Java later.

Ancient History

When I was a young man, I found computers fascinating. What I did not find so fascinating was the heavily controlled access to them. In 1978, someone left the specifications for the Zilog Z80 processor at my house while he wandered cross-country. I had abandoned computer work for something more physically stimulating (no desks for me), but I had a real awakening as I paged through the Z80 specs. This was a real processor on a single chip!

After a bit of retooling at the University of Maryland, I quickly landed a job working for a company that was using the Z80 in an embedded system. The development system came from Zilog, and an average compile took about 15 minutes, giving me lots of time to read science fiction in between writing and testing.

The Z80 was an Intel 8080 clone with some additional instructions, for example, for block copies. This particular embedded system was designed for digitizing graphic data, for example, turning an aerial photograph into a set of labeled points. Here is how it worked.

The digitizer itself was a large table with a glass top. The underside of the glass had precisely aligned wires epoxied to it. The wires were attached to ground at one end, and to eight-bit shifters connected in series and a large power supply on the other end. A short pattern of ones gets shifted through the shifters, sending current down the wires, which generates a magnetic field. As the clock sent to the shifters is carefully regulated, this results in a magnetic field traveling across the digitizer's top, first for the x coordinate, then for the y.

To complete the setup, the operator must use a calculator-sized device to position a cursor over the point to digitize. As the magnetic field sweeps under the cursor, a coil detects the field, and the precise moment the field reaches a particular magnitude, the shifting stops and the number of clock ticks is recorded. By interpolation, the 1/4" grid of wires could measure to within three thousandths of an inch -- theoretically. In practice, outside influences, including the magnetic interaction of the cursor itself, affected the readings.

The Z80's part in this had to do with collecting the number of clock ticks and converting it into x and y coordinates. This could be presented in several formats, as well as scaled or offset, and streamed out a serial port for data collection. A later effort included using Zilog Parallel Input/Output (PIO) chip to build a driver for a magnetic-tape unit (which actually worked).

In a later adventure, I consulted as a tech writer for Morrow Designs, one of the manufacturers killed by PC clones. I would come in, read the circuit diagrams and the code used by the embedded processor, and write the documentation. I liked this work because it was easy to do (while still stretching my mind a bit), and I did a much better job writing docs than I did coding large projects.

George Morrow often used microprocessors in disk controllers. He designed a floppy-disk controller that could read and write six different formats and used a Z80 to do almost everything. In another design (not by George this time), an early, ten-instruction RISC processor was used in a hard-disk controller with a channel-style control architecture.

With this sort of background, it is easy to see that I had a skewed view of what embedded systems look like today.

Appliances

A different view of embedded systems comes from network appliances. One of the first was Auspex, with a multiprocessor system supporting NFS and using either a Sun or an IBM workstation as a controller. Or Network Appliance itself, with its own operating system dedicated to file serving.

Then, there are the network devices, such as Ascend routers (BSDi), Juniper routers (FreeBSD), Cisco NetCache (BSDi), Big Ip from F5 (BSDi), the Linux router project, and many more. The lure of rock-solid IP stacks and low cost (especially when compared to Microsoft), as well as high performance has made the BSDs and Linux the operating systems of choice when it comes to building network devices with some built-in smarts.

I have been asking around and hope that future issues of ;login: will contain descriptions of some of these systems and how they were built. Also, there is a very active Linux community working on embedded-systems issues. You can find the Embedded Linux Consortium through <http://www.linuxdevices.com/>, as well as another embedded Linux portal at <http://www.linux-embedded.com/>. Both BSDi and Red Hat offer embedded-systems developer kits. The FreeBSD home page <http://www.freebsd.org> has a link to PicoBSD, a version suitable for embedded systems.

I think that examining UNIX versions used in these implementations is very useful, as it makes it clearer how you can build a true bastion host — a system stripped down to its bare essentials. For example, booting most *nices takes the boot loader, the kernel, /etc/init, and /bin/sh. Everything else is optional, although it's hard to get much done with just the shell. For a router, you might want to add gated and some management software.

An interesting addition in the Linux world is the BusyBox, a set of tools maintained by Erik Andersen (<http://busybox.lineo.com/>) and sponsored by Lineo, a company developing a version of Linux especially for embedded systems. BusyBox consists of a shell and other tools useful in a constrained environment (or what I used to think of as an energy-repair diskette).

Other embedded systems include the much-maligned network computers. These diskless workstations are making somewhat of a resurgence. Personally, I would prefer to see nothing but diskless workstations on the desks sitting in office workers' cubicles. Take away the disk, and you take away most of the maintenance and administration problems, and do wonders for security. Of course, you move that to a central server, which, while much easier to administer, creates an attractive single point of failure. Also, the workers will be really pissed off when you take their games away from them.

Then, there is the really basic embedded system, such as WebTV. I got to set one up the other day (don't laugh). I had just gotten a massage, and the masseuse mentioned that she had just bought a WebTV but was worried that she couldn't set it up. Her problem wasn't the setup, but the array of cables needed to put it together. The masseuse had already set up her new VCR, connected the cable input, and got the TV working, so I wondered what was so difficult about the WebTV.

For me, it was nothing, just standard cables, power supplies, the phone cable, and a printer. Once everything was connected, we turned on the power and the system first dialed an 800 number, found the nearest point-of-presence, hung up and redialed a local number, and started stepping through the sign-up instructions. I left at that point, but called back later to see if she had been able to finish the sign-up and send email. Boy, was she excited. The system was really working for her, and even the printer worked (an addition to the system, and it just worked).

The Future

You probably see the problem. It wasn't the software, which guided my friend through unerringly. (I was impressed.) It was that mass of unfamiliar cables that had her stumped. Really, everything was simple, as the audio/video cables were color-coded, and the printer cable could only fit one way. Trivial, for an engineer.

But this business about connecting things together and configuring them so they will work together is exactly the thing that future embedded-systems designers want to avoid. The best-known example of this is what are called Smart Spaces. A Smart Space is a wireless island where available services and devices discover one another, do whatever self-configuration is necessary, and adjust to changes that occur in the environment, all without the intervention of a system administrator (or masseuse or her client).

This part really fascinates me. Imagine walking into a conference room, having your laptop discover the LCD projector, the printer, the local nameserver, and the gateway to the Internet, all without you doing anything at all. You make your presentation using the LCD projector, perhaps print a couple of pages, while having access to sites on the Internet without doing any configuration at all. I know that DHCP takes us part of the way there (automatic IP address assignment, along with the nameserver and gateway address), but what about talking to the projector and the printer?

Back in 1994, I had the opportunity to interview Bill Joy in Aspen. I was playing at being a print journalist, as the real journalist assigned to the interview hadn't been able to understand what Joy was trying to tell him. Joy actually can be crystal clear (see his April Wired article entitled "Why the Future Doesn't Need Us" <http://www.wired.com/wired/archive/8.04/joy.html>). Joy repeats a chilling view of the future of humanity when computers become smarter than humans — from the Unabomber's manifesto. Joy goes on to describe the dangers inherent in gene tailoring as well as nano-technology. A very thoughtful piece, and, given the current level of sanity in the world, it left me feeling rather gloomy about the not-too-distant future.

But back to embedded systems. In true paparazzi style, I had cornered Joy at a think-tank meeting (the Aspen Institute) and asked him if he would meet with me to complete the interview. We did get to meet in an outdoor cafe, where Joy tried to explain to me (without giving away Sun secrets) his view of the future — Java.

After being bored by my obvious questions about the beginnings of BSD, Joy got very intense, first when talking about future design of SPARC architectures. It was nice to be talking to someone who actually was creating designs for five or more years into the future. But the other thing he started talking about perplexed me at the time. Joy pulled out his cell phone. "There is no reason why I shouldn't be able to get this cell phone to work with a printer by just setting it next to it," said Joy. At the time, I thought, big deal, he's talking about infra-red networking. Of course, I was wrong. Joy was talking about Java, something it took me years to realize.

Joy's vision of the future had Sun software (and perhaps hardware) sitting in the middle of it. This vision has been somewhat derailed over time, but I believe Java, and the set of specifications for embedded systems, Jini, is very real. Again, you can read Wired (<http://www.wired.com/wired/archive/6.08/jini.html>) or visit Sun's own site (<http://sun.com/jini/>).

Before getting into Jini, I'd like to back off a bit, and talk about the requirements for Smart Spaces. Ideally, in a Smart Space world, there are no cables. Everything has its own networking, and just bringing devices into proximity is sufficient for configuring the devices. Thus, if the WebTV I installed had been Smart Space—ready, all it would have taken to get it working would have been to turn it on.

I must pause at this point, because the thought of all this wireless self-configuration going on gives the security person in me the heebee geebees. Given the software industry's track record for designing secure software (zip), this whole idea scares me silly. If I don't have to connect the keyboard, how do I know which server I am typing to (or how many, because I am truly paranoid)? Cables provide a warm-fuzziness, as I know that the keyboard is physically connected to that workstation, and that if I have protected the X-server properly, I don't have to worry about my keypresses winding up somewhere else.

Perhaps I am being a bit extreme when I write that no secure software has ever been written. There was hello.c, and lots of work has been done to make sendmail secure, and to get the bugs out of SSH. The other factor that frightens me is the market's willingness to accept and use insecure solutions. Today the firewalls of choice specialize in performance, not security. E-commerce sites will initially dispense with having any firewall at all, because getting up and running is all-important prior to the IPO. And, of course, firewalls are not all there is to security — they just cover one door into the organization. Sigh.

Back to Smart Spaces. Devices in a Smart Space are supposed to identify themselves to the other devices and to be able to find what they need. For example, imagine you are sitting in the terminal room at a USENIX conference, and you want to print something. If it were a Smart Space, you would just print, trusting your operating system or application to locate a nearby printer, load the correct device driver and filters for it, and queue up your print job. And if, the next time you go to print, that printer is no longer available (perhaps it is out of paper), the Smart Space specs will not let you down, but will seamlessly find another printer and go through the self-setup again, without intervention.

Suppose the entire conference center is a Smart Space. Forget about the terminal room, as wherever you go, you will be connected to a network within the conference center, which will provide you with a nameserver and gateway to the Internet. As you roam, your IP address will change, but your connections will persist. You may have read about these issues in USENIX papers about mobile computing, but they are part of today's embedded-systems world as well.

There is also the notion of application dependence. This may not be a problem if everyone uses Microsoft applications and keeps them updated to the most recent version. Gag. More realistically, suppose the person you are meeting with wants to transfer his or her new opus, but it is written in Word, and all you have is a PDA. If the Smart Space is really smart, it will transparently handle converting whatever application-specific format the document is in into something your system can handle.

That seems to be asking a lot from Smart Spaces (perhaps they should be called "Genius Spaces" instead). Kevin Mills, in his paper, goes further when he talks about having data that is tied to a particular context, for example, a committee meeting. Whenever and wherever that committee meets, its minutes and other appropriate information remain accessible. Now that is really out there. But so were 10-gigabyte hard drives for under $200.

Jini

Some of these problems have already been successfully attacked through Java Jini, a set of specifications and interfaces for discovery and sharing of networked services. O'Reilly & Associates had kindly sent me a copy of Jini in a Nutshell, by Scott Oaks and Henry Wong. I had asked if the authors wanted to write this part, and got the book instead. So here goes. We are all most certainly aware that any Java Virtual Machine (JVM) should be able to run any compliant Java code. This concept is one of the founding ideas behind Java — write once and run anywhere. But it will take more than that to make Smart Spaces work.

And this is where the Jini specifications come in. The specs define interfaces that must be supported for Jini-enabled services. The goal behind these interfaces is that services will be able to interact in a dynamic fashion without human interaction. No setup, administration, or guidance. These features can be summed up as follows (quoting page 4 of the book):

• The Jini environment requires no user intervention when services are brought online or offline (other than starting the service, e.g., turning on the Jini-enabled device or starting the Jini software service).
• The Jini community is self-healing: it can adapt when services (and consumers of services) come and go.
• Consumers of Jini services do not need prior knowledge of the services implementation. Instead, the consumer loads the service implementation dynamically, with no configuration or user-intervention required.

The book goes on to explain how to download the free developer's kit, which requires version 2 of the Java Developer's Kit (JDK 2). Sun provides a registry service (reggie), a transaction service (mahalo), temporary storage for objects in JavaSpaces (outrigger), a lookup service (fiddler), an event mailbox (mercury), and a lease-renewal service (norm). When I encountered these names, I wondered if the Jini team had holed up in Hawaii instead of Mountain View or Aspen.

The authors go on to point out that Jini does have support for security, something not available with Microsoft CE implementations. (Sorry, digital signatures, as found in DCOM, are not sufficient protection.) Jini security mechanisms are identical to JDK 2 and permit access control to all system resources based on the classpath, codebase, and/or digital signature. Chapter 12 of the book goes into a little detail about security. For more details, you must read the book Java Security, written by Scott Oaks and published by O'Reilly in 1998. Not all of the over 400 pages have to do with access control; there are also chapters on encryption, message digest, and other topics. What pleased me is that Jini does have support for fine-grained security.

The authors go on to describe installing, setting up, and starting host-based Jini services. Once you get all of this going, you can begin writing your own Jini services, using the examples found in the book. I am no longer proficient in Java, but the book appears to be thorough. If you have had good or bad experiences using this book, send me some email, and I will share it.

Aroma

A related project mentioned in the proceedings was AirJava, renamed "Aroma" after Sun complained. Kevin Mills of NIST plans to create a working pico-cellular network-based test platform in the near future. The goal would be to create a functioning testbed that included a processor, the network interface, some memory, and perhaps a PC card interface for those who wanted to get some practical experience with Jini. See my interview with Mills and Dima in this issue of ;login:.

Looking again at the proceedings, there are papers about massively distributed systems (just imagine your car's ignition system processor participating in the search for extraterrestrial intelligence [SETI] while you are stuck in traffic), learning algorithms (intelligent embedded systems with pattern recognition), and virtual user interfaces (a method for creating a user interface for anything your PDA finds in the pico-cellular network neighborhood). I particularly liked this last one (Kangas and Röning <macconen@ee.oulu.fi> and <jjr@ee.oulu.fi>, as they explicitly mention the "limited functionality" of VCR and microwave-oven displays. They also point out the limitations in input as well, and include cell phones in their lists of devices.

There are other systems I have not mentioned here. QNX, a realtime OS, can be used in embedded systems, and Inferno will be used in telco equipment (unless Lucent goes out of business, which I really doubt). Microsoft is not going to rest while this goes on. (It does own WebTV, and I will confess that the user interface passed the masseuse test.)

I have always wanted to live in an intelligent house and drive an intelligent car, and am somewhat disappointed at the slow pace of change. Using X10 controllers to turn lights off and on is not very exciting to me, but having access to information wherever there is a telephone in my house is. Why should I have to find the PDA or the Rolodex just to call someone, or to add toilet paper to the shopping list? Why doesn't my car have a nice GPS in it (because it would be stolen?), as well as a method for self-diagnosis that doesn't involve coded beeps or display lights flashing?

We are living on the cusp of a new age. I decided to write about this because it excites me, but also because I believe it will affect the USENIX community before it affects the general public. That is because we manage a lot of the computing and networking infrastructure in the world today, and embedded systems may well add to our load.

And finally, I want embedded systems to be based on open standards with good designs. These standards must include support for security and for extensibility, and not stifle innovation through complexity or license fees. Examining history shows us that the standards are most commonly based upon whatever is widely used (and you wonder why I avoid using Microsoft products?). The process for creating the standards for the future of embedded systems is happening now, and it is important that we be at least aware of it.

Oops, gotta go, my refrigerator just opened a window on my screen telling me that the ale I put in it has reached 10 degrees C. If you do want to hear more about embedded systems, let me know.