The following paper was originally published in the Proceedings of the USENIX Conference on Object-Oriented Technologies (COOTS) Monterey, California, June 1995 For more information about USENIX Association contact: 1. Phone: 510 528-8649 2. FAX: 510 548-5738 3. Email: office@usenix.org 4. WWW URL: https://www.usenix.org Using meta-ob jects to support optimisation in the Apertos operating system Jun-ichiro Itoh. Keio University, Kanagawa, Japan. Rodger Lea. and Yasuhiko Yokote. Sony CSL, Tokyo, Japan. Abstract The Apertos OS has explored the use of the meta-object model and reflection as a means to build highly flexible operat- ing systems. While the benefits of such a system are great, the performance cost of a clean consistent use of the meta ob- ject model is high. Our initial work ac- cepted this because of our desire to explore the model to its fullest. Recently we have turned our attention to optimisation tech- niques. Our work is interesting because we have tried to use the flexibility of the model to support our optimisations. This is in contrast to many systems that restrict or even compromise their initial model for the sake of performance. This paper outlines the Apertos system and discusses the op- timisations we have made. Introduction Recent trends in networking, multi-media and mobile computing have placed new requirements on the flexi- bility and adaptability of existing operating systems. It is increasingly recognised that existing OS architectures are not sufficiently flexible to meet these demands. As such there is significant interest in building more flexible, or adaptive operating systems. However, this flexibility often has a performance price and OS designers are faced ______________________________________ itojun@mt.cs.keio.ac.jp with a trade off between flexibility and performance. In- evitable, performance becomes the overriding factor, and the clean conceptual model originally proposed to achieve flexibility is compromised. In this paper we describe our work with the Aper- tos operating system which is based on the meta-object model. We outline the design and overall implementa- tion of the system and show how we have explored the meta-object model to its fullest. In particular we discuss how we have used the meta-object model consistently throughout the entire system. Our initial implementa- tion suffered from this desire for a clean model by hav- ing poor performance. Recently we have been exploring how to optimise the system. Our work differs from much other optimisation exercises in that we are endeavouring to exploit our model to support optimisation, rather than breaking the model. We feel that our initial experience is promising, and that the techniques we are adopting will be beneficial to other groups who are exploring the ob- ject oriented paradigm for the construction of operating systems. This paper begins with an overview of the different approaches to system flexibility; it then briefly describes the object/meta-object model; we present an overview of our implementation of this model in the Apertos system; finally we discuss our current work to optimise our system within the framework of the object/meta-object model. OS flexibility We can categorise flexibility in work to date as follows: flexibility in the way we combine the components of the system at build time, flexibility in the way the systems adapts to application needs and flexibility in the way the system and its interfaces evolve over time. Flexibility though modularity: Modularity is a key technique in building flexible software since it encapsu- lates components and makes them easier to change or evolve. Micro-kernel architectures have popularised such modularity and have demonstrated benefits in their over- all flexibility[3 ][2 ]. Object oriented operating systems have extended this modularity with a finer degree of gran- ularity and used techniques such as abstract interfaces and inheritance hierarchies to support a high degree of flexibility[10 ]. Systems such as Choices[6 ] have exploited this software engineering model to support an extremely sophisticated mix and match approach to the initial build phase, in some cases exploiting it to specialise systems not only for hardware but for the intended applications. Flexibility through run-time adaptation: Most operating systems have supported some degree of run time adaptation; for example scheduling policies are of- ten adapted to deal with changing application loads. Re- cently there has been much interest in the use of more sophisticated techniques to not only choose between ex- isting policy modules, but also to dynamically generate code that is optimised to a set of run-time constraints [13 ] [4 ]. Flexibility of interface evolution: Flexibility to support the evolution of system interfaces is more dif- ficult to achieve. Some of the work on object oriented systems has attempted to exploit type conformance [5 ] or similar means [7 ] to achieve it. It is clear that the operating system community is aware of and is actively addressing the issues of flexibility. We argue however, that although this work is moving in the right direction, it is ad hoc in that is opens up pieces of a system without an overall architectural model for making that flexibility intrinsic and so available to all OS components. Reflection and its use by Apertos The Apertos system (formally Muse)[15 ][16 ] has been in- vestigating a flexible open OS architecture since 1988 and has concentrated its efforts on the meta-level program- ming model. Simply put, a meta-level architecture is one in which all system components have a meta-level component that holds information about that component and how it is used; in essence the meta-level defines control and pol- icy. This meta-level component can be viewed as an en- vironment for the component and provides an interface to manipulate aspects of the component's environment. This is traditionally called the meta-interface or meta- object protocol (MOP)[9 ]. However it could equally be called its management interface. For example, a thread in an object oriented operat- ing system is represented by a simple object that has a 'standard' interface such as start and yield. In addition, somewhere in the kernel is the scheduling code which de- cides which thread to execute next. This algorithm is often referred to as the policy associated with the sim- ple object. In our terms this algorithm is part of its meta-object. In a traditional OS the implementation of the policy and its application to the thread was a closed a-priori decision made by the OS designers. In a more flexible system we would like to be able to open up that algorithm, make it accessible to other parts of the system and perhaps even change it. By providing an interface to the scheduler meta-object, a MOP, we are able to ex- pose that algorithm and allow system designers and even applications to use it. Reflection can be viewed as a use of this basic founda- tion. Reflection is the process in which we step back from computation that is on behalf of, or part of the applica- tion and carry out computation on behalf of the system itself. To achieve this it is necessary to be able to access a description of the system. When we have finished this computation we will have changed the description of the system. These changes are then 'reflected' back to the actual system. Hence reflection gives us a way to cleanly make changes to the system. Using the example above, when we use the MOP interface to make a scheduling de- cision, e.g. re-order the scheduler queue, we are making a reflective computation at the meta-level. It is clear that this ability to access and manipulate the basic features of an OS is useful; some of the work cited above is using a variety of different techniques to do just that. What is different about the Apertos system is that the entire operating system is built using this meta- object architecture and so all aspects of the system are open and can be easily changed. It is our belief that only by doing this in a completely consistent manner will we be able to fully realise the benefits of the flexibility it provides. The Apertos operating system As mentioned above, Apertos is an investigation into how we can apply this meta-object model to all aspects of the operating system and what are the costs and drawbacks. The Apertos system generalises some of the ideas men- tioned above. Each simple object in Apertos is managed by a set of meta-objects. The meta-objects are associated with the simple object via a descriptor object that holds a reference to all the required meta-objects. This set is referred to as a meta-space which is also represented by an object (which we call a reflector). The meta-space constitutes the execution environment for an object. At any point in time there will be many meta-spaces co- existing in the system. Interaction between objects is achieved through Meta- [No figure included in ASCII version] Figure 1: The object/meta-object relationship and the reflector hierarchy Core, a minimal system component that allows an object to call into its meta-space. MetaCore can be likened to a micro kernel, however rather than implementing a simple set of base abstractions, it implements only the means to move from a simple object into its meta-space. An object that wishes to access another object does so by calling into its meta-space; the target object is lo- cated and the invocation dispatched. In the cases where an object wishes to compute not at the base level, but at the meta-level, the same invocation model is used. How- ever the target for the invocation is a meta-object and not a base object. This common invocation path implies that all aspects of the invocation are also open to change including look-up, dispatch and invocation semantics. The last extension that Apertos supports is a means of allowing objects to change their behaviour not by using the MOP to modify some feature of their existing envi- ronment, but by migrating themselves to a completely new environment which supports a different set of poli- cies. Apertos supports a simple form of type conformance to allow this. Implementation In the previous sections we outlined the conceptual frame- work that Apertos supports, i.e. the object/meta-object model, and discussed how the Apertos system conforms to this conceptual framework. In this section we discuss in more detail how the actual implementation of these concepts is carried out. In fig.2 the abstract meta-architecture shown in fig.1 is visible as the separate meta-spaces mCOOP , mZero and mCore . [No figure included in ASCII version] Figure 2: Implementation hierarchy of meta-spaces MetaCore MetaCore acts in a manner similar to that of a micro- kernel in more traditional systems. However, rather than providing a minimal set of resource abstractions, its role is restricted to supporting the object/meta-object model, the base notion of activity and interrupt dispatching. This allows a very small core object (currently around 3K) which aids both understanding and porting. Meta- Core supports the following operations: ffl M(meta): Make a request for meta-computing. This causes the execution of an object to be suspended and control to pass from the object to its meta- object. ffl R(reflect): Resume object execution, i.e. pass con- trol from the meta-object back to a base object. ffl CBind: Specify a recipient for an interrupt. Inter- rupts will be delivered to the recipient as messages. ffl CUnbind: Break the association between a recipi- ent and an interrupt. MetaCore is shown in fig.2 as the vertical box running alongside the meta hierarchy. Since MetaCore supports the basic facilities for meta-computing, i.e. the reify(M) and deify(R) operations, it is accessible and used by all meta-spaces. In table 1 we give performance figures for these basic operations1 . Note that operations without trap occur when we do not cross a protection boundary. Table 1: Execution cost of MetaCore primitives (in sec) ____________________________________________ ______primitive____________on_i486__________ _ M _ 21.1 _ _ M(w/o trap) _ 13.0 _ _ R _ 22.6 _ _ R(w/o trap) _ 8.8 _ _ CBind _ 4.1 _ ___CUnbind___________________________3.5_____ ______________________________________ 1All performance figures from a 486DX2-66MHz, 16MByte PCAT compatible. The key meta-spaces mCOOP is the first level meta-space for a base level object, it implements the scheduling policy and provides a concurrent object programming model for base objects. mZero is one of the terminal meta-spaces for all other meta-spaces and is thus accessed by all other meta-spaces. This allows some operations, eg. context location. to be optimised. mCore is the reflector for mZero and contains meta- objects to handle activity, physical memory and virtual memory. Although, as mentioned above, mZero is the shared reflectors for all others, it too needs a reflector, hence the use of mCore . To halt to potentially infi- nite hierarchy of meta-spaces we make the meta-space for mCore be mZero . Although this introduces a circular dependency between the two it allows us to 'bottom out' when moving down through the meta hierarchy. Activity The underlying CPU is abstracted in a structure known as a context. Contexts2 are created by the meta-object exec which resides in mCore . Contexts are actually ab- stracted by a notion of Activity which is the unit of manipulation at the meta-level (see fig.3). The context structure is seen by MetaCore as it uses this to set up the hardware to support the actual execution of an activity. The context structure contains the pointer to the meta- space for that context. This enables MetaCore to move from the currently executing context to its meta-space when the M operation is called. [No figure included in ASCII version] Figure 3: Activity and the execution path to exec metaobject Optimisations Our initial work on optimisation has concentrated on its use in the lowest levels of the OS, i.e. the hardware inter- ______________________________________ 2Contexts in Apertos should not be confused which the general OS concept of a context as an address space. They are simply the state of the machine registers face. We begin this discussion by outlining some of the details of the driver metaspaces. mDrive and mSystem are meta-spaces for device driver programming (see fig.2). They implement a con- current object environment where device drivers are writ- ten in the same way as any other concurrent object appli- cation, i.e. we are able to hide the restrictions normally visible to device driver programmers. mDrive supports device driver objects that have direct physical access to hardware and manage interrupts, and mSystem supports device driver objects that have no access to hardware. We implement device drivers as single-threaded con- current objects. The advantage of this is that we can completely hide any complex mutual exclusion operation normally associated with device driver programming. In- stead the details are managed automatically by the meta- objects in the device driver meta-space. Device driver objects are programmed without recourse to shared data structure and communication between objects is carried out by passing messages. Service entry points and in- terrupt handlers are implemented as methods of device driver objects which are invoked on message reception. In this way we can provide a safe programming environ- ment for programmers who write very low-level system programs. One complication when writing device driver objects is that we need to have a way to synchronize interrupt handler methods and hardware manipulation methods so that we can cleanly switch between hardware manipu- lation and high priority interrupts. We achieve this in an object-oriented fashion by using continuation meta- objects. Continuations perform a similar role to 'futures' as used in languages such as Actor[1 ] and ConcurrentSmalltalk[14 ]. However we use them for synchronizing methods of an object, not to synchronize between two objects. Table 2 gives some brief details of execution costs for interrupt handling and using concurrent objects in the driver meta-spaces. Details can be found in [8 ]. As can be seen our approach has been to maintain the concurrent object oriented model supported through- out Apertos even at the device driver level. We believe that this approach provides a cleaner way to write de- vice drivers, which has always been something of a black art, and hides some of the inherent complexity within the meta-space mechanisms. Table 2: Costs of mDrive services, and interrupt opera- tions (in sec) metaoperation Apertos BSD/386 _____ Interrupt message delivery _ 25.0 _ 11.5 _ _____ Null interrupt handler execution _ 44.2 _ 16.2 _ _____ Send metacall overhead on mDrive _ 108.6 _ _ _ Context hand-off Because our system uses the object/meta-object model throughout, the general case execution path constantly moves from base to meta. As discussed above, we are us- ing a concurrent object model so that each object (includ- ing meta-objects) has it's own execution context. Thus our system is forced to constantly switch contexts as it traverses the meta-space hierarchy. Obviously our first target for improvement is in the implementation of this context switching. We have implemented a variant of thread hand-off which we refer to as context hand-off. Context hand-off will occur when an object issues a synchronous (Call-and-Reply style) method call to an- other object. In our basic implementation, there will be at least 4 context switches3 when issuing synchronous method call, i.e. switch to reflector for call, go up to callee, switch to reflector for reply, go back to caller. If it is safe to share the execution context the context hand- off mechanism will use a simple function call for the syn- chronous method call. This means that the caller's con- text will be used for executing the callee's method. The following conditions should be satisfied to use con- text hand-off mechanism: ffl caller and callee share the same address space, and are in the same protection boundary. ffl various state information, such as interrupt masks or CPU execution mode, kept in the execution con- text are the same. ffl it is acceptable to skip the scheduler when calling into the reflector. If the timeslice for the caller has expired, or will soon expire, then skipping the scheduler is not acceptable. ______________________________________ 3It is important to bear in mind that we use the term context switch here to mean a lightweight switch between two concurrent objects, not as a heavier weight notion consiting of a complete address space switch. It is the job of the reflector to decide if we are to use the context hand-off mechanism or not. It manages various state values on behalf of base-level objects and monitors behaviors (every external event from an object will go through reflector). This use of state information in the reflector is a classic case of the use of meta-data as a basis for a reflective operation. The reflector uses the data that describes the state of the base object to make a decision about how a system operation, in this case synchronous call/reply is implemented. When it decides that a context hand-off is acceptable, it manipulates the implementation of the mechanism to optimise the code path. I.e., it carries out computation at the meta-level and reflects the changes back into the system. To ensure sufficient application control, a meta-level method which allows context hand-off to be explicitly disabled by a request from the base-level objects is avail- able if needed. The performance benefits of this technique come from the avoidance of context-switches and are bought at the small cost of the decision making code held in the reflec- tor. The performance benefits from this simple technique are impressive, in table 3 we show the benefits gained within the driver environment with the use of context hand-offs. In this evaluation, context hand-off avoids fourteen context switches since we normally need context switches between objects and between schedulers which are also implemented as single-threaded concurrent ob- jects with reflectors. It should also be noted that in the case of the context hand-off decision failing, i.e. deciding that the four criteria mentioned above are not met, then the performance is not affected. Table 3: Costs of mSystem services (in sec) metaoperation Apertos Call-Reply_roundtrip_on_mSystem 772.8 (context_hand-off_not_implemented) Call-Reply roundtrip on mSystem (context hand-off succeed) 7.8 Call-Reply_roundtrip_on_mSystem 693.0 (context_hand-off_failed)________* To confirm that these benefits are consistent at a macro- scopic level we have used the context hand-off mechanism in the implementation of the IP protocol handlers. Based on our design principle, the network protocol handlers are implemented onto an appropriate system service layer for protocol handlers [12 ]. We have implemented IP network protocol handler objects onto the system service layer mSystem . There is a concurrent object per each network protocol, five in total, passing IP packets as message be- tween them. We have measured the execution time after reception of ICMP echo packet through the transmission of ICMP echoreply packet. Table 4 shows the results, in- dicating that optimised execution is 10 times faster than the non-optimised execution4 In the normal case there will be at least 54 context switches (dependent on timing conditions such as timer interrupts) during the execution of the non-optimised execution. In the optimized execu- tion we avoid all the context switches. Table 4: Costs of network protocol handlers on mSystem (in sec) metaoperation Apertos ICMP echo-echoreply roundtrip 3466.0 ICMP_echo-echoreply_roundtrip(optimised) 362.3 Message batching We have also implemented a technique which exploits message asynchrony to allow us to batch messages based on message send patterns. Again we use the meta-object model by using reflectors to gather statistics on message communication patterns. As discussed above, this is possible because all message communication goes through the reflector. We then al- low the meta-space to re-configure itself to change the semantics of the send from a simple send per invocation to a batched send model. As before, the meta-space can be configured to not batch asynchronous messages by in- voking a meta-level method from the base-level object. Fig.4 gives a graph of the initial performace speed up using this batched message send technique. The horizon- tal axis indicates the number of asynchronous messages to be actually batched into one packet; the vertical axis indicates the time to process all the message sending re- quests. As you can see, if the batching is enabled and there is only one message in a packet, there is a slight overhead. The source of the overhead is the processing time for batching. We can reduce execution time if we can batch 2 or more messages into single packet. ______________________________________ 4We do not provide performance figure for the UNIX variant because the implementations are completely different. [No figure included in ASCII version] Figure 4: Effect of simple message batching While the use of message batching is in itself not a new technique, the ease with which we can add this technique to our system is a function of the meta-space architec- ture. Since the system implements message send as a meta-operation, and since we have a meta object that de- scribes how that operation is performed it becomes sim- ple to use reflection to change the way we send messages. Again, at the implementation level, because our model moves from base to reflector on all meta-operations, it provides a convinient place to monitor system operation. In this case we simply instrument the reflector to moni- tor message traffic, that is, make message traffic statistics into meta-data, and use this meta-date as the basis for the reflective operation. Dynamic object reallocation, and context handoff A related technique we have adopted is the use of ad- dress space change combined with context hand-off. The context hand-off technique only works when two objects share the same protection domain limiting its use. How- ever, Apertos implements every system entity using fine- grained concurrent objects allowing several objects to be allocated onto a single address space. In Apertos, un- like UNIX and other operating system with fixed design policies, various aspects of the system such as protection, activity (CPU thread allocation), object persistence, and message passing semantics can be tailored to meet the needs of the system designers. Therefore, if performance is the critical issue for the objects and protection is not an issue, (for example they are written using a language sys- tem with strong type inference) the system designer can use a meta-space which allocates objects onto the same address space and always utilizes the above described op- timisation techniques. If performance is not an issue and protection is important, one can provide a meta-space which implements strict protection policies. An object can choose one of these policies on the fly, by migrating between two meta-spaces. Again Apertos utilizes fine-grained concurrent object, these objects can be moved from the current address space to other address spaces. By doing so, we can tailor object allocation on a per address space basis allowing us to maximize the utilization rate of the optimisation techniques. This technique resembles the load balancing techniques used between remote CPUs. Run-time code generation The last area of work that we have explored is the use of run time code generation and partial evaluation. It is often the case that we can optimise the general case execution path provided at initial system load based on application requirements. Again we use the meta-space to gather run time infor- mation about the execution characteristics of the appli- cation. By using the meta-object interface we are able to adapt the code execution to the application require- ments. This work is based on our initial investigation in the Cognac language [11 ]. Cognac uses static analysis and type information to drive run-time binding of an object to its meta-objects. By dynamically changing these run- time bindings we are able to maintain the same base pro- gramming interface, but change the semantics of the sys- tem level operations that support the application object. Also we are investigating ways to utilize partial evalu- ation techniques to flatten out meta-object hierarchies, which can then be statically linked into the base-level object. Conclusion We believe that operating system flexibility is the most important goal of current research. However we would argue that flexibility must be provided within the frame- work of a clean and consistent architecture. Our ap- proach, the Apertos operating system, uses the meta- object model and is targetted toward implementing the whole operating system within a reflective programming model. This approach has produced an operating system which is highly flexible at all levels, however, it has some performance drawbacks. In this paper we have discussed ways to minimize the execution cost by using the meta- object model in full and have shown that it is possible to exploit meta-object to gain significant performance ad- vantages without breaking the model. Current and future work We are currently working on the use of Apertos in a dis- tributed multi-media environment. In our initial set up we have ported Apertos to a number of Sony News 5000 workstations with MIPS R4000 processors and a video server using the R3000 processor. These machines are interconnected with a small experimental ATM network. We are experimenting with Apertos in two areas; the first uses the meta-object model to support QofS con- straints and exploits our flexible architecture to adapt the resource scheduling policies to varying simulated sys- tem loads and failures. The second area of work uses Apertos in a proprietary graphics engine attached to the above network where we are experimenting with proto- cols for distributed shared virtual environments. In par- ticular we are looking at the issues of distributed con- sistency where we are exploring the use of migration be- tween meta-spaces to dynamically change the consistency associated with shared objects. Also, we plan to inves- tigate the use of object migration (to the remote host) for downloading the whole system code including device drivers, to the graphics engine. Using meta-spaces as a clear abstraction for various visibility control, such as host boundary, persistent objects and volatile objects, is also planned and will be carried out in the next develop- ment phase. In addition we are continuing to analyse the perfor- mance of our system and hope to extend the techniques discussed above to all meta-spaces. We hope to report in the future on the success of this work. Availability of Apertos Operating System The Apertos system is available for research purpose and information, along with a collection of papers can be found by accessing the following URL: https://www.csl.sony.co.jp/ ftp://ftp.csl.sony.co.jp/ References [1] Gul A. Agha. Actors: A Model Of Concurrent Com- putation In Distributed Systems. Technical Report 844, MIT, June 1985. Technical Report 844. [2] Tevanian Avadis Jr. and Richard F. Rashid. MACH: A Basis for Future UNIX Development. Technical report, Department of Computer Science, Carnegie Mellon University, June 1987. [3] Nariman Batlivala, Gleeson Barry, Hamrick Jim, Lurndal Scott, Price Darren, Soddy James, and Abrossimov Vadim. Experience with SVR4 over CHORUS. In Micro-kernel and other kernel archi- tectures, pp. 223-241. USENIX, April 1992. [4] Brian Bershad. SPIN - an extensible micro-kernel for application specific operating system services. In Proceedings of the 6th ACM-SIGOPS European Workshop: Matching Operating Systems to Applica- tion Needs, September 1994. [5] C Bryce, V Issarny, G Muller, and I Puaut. To- wards safe and efficient customization in distributed systems. In Proceedings of the 6th ACM-SIGOPS European Workshop: Matching Operating Systems to Application Needs, September 1994. [6] Roy H. Campbell, Nayeem Islam, Ralph Johnson, Panos Kougiouris, and Peter Madany. Choices, Frameworks and Refinement. In Proceedings of the 1991 International Workshop on Object Orientation in Operating Systems, pp. 9-15. IEEE Computer Society Press, October 1991. [7] Graham Hamilton and Panos Kougiouris. The Spring nucleus: A microkernel for objects. In USENIX 1993 Summer Technical Conference Pro- ceedings. USENIX Association, June 1993. [8] Jun-ichiro Itoh, Yasuhiko Yokote, and Mario Tokoro. SCONE: A New Execution Model for Low- Level System Programming based on Concurrent Objects. Technical Report SCSL-TM-95-005, Sony Computer Science Laboratory Inc., 1995. submitted to OOPSLA95. [9] Gregor Kiczales, Jim des Rivieres, and Daniel G. Bobrow. The Art of the Metaobject Protocol. The MIT Press, 1991. [10] Rodger Lea, Christian Jacquemot, and Eric Pillevesse. COOL: System support for distributed programming. Communications of the ACM, Vol. 36, No. 9,, September 1993. [11] Kenichi Murata, Nigel Horspool, and Yasuhiko Yokote. Design and specification of cognac. Tech- nical Report SCSL-TM-94-006, Sony Computer Sci- ence Laboratory Inc., 1994. [12] Kenichi Murata and Yasuhiko Yokote. A Reflec- tive Network System Using Concurrent Objects and Meta Architectures. Technical Report SCSL-TM-93- 010, Sony Computer Science Laboratory Inc., July 1993. [13] Carlton Pu and Jon Walpole. A study of dynamic optimisation techniques: Lessons and directions in kernel design. Technical Report CS/E 93-007, Ore- gon Graduate Institute (OGI), April 1993. Technical Report CS/E 93-007. [14] Yasuhiko Yokote. The Design and Implementation of ConcurrentSmalltalk. World Scientific Publishing, 1990. [15] Yasuhiko Yokote. The Apertos Reflective Operat- ing System: The Concept and Its Implementation. In Proceedings of Object-Oriented Programming Sys- tems, Languages and Applications in 1992. ACM Press, October 1992. Also appeared in SCSL-TR- 92-014 of Sony Computer Science Laboratory Inc. [16] Yasuhiko Yokote, Gregor Kiczales, and John Lamp- ing. Separation of Concerns and Operating Systems for Highly Heterogeneous Distributed Computing. In Proceedings of the 6th ACM-SIGOPS European Workshop: Matching Operating Systems to Applica- tion Needs, September 1994.