################################################ # # # ## ## ###### ####### ## ## ## ## ## # # ## ## ## ## ## ### ## ## ## ## # # ## ## ## ## #### ## ## ## ## # # ## ## ###### ###### ## ## ## ## ### # # ## ## ## ## ## #### ## ## ## # # ## ## ## ## ## ## ### ## ## ## # # ####### ###### ####### ## ## ## ## ## # # # ################################################ The following paper was originally published in the Proceedings of the USENIX 1996 Conference on Object-Oriented Technologies (COOTS) Toronto, Ontario, Canada, June 1996. For more information about USENIX Association contact: 1. Phone: 510 528-8649 2. FAX: 510 548-5738 3. Email: office@usenix.org 4. WWW URL: https://www.usenix.org The Object Group Design Pattern Silvano Maffeis* maffeis@acm.org Olsen & Associates Zurich, Switzerland Abstract This paper describes "Object Group", an object behavioral pattern for group communication and fault-tolerance in distributed systems. The Object Group pattern supports the implementation of replicated objects, of load sharing, and of efficient multicast communication over protocols such as IP- multicast or UDP-broadcast. Application areas of the pattern are fault-tolerant client/server systems, groupware and parallel text retrieval engines. Events within an Object Group honor the Virtual Syn- chrony model. With Virtual Synchrony, the size of an object group can be varied at run-time while client applications are interacting with the group. A replicated state remains consistent in spite of objects entering and leaving the group dynamically and in spite of failures. The Object Group pattern has been implemented in the Electra and in the Orbix+Isis CORBA Object Request Broker. Keywords: Distributed Systems, Design Patterns, Replication, Group Communication, Virtual Syn- chrony. 1 Intent The Object Group design pattern provides a local surrogate for a group of objects distributed across net- worked machines. The members of an object group have a consistent view of which other objects are also part of the group and are notified of objects joining and leaving the group. The pattern also provides a high-level, object oriented interface to low-level multicast facilities such as Ethernet, ATM, UDP-broadcast and IP-multicast. 2 Also Known As Server Group Service, Replicated Object Pattern, Active Replica Pattern. 3 Motivation To illustrate the object group pattern, consider the following CORBA IDL inter* *face: _____________________________________________________________________________ | // IDL | | interface Directory { | | | | void install(in string key, in any value) | | raises (ENTRY_EXISTS); | | void lookup(in string key, out any value) | | raises (NO_SUCH_ENTRY); | | | | void remove(in string key, out any value) | | raises (NO_SUCH_ENTRY); | | }; | |____________________________________________________________________________| *Supported by the Swiss National Science Foundation (Schweizer Nationalfonds* *) while on leave at Cornell University, Dept. of Computer Science. The Directory interface could be used to implement an online phone book service. To ensure high availability of the service, several copies of the Directory are created on di* *fferent hosts and are joined to an object group. Client applications use one object reference to represent the* * whole group without being concerned about the references of the individual group members. The remove and install operations are multicast to the group because these m* *odify the replicated state of the directory. A lookup operation, on the other hand, needs to be directed * *to only one group member as the state of the service remains unaltered. Each group member replies to a remove operation. However, the client's runti* *me system returns only the first arriving member reply (transparent multicast style). If required, cl* *ients can access the replies of all group members using a non-transparent multicast style. An object group request succeeds as long as at least one group member surviv* *es the request. The replication degree of an object can be increased at run-time. When an object r* *equests to join a group, the runtime system first suspends group invocations. Then the runtime system reque* *sts the internal state1 of a group member by issuing its get_state member function. Subsequently, the run* *time transfers the state over the network to the newcomer object and invokes its set_state function wit* *h the state as a parameter. Finally, group invocations are resumed. The whole procedure is fault-tolerant;* * if the state-sending member fails, state transfer is restarted with another member. The underlying group management protocol guarantees a consistent replicated * *state of the Directory even when several clients fire install and remove operations while a new objec* *t is joining the group. This execution style is called Virtual Synchrony [2]. 4 Applicability The object group pattern can be very effective when there is a need for fault-* *tolerance, efficient dissemination of data, load-sharing or a combination thereof. Some interesting application a* *reas are: o Fault-tolerant client/server systems. The proposed pattern supports the i* *mplementation of highly available objects. Object groups can provide active replication, pa* *ssive replication, and multi- versioning. o Groupware. Object groups facilitate the implementation of applications su* *ch as teleconferencing, video-on-demand, distributed whiteboards and other kinds of groupware. Inf* *ormation flows between groups of individuals can be modeled and implemented in an efficient manne* *r. o Ticker services for financial information. In such applications, a continu* *ous data stream (for example quotes from a stock exchange) needs to be transmitted to a possibl* *y large set of receivers (for example to trader workstations) in an efficient and reliable manner (Figur* *e 1). The pattern allows the efficient transmission of data over protocols such as IP-multicast. Re* *liability is ensured by object replication and reliable multicast. o Scalable text retrieval systems. A parallel text retrieval engine can be i* *mplemented in the form of an object group with N members (Figure 2). Each group member is assigne* *d a portion of the text database. To perform a search, a client application multicasts the sought * *text pattern to the group. Each of the N group members needs to search only 1/N of the database. As i* *s explained below, group members have a consistent opinion on the group size in spite of failures a* *nd in spite of retriever-objects that join the group at run-time to increase parallelism. For this kind of * *application, a nearly linear speedup can be achieved by employing the object group pattern. o Caching. In certain situations, the response time of a service is decrease* *d if provided by an object group since replicas of the service can be placed at the sites where the s* *ervice is frequently accessed. In the extreme case, a replica is created per client application and place* *d on the clients' machines. A client transmits read-only requests only to the local replica; write-req* *uests, on the other hand, are multicast to all replicas. ________________________________1 for example, the entries in an electronic phone book. o Network management. Object groups offer a convenient solution to the probl* *em of collecting and propagating management and monitoring information in distributed applicati* *ons. o Object migration. Requests are multicast through opaque object references * *and senders need not know the network addresses of the members. Consequently, an object can le* *ave a group, move to another place and then rejoin the group and continue working. Object group* *s hence offer support for mobility and for system reconfiguration. This kind of object migratio* *n is useful for preventive maintenance and for coarse grained load balancing. Figure 1: Reliable Stock Exchange Ticker. 5 Virtual Synchrony The main requirement our pattern makes on the underlying communication subsyst* *em is that it implements the Virtual Synchrony model. Virtual Synchrony makes possible the run-time rep* *lication of mutable objects and the consistent detection of failures. Moreover, the notion of consistent m* *anagement of group membership information is a fundamental component of the Virtual Synchrony model. The mot* *ivation behind Virtual Synchrony is to allow programmers to assume a closely synchronized style of di* *stributed execution, even though the underlying system is asynchronous [2]: o The execution of an object consists of a sequence of events which may be i* *nternal computations, request transmissions, request dispatch, failure notifications or changes to the m* *embership of groups of which the object is a member. o A global execution of the system consists of a set of object executions. A* *t the global level requests are sent by multicast or unicast. o Any two objects that receive the same multicasts or observe the same group* * membership changes see the corresponding local events in the same relative order. Figure 2: Scalable Text Retrieval Engine. o A multicast to an object group is delivered to its full membership. The se* *nd and delivery events appear as a single, instantaneous event. o However, the model provides only the illusion of synchronous execution: Wh* *enever possible, synchro- nization is relaxed to reduce the degree to which processes can delay one * *another, and to better exploit the parallelism inherent to an application. 6 Structure and Participants Figure 3 depicts structure and participants of the object group pattern. Figur* *e 4 shows a possible layering of the communication subsystem. Client Application The client application creates an instance of an object group reference and bi* *nds the reference to a target group by using a name server to perform the name-to-object mapping. After succ* *essful binding, operations issued through the reference are transmitted to the object group. Figure 3: Structure of the Object Group Pattern. Figure 4: Layers of the Communication Subsystem. Object Group Reference An object group reference is used as a "smart pointer" to an object group. The* * object group reference is a surrogate object with the same interface as the group members. When a member f* *unction is invoked on the group reference, the runtime marshals the arguments, multicasts the request to* * the group members, waits for one or more member-replies, unmarshals the replies and passes the result b* *ack to the client. An object group reference can be used like a conventional CORBA object reference. Communication Subsystem The communication subsystem (or runtime) provides a low-level interface to rel* *iable multicast, group man- agement and light-weight processes. In this paper, we assume that the communic* *ation subsystem is part of a CORBA object request broker. We suggest implementing the communication subsy* *stem on top of toolkits such as Horus [19], Isis [2], Consul [12], Phoenix [11], Totem [13], or Transi* *s [1] as these provide low-level system support for process groups, reliable multicast and Virtual Synchrony. F* *or the sake of flexibility and portability, the communication subsystem implements a generic interface. An Ad* *aptor Object is used to map the generic interface onto the real interface provided by the toolkit (Fig* *ure 4). Object Implementation An object group consists of one or more object implementations. An object impl* *ementation provides the functionality of the service; for example, it implements a phone directory or * *a receiver of stock exchange quotes. Each group member implements the same IDL interface. Object Group The object implementations form a logical object group. The ORB ensures that e* *ach group member receives all requests that the client applications have submitted through the object re* *ferences (reliable multicast). Optionally, the ORB may guarantee that each object implementation dispatches r* *equests in exactly the same order (totally ordered multicast). Each group member obtains a unique rank number. The oldest member is assigne* *d rank 0, the next- oldest obtains rank 1 and so forth. Whenever an object joins or leaves (either* * explicitly or by crashing). the group, the group members obtain a view change notification from the ORB to* * inform them of their new ranks and of the number of group members. 7 Collaborations Server Side If an object implementation wants to join a group, it obtains an object group * *reference from a name server or by using the CORBA string_to_object function. Subsequently, it invokes a jo* *in function with the reference as a parameter. (There is also a leave and a create_group function).* * After calling join, its set_state member function is invoked by the ORB to update its internal state t* *o the state of the other group members. In the above Directory example, the state consists of the strin* *g/any entries maintained by the service. Client Side When an application wants to send a request to an object group, it obtains an * *object group reference either from a name server or by using the CORBA string_to_object function. Now the ap* *plication can submit requests through the group reference in a type-safe manner. If required, the c* *lient application can request that each group member's reply be returned (non-transparent multicast). View Management At all times, a member of a group has a certain view of which other objects be* *long to its group. A view is an ordered set of object references; there is one reference per group membe* *r. The first entry of a view (view[0]) represents the oldest member, the second entry (view[1]) the second * *oldest and so forth. Each group member knows its own index in the view. This ranking is consistent among* * all group members. Join and leave operations trigger the installation of a new view at each group memb* *er's runtime system. View management is an important component of the Virtual Synchrony model. To* * hold views consistent, the object group pattern employs a view management protocol similar to the fol* *lowing (Figure 5). The oldest member is said to be the coordinator of the group. When an object* * wishes to join a group, the object's runtime sends a JOIN_REQUEST message to view[0] (to the coordinator).* * The coordinator multicasts a FLUSH_REQUEST message to the group members. Upon receipt of a FLUSH_REQUEST,* * a member submits any cached requests it needs to have delivered in the old view. Each member confir* *ms that it is done with the flushing by returning a FLUSH_ACK message to the coordinator. A member is not * *allowed to submit any more requests since the view is now unstable. When the coordinator has receive* *d a FLUSH_ACK from every surviving member, it sends one or several STATE_MSGs containing its internal s* *tate to the newcomer object. After this state transfer is completed, the coordinator multicasts a VIEW mess* *age containing the new view. When a member receives a VIEW message, it updates its local view vector with t* *he data in the message. The view is stable and the members may continue to submit requests. If the coordinator fails, the group members receive a failure notification f* *rom the runtime and the member with object reference view[1] becomes the new coordinator. The view transfer i* *s then restarted with the new coordinator. The view management mechanisms are mostly transparent to the user of the obj* *ect group pattern. How- ever, by overloading specific member functions, an application dependent actio* *n can be triggered whenever a new view is installed. Variations of this view management protocol are discu* *ssed in [17, 15, 7, 4]. 8 Consequences The object group pattern offers the following main benefits: o Fault-tolerance through active replication, passive replication or multi-v* *ersioning. This makes possible fault-tolerant client/server systems. o Efficient group communication over Ethernet, IP-multicast and other facili* *ties that enable multicast. The object group pattern provides a high-level, invariable interface to su* *ch facilities. This supports the implementation of groupware systems and of certain applications in the* * financial domain. Figure 5: View Management Protocol. Object N requests to join a group containi* *ng three objects. Object C is the coordinator of the group. o Load sharing through a group of objects in which each member performs only* * a part of the computations associated with a client's requests. This makes possible parallel text ret* *rieval applications and the like. o Scalability through the replication of services which are mainly accessed * *by read-only operations. o System reconfiguration through dynamic object migration. o Last but not least, an object-oriented interface to toolkits like Horus an* *d Isis is provided. The object group pattern has following drawbacks: o An efficient and robust implementation of the pattern requires sophisticat* *ed system support not yet available in today's commonly used operating systems. This system support * *includes: reliable, totally ordered multicast of network messages, a group management protocol to ensu* *re that object group members have consistent views on which objects are in the group, failure d* *etection and consistent propagation of failure notifications. o Fault-tolerant name servers must be provided, allowing objects to retrieve* * the references of the groups they want to join. o Fault-tolerance is not transparent to the programmer. Programmers have to * *deal with object groups, with join and with leave requests. For stateful objects, programmers need* * implement state trans- fer upcalls unless an IDL compiler is provided to generate state-transfer *code out of enriched IDL specifications. 9 Implementation Group Management API In a CORBA Object Request Broker, the object group pattern can be implemented * *as an extension of the CORBA Basic Object Adapter (BOA): _____________________________________________________________________________ | // C++ | | class GroupBOA: virtual public BOA { | | public: | | | | // Operations on object groups: | | // | | static void create_group(Object_ptr group, | | const ProtocolPolicy& policy | | =default_protocol_policy); | | void join(Object_ptr group); | | void leave(Object_ptr group); | | static void destroy_group(Object_ptr group); | | | | | | virtual void get_state(AnySeq& state, | | Boolean& done); | | virtual void set_state(const AnySeq& state, | | | | Boolean done); | | virtual void view_change(const View& newView); | | }; | |____________________________________________________________________________| The create_group method creates a new object group and binds the object refe* *rence group to it. The ref- erence can be installed in a name server or converted to a human-readable stri* *ng by the ORB::object_to_string operation. The policy argument tells the underlying toolkit what kind of multi* *cast protocol to employ; for example, total ordering or causal ordering [8, 5]. If the BOA is configure* *d to run on Isis, the policy object selects one of the Isis abcast, cbcast, fbcast or gbcast protocols. In * *the Horus configuration, the policy object selects a Horus protocol stack2 [18] . The policy-object mechanism can * *be extended to cover quality of service guarantees. For example, the minimum bandwidth necessary to sustain* * a certain service can be defined through a policy object. Objects in the network join or leave a group by retrieving its reference fro* *m a name server and by issuing the join or leave operation with the reference as a parameter. destroy_group i* *rrevocably destroys an object group but without destroying the group-members themselves. State Transfer When an object joins a non-empty group, the ORB requests the internal state (t* *hat is, the values of the instance variables) of some group-member by invoking its get_state method. Sub* *sequently, the runtime transfers the state to the newcomer and invokes the newcomer's set_state metho* *d. The runtime can transfer a large state in fragments by repeatedly invoking the state transfer functions* * until TRUE is assigned to the done argument of get_state. An object-state is represented as a sequence of CO* *RBA any objects. State transfer is necessary for redundant computations to permit the replica* *tion-degree of a stateful object to be increased at run-time. The programmer must write application-specific g* *et_state and set_state methods. These methods can also be used to checkpoint the state of an object t* *o non-volatile storage or to perform object migration. The view_change method of an object is invoked whenever another object joins* * or leaves the group. The newView object contains the new cardinality of the group as well as the ob* *ject references of the group members.________________________ 2Horus supports a variety of ordering and reliability protocols, as well as * *communication over UDP, IP-multicast, ATM, Mach messages, and so forth. Call Styles The programmer may specify how many member-replies the runtime should collect * *during an object-group invocation. Furthermore, operations that leave the state of an object unchange* *d can be tagged as readonly. Such operations are automatically transmitted by point-to-point communication * *to only one group member. To help implement this, we suggest that a readonly operation attribute be prov* *ided by the IDL. Following call styles can be selected by the programmer on a per-call basis: o ALL: This call type demands that the replies of all operational group-memb* *ers be collected and returned to the caller. The caller is suspended until all group members ha* *ve replied. o MAJORITY: The call is active only until a majority of the members have rep* *lied. o ONE: The call is active only until the first member-reply is received. Th* *is call-style is used for transparent multicast. o N: Any number ranging rom 1 to the number of members of the group can be s* *pecified. If the specified number of replies cannot be collected due to a failure, an exception is th* *rown. Object Migration In order to migrate an object implementation that is the only member of a cert* *ain group, one has to instantiate a new object implementation on the destination machine and to join* * the object to the group. The state of the obsolete object will automatically be transferred to the newc* *omer object and the two objects will be synchronized. The obsolete object can then be destroyed. 10 Sample Code The following code illustrates the replication of a Directory object and the i* *nteraction with the resulting Directory object group. Server Side The code fragment below demonstrates a server application that instantiates an* * implementation of interface Directory, creates an object group if given the -c option, and joins the newly* * created object to the group. The -c option also causes a group reference to be installed into the naming se* *rvice. The -j option causes the Directory object implementation to join an existing group. ______________________________________________________________________________ | // C++ | | main(int argc, char **argv) { | | Object_var obj, group; | | // a reference to the naming service: | | | | extern CosNaming::NamingContext_ptr nc; | | // create an implementation of interface Directory: | | _im_directory obj0; | | ::: | | | | switch(argv[1][1]){ | | case 'c': // create and join a group: | | // get a reference for obj0: | | obj = obj0.create(rd, pt, dpt); | | | | // create an empty object group: | | obj0.create_group(obj); | | // install group reference into name server: | | nc->bind("directory", obj); | | | | // join obj0 to group: | | obj0.join(obj); | | break; | | case 'j': // join an existing group: | | | | // retrieve group reference from name server: | | group = nc->resolve("directory"); | | // join obj0 to group: | | obj0.join(group); | | | | break; | | default: | | cerr << "usage: " << argv[0] << " -c|-j\n"; | | exit(1); | | | | }; | | } | | | |_____________________________________________________________________________| Client Side The following code fragment shows a procedure that is part of a client applica* *tion. The object group reference is retrieved from the name server. Subsequently, a transparent and a* * non-transparent request is issued through the reference. A non-transparent request is appropriate when th* *e members of a Directory group provide different information; for example, one member might maintain bu* *siness phone numbers, another|member|home numbers: * * || ______________________________________________________________________________ | // C++ | | void clientProc(){ | | | | extern CosNaming::NamingContext_ptr nc; | | Any entry; AnySeq *entries; | | char *number; | | Environment env; EnvironmentSeq envs; | | | | directory_var dir = | | Directory::_narrow(nc->resolve("directory")); | | | | // transparent multicast | | | | dir->lookup("J. Smith", entry, env); | | entry >>= number; | | cout << "The phone number of J. Smith is " | | << number << "\n"; | | | | ::: | | | | // non-transparent multicast | | dir->lookup("J. Smith", entries, envs); | | | | for(i =0; i < entries->length(); i++){ | | (*entries)[i] >>= number; | | cout << "A phone number of J. Smith is " | | << number << "\n"; | | | | }; | | delete entries; | | } | |_____________________________________________________________________________| 11 Known Uses The object group pattern has been implemented in the Electra [9, 10] and in th* *e Orbix+Isis [6] ORB. Electra Electra is a flexible CORBA-2 Object Request Broker based on the object group * *paradigm. In Electra, the object group pattern is implemented as an extension of the CORBA Basic Object * *Adapter (BOA), as was proposed in Section 9. Electra supports the dynamic replication of CORBA objec* *ts, failure detection, asyn- chronous communication and light-weight processes. The ORB is portable; the pr* *esent version of Electra runs on both Horus and Isis. Electra is publicly available through the Web at http:* *//www.olsen.ch/~maffeis/electra.html. Orbix+Isis Orbix+Isis is an adaptation of Orbix from Iona Ltd. that runs on top of the Is* *is toolkit. To become a member of an object group, an object implementation must inherit from a base class th* *at implements either an Active Replica or an Event Stream group style. The Active Replica object group offers* * three communication styles: Multicast, Client's Choice, and Coordinator/Cohort. Orbix+Isis is a commercial* * product available from Isis Inc. For more information contact info@isis.com. 12 Related Patterns Proxy Pattern A Proxy provides a surrogate or placeholder for another object to control acce* *ss to it [16, 3]. Analogously, an Object Group Reference provides a surrogate or placeholder for a group of o* *bjects. Coordinator/Cohort Pattern Coordinator/Cohort [2, 10] is a form of redundant computation in which only on* *e object (the coordinator) performs the computations associated with the client requests. Several cohort * *objects are associated with a coordinator, acting as "hot standbys". When the coordinator fails, one of its * *cohorts takes over. Coordinator/Cohort is a special form of the object group pattern: coordinato* *r and cohorts make up an object group in which the oldest group member is the coordinator. When the coo* *rdinator fails, the cohorts receive a view change notification from the ORB and the oldest cohort becomes * *the new coordinator. New cohorts can be included at run-time. Event Channel Pattern The Event Channel Pattern [14] provides the abstraction of a highly available,* * persistent message bus. The Event Channel allows an object to "post" requests and to "subscribe" for reque* *sts it is interested in3. Posting and subscription can be by ASCII strings that represent topics of interest. Objects can post requests for receivers that happen to be unavailable. To t* *hat purpose, the Event Channel provides persistency of requests. Receivers may later connect to an Ev* *ent Channel to retrieve a backlog of requests. Thus, clients are decoupled from servers and communicatio* *n is asynchronous. The Event Channel pattern can be seen as a variation of the object group pat* *tern. The patterns differ mainly in that Event Channel allows its members to selectively listen only for* * certain requests. Another difference is that Event Channels allow the spooling of requests on non-volati* *le storage so that requests can be directed to group members that are temporarily unavailable. Object Group, o* *n the other hand, excludes any member that has been unresponsive for a relatively short period of time, t* *ypically in the range of 10 to 30_seconds._____________________ 3similar to Usenet. Acknowledgements The author would like to thank Robert Ward for many helpful comments. A prelim* *inary version of this material was discussed at the OOPSLA'95 Workshop on Design Patterns for Concur* *rent, Parallel, and Distributed Object-Oriented Systems. The author would like to thank the worksh* *op participants for their valuable suggestions and for their encouragement. References [1]Amir, Y., Dolev, D., Kramer, S., and Malki, D. Transis: A Communication Su* *b-System for High Availability. In 22nd International Symposium on Fault-Tolerant Comput* *ing (July 1992), IEEE. [2]Birman, K. P., and van Renesse, R., Eds. Reliable Distributed Computing wi* *th the Isis Toolkit. IEEE Computer Society Press, 1994. [3]Gamma, E., Helm, R., Johnson, R., and Vlissides, J. Design Patterns: Eleme* *nts of Reusable Object-Oriented Software. Addison Wesley, 1995. [4]Golding, R. A., and Taylor, K. Group Membership in the Epidemic Style. Tec* *h. rep., University of California, Santa Cruz, 1992. [5]Hadzilacos, V., and Toueg, S. Fault-Tolerant Broadcasts and Related Proble* *ms. In Distributed Systems, S. Mullender, Ed., second ed. Addison Wesley, 1993. [6]Isis Distributed Systems, Inc., Iona Technologies, Ltd. Orbix+Isis Program* *mer's Guide, 1995. Document D071-00. [7]Jahanian, F., Fakhouri, S., and Rajkumar, R. Processor Group Membership Pr* *otocols: Spec- ification, Design and Implementation. In Proceedings of the 12th Symposium * *on Reliable Distributed Systems (Princeton, New Jersey, Oct. 1993), IEEE. [8]Lamport, L. Time, Clocks and the Ordering of Events in a Distributed Syste* *m. Communications of the ACM 21, 7 (July 1978). [9]Maffeis, S. Adding Group Communication and Fault-Tolerance to CORBA. In Pr* *oceedings of the 1995 USENIX Conference on Object-Oriented Technologies (Monterey, CA, June * *1995), USENIX. [10]Maffeis, S. Run-Time Support for Object-Oriented Distributed Programming. * *PhD thesis, University of Zurich, Department of Computer Science, 1995. [11]Malloth, C. P., Felber, P., Schiper, A., and Wilhelm, U. Phoenix: A Toolki* *t for Building Fault-Tolerant, Distributed Applications in Large Scale. In IEEE SPDP-7 Wor* *kshop on Parallel and Distributed Platforms in Industrial Products (San Antonio, TX, Oct. 1995), * *IEEE. [12]Mishra, S., Peterson, L. L., and Schlichting, R. D. Consul: A Communicatio* *n Substrate for Fault-Tolerant Distributed Programs. Distributed Systems Engineering Journa* *l 1, 2 (Dec. 1993). [13]Moser, L. E., Melliar-Smith, P. M., Agarwal, D. A., Budhia, R. * *K., Lingley- Papadopoulos, C. A., and Archambault, T. P. The Totem System. In Proc. of t* *he 25th Annual International Symposium on Fault-Tolerant Computing (Pasadena, CA, June 199* *5). [14]Object Management Group. Common Object Services Specification Volume I. OM* *G Document 94-1-1. [15]Ricciardi, A. M. The Group Membership Problem in Asynchronous Systems. PhD* * thesis, Department of Computer Science, Cornell University, Ithaca, New York, Nov. 1992. No. 9* *2-1313. [16]Shapiro, M., et al. SOS: An Object-oriented Operating System - Assessment* * and Perspectives. Computing Systems 2, 4 (Dec. 1989). [17]van Renesse, R. The Horus Uniform Group Interface. Horus Documentation. [18]van Renesse, R., and Birman, K. P. Protocol Composition in Horus. In Proce* *edings of the 14th Annual ACM Symposium on Principles of Distributed Computing (Ottawa, Ontari* *o Canada, Aug. 1995). [19]van Renesse, R., Birman, K. P., and Maffeis, S. Horus: A Flexible Group C* *ommunication System. Communications of the ACM 39, 4 (Apr. 1996).