Introduction

The cost and complexity of administration of systems is now the dominant factor in total cost of ownership for both hardware and software.In addition, since human operator error is the source of a large fraction of outages [5], attention has recently been focused on simplifying and ultimately automating administration and management to reduce the impact of failures [13,19], and where this is not fully possible, on building self-monitoring components [20]. However, fast, accurate detection of failures and recovery management remains difficult, and initiating recovery on ``false alarms'' often incurs an unacceptable performance penalty; even worse, initiating recovery on ``false alarms'' can cause incorrect system behavior when system invariants are violated [20].

Operators of both network infrastructure and interactive Internet services have come to appreciate the high-availability and maintainability advantages of stateless and soft-state [33] protocols and systems. The stateless Web server tier of a typical three-tier service [3] can be managed with a simple policy: misbehaving components can be reactively or proactively rebooted, which is fast since they typically perform no special-case recovery, or can be removed from service without affecting correctness. Further, since all instances of a particular type of stateless component are functionally equivalent, overprovisioning for load redirection [3] is easy to do, with the net result that both stateless and soft-state components can be overprovisioned by simple replication for high availability.

However, this simplicity does not extend to the stateful tiers. Persistent-state subsystems in their full generality, such as filesystem appliances and relational databases, do not typically enjoy the simplicity of using redundancy to provide failover capacity as well as to incrementally scale the system. We argue that the ability to use these HA techniques can in fact be realized if we subdivide ``persistent state'' into distinct categories based on durability and consistency requirements. This has in fact already been done for several large Internet services [31,38,28], because it allows individual subsystems to be optimized for performance, fault-tolerance, recovery, and ease-of-management.

In this paper, we make three main contributions:

1. We focus on user session state, which must persist for a bounded-length user session but can be discarded afterward. We show why this class of data is important, how its requirements are different from those for persistent state, and how to exploit its consistency and workload requirements to build a distributed, self-managing and recovery-friendly session state storage subsystem, SSM. SSM provides a probabilistic bounded-durability storage guarantee for such state. Like stateless or soft-state components, any node of SSM can be rebooted without warning and without compromising correctness or performance of the overall application. No node performs special-case recovery code. Additional redundancy allows multiple simultaneous failures. As a result, SSM can be managed using simple, ``stateless tier'' HA techniques for incremental scaling, fault tolerance, and overprovisioning.

2. We demonstrate the resulting simplicity of recovery management by combining SSM with a generic statistical-monitoring failure detection tool. Pinpoint looks for ``anomalous'' behaviors (based on historical performance or deviation from the performance of peer nodes) and immediately coerces any misbehaving node to crash and reboot. Although false positives do occur, the simplicity and low cost of recovery (crash and reboot) makes them a minor consideration, greatly simplifying SSM's failure detection and management strategy. Combined with SSM's additive increase/multiplicative decrease admission control that protects it from overload, the result is a largely self-managing subsystem using entirely generic detection and recovery techniques.

3. We summarize the design choices and lessons, along with the system architecture requirements that allow the approach to work, and highlight design principles that can be applied to other systems.

In Section 2, we define a category of session state, its associated workload, and existing solutions. In Section 3, we present the design and implementation of SSM, a recovery-friendly and self-managing session state store. In Section 4, we describe the integration of SSM with Pinpoint to enable the system to be self-healing. In Section 5, we present benchmarks demonstrating the features of SSM. In Section 6, we insert SSM into an existing production internet application and compare its performance, failure, and recovery characteristics with the original implementation. In Section 7, we discuss the design principles extracted from SSM. We then discuss related and future work, and conclude.