Network-Sync Option

Assuming that an external client interacts with a primary site and the primary site implements some higher level remote mirroring protocol, network-sync enhances that remote mirroring protocol as follows. First, a host located at the primary site submits a write request to a local storage system such as a disk array (e.g. [12]), storage area network (e.g. [19]), or file server (e.g. [31]). The local storage system simultaneously applies the requested operation to its local storage image and uses a reliable transport protocol such as TCP to forward the request to a storage system located at the remote mirror. To implement the network-sync option, an egress router located at the primary site forwards the IP packets associated with the request, sends additional error correcting packets to an ingress router located at the remote site, and then performs a callback, notifying the local storage system which of the pending updates are now safely in transitEgress and ingress routers operate as gateway routers between datacenter and wide-area networks, where egress routers send packets from local datacenter networks to the wide-area network and ingress routers receive packets from the wide-area network and forward packets to local datacenter networks. Generally, egress routers also function as ingress routers and visa versa since they handle duplex traffic.. The local storage system then replies to the requesting host, which can advance to any subsequent dependent operations. We assume that ingress and egress routers are under the control of site operators, thus can be modified to implement network-sync functionality.

Figure 3: Network-Sync Remote Mirroring Option. (1) A primary-site storage system simultaneously applies a request locally and forwards it to the remote mirror. After the network-sync layer (2) routes the request and sends additional error correcting packets, it (3) sends an acknowledgment to the local storage system -- at this point, the storage system and application can safely move to the next operation. Later, (4) a remote mirror storage system receives the mirrored request--possibly after the network-sync layer recovered some lost packets. It applies the request to its local storage image, generates a storage level acknowledgment, and (5) sends a response. Finally, (7) when the primary storage system receives the response, it knows with certainty that the request has been mirrored and can gargage collect.

Table 1: Comparison of Mirroring Protocols.

Mirror	Mirror	Mirror-ack	Mirror-ack	Rolling Disaster
Solution	Update	Receive	Latency
				Local-only	Local +	Local +	Local+Mirror
				Failure	Pckt Loss	NW Partition	Failure
Local-sync	Async- or Semi-sync	N/A	N/A	Loss	Loss	Loss	Loss
Remote-sync	Synchronous	Storage-level ack (7)	WAN RTT	No loss	No loss	No loss	Maybe loss
Network-sync	Semi-sync	nw-sync feedback (3)	$\approx$ Local ping	$\approx$ No loss	$\approx$ No loss	Loss	Loss

Later, perhaps 50ms or so may elapse before the remote mirror storage system receives the mirrored request--possibly after the network-sync layer has reconstructed one or more lost packets using the combination of data received and error-recovery packets received. It applies the request to its local storage image, generates a storage level acknowledgment, and sends a response. Finally, when the primary storage system receives the response, perhaps 100ms later, it knows with certainty that the request has been mirrored and can garbage collect any remaining state (e.g. [19]). Notice that if a client requires the stronger assurances of a true remote-sync, the possibility exists of offering that guarantee selectively, on a per-operation basis. Figure 3 illustrates the network-sync mirroring option and Table 1 contrasts it to existing solutions.