During recovery, all four regions of the disk are locked exclusively
by the recovery process. The Hummingbird interface comes alive to
the application early in the process - after the hot cluster log of the
previous session has been recovered. However, Hummingbird will not write
new files or clusters to disk until recovery has completed;
write_file() calls will fail. It continues
to recover the clusters in the background and rebuild the
in-memory meta-data. During this phase, requests that do not hit in the
currently-rebuilt meta-data are checked in the file-to-cluster mappings
to identify the cluster that needs to be recovered. Since the
file-to-clusters mapping can be out-of-date (due to a crash), a
file without a file-to-cluster mapping is
viewed as not in the file system.
We now outline the sequence of events during crash recovery;
recovery is much simpler during a planned shutdown.
(1) Crash or power failure. The system reboots and enters recovery mode.
(2) The hot cluster log is scanned, the hot clusters are read in, and
their contents are used to initialize in-memory meta-data.
(3) The file-to-cluster mappings are read in from disk.
(4) The delete log is scanned and records are applied, modifying
meta-data as necessary.
(5) Proxy service is enabled. Hummingbird will now service requests.
(6) During idle time, the recovery process scans the cluster region, rebuilding
the in-memory meta-data for all files and clusters. As files are
recovered, they are available to the application.
(7) Recovery is complete. All files and clusters are now
available. Hummingbird can now write new clusters to disk.