Hummingbird's Memory Objects

Hummingbird stores two types of objects in main memory: files and clusters. Files are variable-sized cacheable objects, such as HTML pages and images. Clusters are fixed-sized objects containing files and some meta-data. The caching web proxy provides locality hints to Hummingbird by requesting that certain files be co-located. These files are usually the HTML page and its embedded images. Hummingbird uses these hints to pack files into clusters. However, the packing occurs as late as possible, which is when space is needed in the main memory. When the contents of a cluster is written to the disk, its associated main memory is freed. Only a small amount of meta-data is left behind in the memory in order to facilitate fast lookup of cached files. Clusters are read into memory and written to disk in one I/O operation to amortize the cost of the I/O. The total size of live objects in Hummingbird is bounded.

Hummingbird maintains three types of meta-data information: file system meta-data, file meta-data, and cluster meta-data. File and cluster meta-data are fixed-sized objects, which are associated with the variable-sized files and clusters. The file system meta-data is needed for the system to maintain state and manage the memory.

In summary, Hummingbird stores fixed-sized and variable-sized objects in memory for various durations (not all objects are short-lived or long-lived). Some memory objects, such as frequently accessed files and clusters may stay in memory for an extended period of time, while other objects, such as files and clusters which are rarely used, stay in memory only for a brief time. Some meta-data objects are never deleted from memory or have very long lifetimes.