Building a Time Machine for Efficient Recording and Retrieval of High-Volume Network Traffic

There are times when it would be extraordinarily convenient to record the entire contents of a high-volume network traffic stream, in order to later ``travel back in time'' and inspect activity that has only become interesting in retrospect. Two examples are security forensics--determining just how an attacker compromised a given machine--and network trouble-shooting, such as inspecting the precursors to a fault after the fault. We describe the design and implementation of a Time Machine to efficiently support such recording and retrieval. The efficiency of our approach comes from leveraging the heavy-tailed nature of network traffic: because the bulk of the traffic in high-volume streams comes from just a few connections, by constructing a filter that records only the first N bytes of each connection we can greatly winnow down the recorded volume while still retaining both small connections in full, and the beginnings of large connections (which often suffices).