We find that the sendmsg() change solves most of the filesystem-related blocking. However, one open() call in Flash still shows occasional blocking at the label ``biord'' (reading a disk block), but only after the server has been running for some time and under heavy workloads. Only revealing which call induced the problem may not suffice a complete picture, because the reason of invoking that call is unclear. In a system with multiple identical system calls, existing tools do not have an efficient way to find which one causes the problem and the calling path involved.
Because DeBox information is returned in-band, the user-space context is also available once kernel performance anomaly is detected. On finding a blocking invocation of open(), we capture the path through the user code by calling abort() and using gdb to dump the stack2. This approach uncovers a subtle performance bug in Flash induced by mapped-file cache replacement. Flash has two independent caches - one for URL-to-filename translations (name cache), and another for memory-mapped regions (data cache). For this workload, the name cache does not suffer from capacity misses, while the data cache may evict the least recently used entries. Under heavy load, a name cache hit and a data cache capacity miss causes Flash to erroneously believe that it had just recently performed the name translation and has the metadata cached. When Flash calls open() to access the file in this circumstance, the metadata associated with the name conversion is missing, causing blocking. We solve this problem by allowing the second set of helpers, the read helpers, to return file descriptors if the main process does not already have them open. After fixing this bug, we are able to handle 390 simultaneous connections, and a 1.34GB dataset.