ParaSync: Exploiting Fine-Grained Parallelism for Efficient File Synchronization

File synchronization (sync) based on Content-Defined Chunking (CDC) is gaining increasing importance for data migration over networks owing to its effectiveness in detecting and eliminating duplicate data within synchronized files. CDC-based sync schemes typically comprise three phases: file chunking, chunk matching, and delta reconstruction. Unfortunately, existing sync schemes fail to exploit parallelism inherent in these phases due to two dependencies: a sequential bottleneck in chunking, where checksums are computed only after boundaries are finalized, and rigid client-server stalls that serialize matching and reconstruction. This paper presents ParaSync, a novel CDC-based file sync scheme that breaks these dependencies to exploit fine-grained parallelism. First, ParaSync’s multi-threaded chunking algorithm reduces checksum computation to a lightweight combination step, decoupling it from boundary identification while preserving invariability. Second, ParaSync designs a streaming chunk matching method that removes the all-or-nothing exchange dependency on both the client and the server sides. Finally, ParaSync introduces an efficient absolute-offset-based pipelined delta reconstruction process that maximizes the overlap between network and disk I/O operations. We have done extensive experiments over both WANs and LANs using diverse real-world datasets. The results show that compared to the state-of-the-art file sync schemes, ParaSync achieves up to 7.6× speedup for file chunking and significantly improves the overall sync performance by up to 3.7×, while maintaining a consistent level of network traffic.