10–20x Faster Software Builds

Almost all software projects with more than a few dozen developers are plagued by slow builds that sap productivity, extend release schedules, and impact product quality. Parallel builds offer the potential of significant speedups, but previous attempts at parallelizing builds have had only modest success, primarily due to the lack of complete dependency information. In this talk I will present the architecture of Electric Cloud, a gmake-compatible build system that uses clusters of inexpensive servers to run massively parallel builds. The key to the Electric Cloud approach is that it deduces dependencies on the fly by monitoring file accesses during the build, so that it knows when it is or isn't safe to run build steps in parallel. I will also describe other aspects of the system, such as its versioning network file system and its use of peer-to-peer protocols for moving file data efficiently. Finally, I will compare Electric Cloud to other approaches such as distcc.