Large-Scale Automated Vulnerability Addition and the Search for Truth

Work on automating vulnerability discovery has long been hampered by a shortage of ground-truth corpora with which to evaluate tools and techniques. This lack of ground truth prevents authors and users of tools alike from being able to measure such fundamental quantities as miss and false alarm rates. In this talk, we detail LAVA, a novel dynamic taint analysis-based technique for producing ground-truth corpora by quickly and automatically injecting large numbers of realistic bugs into program source code. Every LAVA bug is accompanied by an input that triggers it whereas normal inputs are extremely unlikely to do so. These vulnerabilities are synthetic but, we argue, still realistic, in the sense that they are embedded deep within programs and are triggered by real inputs. LAVA has already been used to inject thousands of bugs into programs of between 10K and 2M LOC, and we have begun to use the resulting corpora to evaluate bug finding tools. Our vision is to scale up the LAVA infrastructure to enable frequent online self-evaluation. Developers and evaluators of bug finding tools and techniques will be able to obtain fresh corpora seeded with unknown vulnerabilities on demand, submit their results to be graded automatically, and receive feedback in a tight iterative loop. It is our hope that this will encourage lively and healthy competition that is informed by meaningful performance measures.