Large-Scale Evaluation of a Vulnerability Analysis Framework

Ensuring that exploitable vulnerabilities do not exist in a piece of software written using type-unsafe languages (e.g., C/C++) is still a challenging, largely unsolved problem. Current commercial security tools are improving but still have shortcomings, including limited detection rates for certain vulnerability classes and high falsepositive rates (which require a security expert’s knowledge to analyze). To address this there is a great deal of ongoing research in software vulnerability detection and mitigation as well as in experimentation and evaluation of the associated software security tools. We present the secondgeneration prototype of the MINESTRONE architecture along with a large-scale evaluation conducted under the IARPA STONESOUP program. This second evaluation includes improvements in the scale and realism of the test suite with real-world test programs up to 200+KLOC. This paper presents three main contributions. First, we show that the MINESTRONE framework remains a useful tool for evaluating real-world software for security vulnerabilities. Second, we enhance the existing tools to provide detection of previously omitted vulnerabilities. Finally, we provide an analysis of the test corpus and give lessons learned from the test and evaluation.