Representativeness in the Benchmark for Vulnerability Analysis Tools (B-VAT)


Kayla Afanador and Cynthia Irvine, Naval Postgraduate School

Short Preliminary Work Paper


A variety of tools are used to support software vulnerability analysis processes. However, when analysts want to select the optimal tool for a particular use case, or attempt to compare a new tool against others, no standard method is available to do so. In addition, we have determined that although vulnerabilities can be categorized into vulnerability types, these types are often disproportionately represented in current datasets. Hence, when comparative analyses of tools based upon these datasets are conducted, the results are distorted and unrealistic. To address this problem, we are building a Benchmark for Vulnerability Analysis Tools (B-VAT).

Representativeness is a key element of a good benchmark. In this paper, we use stratified sampling to identify a representative set of vulnerabilities from over 800 CWE’s and 75,000 CVE’s. This set becomes the foundation upon which we will build a purpose-based benchmark for vulnerability analysis tools. By using the correlation between current CWE and CVE data, the proposed B-VAT will assess tools using vulnerabilities in the proportions their types occur in the wild.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

@inproceedings {256940,
author = {Kayla Afanador and Cynthia Irvine},
title = {Representativeness in the Benchmark for Vulnerability Analysis Tools (B-VAT)},
booktitle = {13th {USENIX} Workshop on Cyber Security Experimentation and Test ({CSET} 20)},
year = {2020},
url = {},
publisher = {{USENIX} Association},
month = aug,

Presentation Video