Why Can't Johnny Fix Vulnerabilities: A Usability Evaluation of Static Analysis Tools for Security


Justin Smith, Lafayette College; Lisa Nguyen Quang Do and Emerson Murphy-Hill, Google


Static analysis tools can help prevent security incidents, but to do so, they must enable developers to resolve the defects they detect. Unfortunately, developers often struggle to interact with the interfaces of these tools, leading to tool abandonment, and consequently the proliferation of preventable vulnerabilities. Simply put, the usability of static analysis tools is crucial. The usable security community has successfully identified and remedied usability issues in end user security applications, like PGP and Tor browsers, by conducting usability evaluations. Inspired by the success of these studies, we conducted a heuristic walkthrough evaluation and user study focused on four security-oriented static analysis tools. Through the lens of these evaluations, we identify several issues that detract from the usability of static analysis tools. The issues we identified range from workflows that do not support developers to interface features that do not scale. We make these findings actionable by outlining how our results can be used to improve the state-of-the-art in static analysis tool interfaces.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

@inproceedings {255682,
author = {Justin Smith and Lisa Nguyen Quang Do and Emerson Murphy-Hill},
title = {Why Can{\textquoteright}t Johnny Fix Vulnerabilities: A Usability Evaluation of Static Analysis Tools for Security},
booktitle = {Sixteenth Symposium on Usable Privacy and Security (SOUPS 2020)},
year = {2020},
isbn = {978-1-939133-16-8},
pages = {221--238},
url = {https://www.usenix.org/conference/soups2020/presentation/smith},
publisher = {USENIX Association},
month = aug

Presentation Video