Write, Read, or Fix? Exploring Alternative Methods for Secure Development Studies

Authors: 

Kelsey R. Fulton, Colorado School of Mines; Joseph Lewis, University of Maryland; Nathan Malkin, New Jersey Institute of Technology; Michelle L. Mazurek, University of Maryland

Abstract: 

When studying how software developers perform security tasks, researchers often ask participants to write code. These studies can be challenging because programming can be time-consuming and frustrating. This paper explores whether alternatives to code-writing can yield scientifically valid results while reducing participant stress. We conducted a remote study in which Python programmers completed two encryption tasks using an assigned library by either writing code from scratch, reading existing code and identifying issues, or fixing issues in existing code. We found that the read and fix conditions were less effective than the write condition in revealing security problems with APIs and their documentation, but still provided useful insights. Meanwhile, the read and especially fix conditions generally resulted in more positive participant experiences. Based on these findings, we make preliminary recommendations for how and when researchers might best use all three study design methods; we also recommend future work to further explore the uses and trade-offs of these approaches.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

BibTeX
@inproceedings {298860,
author = {Kelsey R. Fulton and Joseph Lewis and Nathan Malkin and Michelle L. Mazurek},
title = {Write, Read, or Fix? Exploring Alternative Methods for Secure Development Studies},
booktitle = {Twentieth Symposium on Usable Privacy and Security (SOUPS 2024)},
year = {2024},
isbn = {978-1-939133-42-7},
address = {Philadelphia, PA},
pages = {81--100},
url = {https://www.usenix.org/conference/soups2024/presentation/fulton},
publisher = {USENIX Association},
month = aug
}

Presentation Video