Code Reviewing as Methodology for Online Security Studies with Developers - A Case Study with Freelancers on Password Storage


Anastasia Danilova, Alena Naiakshina, and Anna Rasgauski, University of Bonn; Matthew Smith, University of Bonn, Fraunhofer FKIE


While ample experience with end-user studies exists, only little is known about studies with software developers in a security context. In past research investigating the security behavior of software developers, participants often had to complete programming tasks. However, programming tasks require a large amount of participants' time and effort, which often results in high costs and small sample sizes. We therefore tested a new methodology for security developer studies. In an online study, we asked freelance developers to write code reviews for password-storage code snippets. Since developers often tend to focus on functionality first and security later, similar to end users, we prompted half the participants for security. Although the freelancers indicated that they feel responsible for security, our results showed that they did not focus on security in their code reviews, even in a security-critical task such as password-storage. Almost half the participants wanted to release the insecure code snippets. However, we found that security prompting had a significant effect on the security awareness. To provide further insight into this line of work, we compared our results with similar password-storage studies containing programming tasks, and discussed code reviewing as a new methodology for future security research with developers.

SOUPS 2021 Open Access Videos Sponsored by

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

@inproceedings {274453,
author = {Anastasia Danilova and Alena Naiakshina and Anna Rasgauski and Matthew Smith},
title = {Code Reviewing as Methodology for Online Security Studies with Developers - A Case Study with Freelancers on Password Storage},
booktitle = {Seventeenth Symposium on Usable Privacy and Security (SOUPS 2021)},
year = {2021},
isbn = {978-1-939133-25-0},
pages = {397--416},
url = {},
publisher = {USENIX Association},
month = aug

Presentation Video