If You Can’t Get Them to the Lab: Evaluating a Virtual Study Environment with Security Information Workers


Nicolas Huaman, Alexander Krause, and Dominik Wermke, CISPA Helmholtz Center for Information Security; Jan H. Klemmer and Christian Stransky, Leibniz University Hannover; Yasemin Acar, George Washington University; Sascha Fahl, CISPA Helmholtz Center for Information Security


Usable security and privacy researchers use many study methodologies, including interviews, surveys, and laboratory studies. Of those, lab studies allow for particularly flexible setups, including programming experiments or usability evaluations of software. However, lab studies also come with challenges: Often, it is particularly challenging to recruit enough skilled participants for in-person studies. Especially researchers studying security information workers reported on similar recruitment challenges in the past. Additionally, situations like the COVID-19 pandemic can make in-person lab studies even more challenging. Finally, institutions with limited resources may not be able to conduct lab studies. Therefore, we present and evaluate a novel virtual study environment prototype, called OLab, that allows researchers to conduct lab-like studies remotely using a commodity browser. Our environment overcomes lab-like study challenges and supports flexible setups and comprehensive data collection. In an iterative engineering process, we design and implement a prototype based on requirements we identified in previous work and conduct a comprehensive evaluation including a cognitive walkthrough with usable security experts, a guided and supervised online study with DevOps, and an unguided and unsupervised online study with computer science students. We can confirm that our prototype supports a wide variety of lab-like study setups and received positive feedback from all study participants.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

@inproceedings {281220,
author = {Nicolas Huaman and Alexander Krause and Dominik Wermke and Jan H. Klemmer and Christian Stransky and Yasemin Acar and Sascha Fahl},
title = {If You {Can{\textquoteright}t} Get Them to the Lab: Evaluating a Virtual Study Environment with Security Information Workers},
booktitle = {Eighteenth Symposium on Usable Privacy and Security (SOUPS 2022)},
year = {2022},
isbn = {978-1-939133-30-4},
address = {Boston, MA},
pages = {313--330},
url = {https://www.usenix.org/conference/soups2022/presentation/huaman},
publisher = {USENIX Association},
month = aug,

Presentation Video