On recruiting and retaining users for security-sensitive longitudinal measurement panels


Akira Yamada, KDDI Research, Inc. and National Institute of Information and Communications Technology; Kyle Crichton, Carnegie Mellon University; Yukiko Sawaya, KDDI Research, Inc.; Jin-Dong Dong and Sarah Pearman, Carnegie Mellon University; Ayumu Kubota, KDDI Research, Inc.; Nicolas Christin, Carnegie Mellon University


Many recent studies have turned to longitudinal measurement panels to characterize how people use their computing devices under realistic conditions. In these studies, participants' devices are instrumented, and their behavior is closely monitored over long time intervals. Because such monitoring can be highly intrusive, researchers face substantial challenges recruiting and retaining participants.

We present three case studies using medium- to large-scale longitudinal panels, which all collect privacy- and security-sensitive data. In evaluating factors related to recruitment, retention, and data collection, we provide a foundation to inform the design of future long-term panel studies.

Through these studies, we observe that monetary and non-monetary incentives can be effective in recruiting panel participants, although each presents trade-offs and potential biases. Contrary to our initial expectations, we find that users do not behave any differently in their first few weeks of participation than in the remainder of their time in the study. In terms of retention, we note that personalized enrollment follow-ups can lower initial dropout rates, but they are challenging and costly to scale. Communication, including following up with inactive users, is vital to retention. However, finding the right balance of communication is equally important. Interfering with a participant's everyday device use is a sure way to lose users. Finally, we present several findings, based on practical experience, to help inform the design of the data collection process in observational panels.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

@inproceedings {281266,
author = {Akira Yamada and Kyle Crichton and Yukiko Sawaya and Jin-Dong Dong and Sarah Pearman and Ayumu Kubota and Nicolas Christin},
title = {On recruiting and retaining users for security-sensitive longitudinal measurement panels},
booktitle = {Eighteenth Symposium on Usable Privacy and Security (SOUPS 2022)},
year = {2022},
isbn = {978-1-939133-30-4},
address = {Boston, MA},
pages = {347--366},
url = {https://www.usenix.org/conference/soups2022/presentation/yamada},
publisher = {USENIX Association},
month = aug,