SoK: I Have the (Developer) Power! Sample Size Estimation for Fisher's Exact, Chi-Squared, McNemar's, Wilcoxon Rank-Sum, Wilcoxon Signed-Rank and t-tests in Developer-Centered Usable Security


Anna-Marie Ortloff and Christian Tiefenau, University of Bonn; Matthew Smith, University of Bonn and Fraunhofer FKIE

Awarded Distinguished Paper!


A priori power analysis would be very beneficial for researchers in the field of developer-centered usable security since recruiting developers for studies is challenging. Power analysis allows researchers to know how many participants they need to test their null hypotheses. However, most studies in this field do not report having conducted power analysis. We conducted a meta-analysis of 54 top-tier developer study papers and found that many are indeed underpowered even to detect large effects. To aid researchers in conducting a priori power analysis in this challenging field, we conducted a systematization of knowledge to extract and condense the needed information. We extracted information from 467 tests and 413 variables and developed a data structure to systematically represent information about hypothesis tests, involved variables, and study methodology. We then systematized the information for tests with categorical independent variables with two groups, i.e., Fisher's exact, chi-squared, McNemar's, Wilcoxon rank-sum, Wilcoxon signed-rank, and paired and independent t-tests to aid researchers with power analysis for these tests. Additionally, we present overview information on the field of developer-centered usable security and list recommendations for suitable reporting practices to make statistical information for power analysis and interpretation more accessible for researchers.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

@inproceedings {289502,
author = {Anna-Marie Ortloff and Christian Tiefenau and Matthew Smith},
title = {{SoK}: I Have the (Developer) Power! Sample Size Estimation for Fisher{\textquoteright}s Exact, {Chi-Squared}, {McNemar{\textquoteright}s}, Wilcoxon {Rank-Sum}, Wilcoxon {Signed-Rank} and t-tests in {Developer-Centered} Usable Security},
booktitle = {Nineteenth Symposium on Usable Privacy and Security (SOUPS 2023)},
year = {2023},
isbn = {978-1-939133-36-6},
address = {Anaheim, CA},
pages = {341--359},
url = {},
publisher = {USENIX Association},
month = aug