Measuring the Effectiveness of Embedded Phishing Exercises


Hossein Siadatii, New York University; Sean Palka and Avi Siegel, Booz Allen Hamilton; Damon McCoy, New York University


Embedded phishing exercises, which send test phishing emails, are utilized by organizations to reduce the susceptibility of its employees to this type of attack. Research studies seeking to evaluate the effectiveness of these exercises have generally been limited by small sample sizes. These studies have not been able to measure possible factors that might bias results. As a result, companies have had to create their own design and evaluation methods, with no framework to guide their efforts. Lacking such guidelines, it can often be difficult to determine whether these types of exercises are truly effective, and if reported results are statistically reliable.

In this paper, we conduct a systematic analysis of data from a large real world embedded phishing exercise that involved 19,180 participants from a single organization, and utilized 115,080 test phishing emails. The first part of our study focuses on developing methodologies to correct some sources of bias, enabling sounder evaluations of the efficacy of embedded phishing exercises and training. We then use these methods to perform an analysis of the effectiveness of this embedded phishing exercise, and through our analysis, identify how the design of these exercises might be improved.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

@inproceedings {205853,
author = {Hossein Siadati and Sean Palka and Avi Siegel and Damon McCoy},
title = {Measuring the Effectiveness of Embedded Phishing Exercises},
booktitle = {10th USENIX Workshop on Cyber Security Experimentation and Test (CSET 17)},
year = {2017},
address = {Vancouver, BC},
url = {},
publisher = {USENIX Association},
month = aug