Replication: Challenges in Using Data Logs to Validate Phishing Detection Ability Metrics


Casey Canfield, Alex Davis, and Baruch Fischhoff, Carnegie Mellon University; Alain Forget, Google; Sarah Pearman and Jeremy Thomas, Carnegie Mellon University


The Security Behavior Observatory (SBO) is a longitudinal fieldstudy of computer security habits that provides a novel dataset for validating computer security metrics. This paper demonstrates a new strategy for validating phishing detection ability metrics by comparing performance on a phishing signal detection task with data logs found in the SBO. We report: (1) a test of the robustness of performance on the signal detection task by replicating Canfield, Fischhoff, and Davis (2016), (2) an assessment of the task's construct validity, and (3) evaluation of its predictive validity using data logs. We find that members of the SBO sample had similar signal detection ability compared to members of the previous mTurk sample and that performance on the task correlated with the Security Behavior Intentions Scale (SeBIS). However, there was no evidence of predictive validity, as the signal detection task performance was unrelated to computer security outcomes in the SBO, including the presence of malicious software, URLs, and files. We discuss the implications of these findings and the challenges of comparing behavior on structured experimental tasks to behavior in complex real-world settings.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

Presentation Audio

@inproceedings {205126,
author = {Casey Canfield and Alex Davis and Baruch Fischhoff and Alain Forget and Sarah Pearman and Jeremy Thomas},
title = {Replication: Challenges in Using Data Logs to Validate Phishing Detection Ability Metrics},
booktitle = {Thirteenth Symposium on Usable Privacy and Security ({SOUPS} 2017)},
year = {2017},
isbn = {978-1-931971-39-3},
address = {Santa Clara, CA},
pages = {271--284},
url = {},
publisher = {{USENIX} Association},