PhishTime: Continuous Longitudinal Measurement of the Effectiveness of Anti-phishing Blacklists


Adam Oest, Yeganeh Safaei, and Penghui Zhang, Arizona State University; Brad Wardman and Kevin Tyers, PayPal; Yan Shoshitaishvili and Adam Doupé, Arizona State University; Gail-Joon Ahn, Arizona State University, Samsung Research


Due to their ubiquity in modern web browsers, anti-phishing blacklists are a key defense against large-scale phishing attacks. However, sophistication in phishing websites—such as evasion techniques that seek to defeat these blacklists—continues to grow. Yet, the effectiveness of blacklists against evasive websites is difficult to measure, and there have been no methodical efforts to make and track such measurements, at the ecosystem level, over time.

We propose a framework for continuously identifying unmitigated phishing websites in the wild, replicating key aspects of their configuration in a controlled setting, and generating longitudinal experiments to measure the ecosystem’s protection. In six experiment deployments over nine months, we systematically launch and report 2,862 new (innocuous) phishing websites to evaluate the performance (speed and coverage) and consistency of blacklists, with the goal of improving them.

We show that methodical long-term empirical measurements are an effective strategy for proactively detecting weaknesses in the anti-phishing ecosystem. Through our experiments, we identify and disclose several such weaknesses, including a class of behavior-based JavaScript evasion that blacklists were unable to detect. We find that enhanced protections on mobile devices and the expansion of evidence-based reporting protocols are critical ecosystem improvements that could better protect users against modern phishing attacks, which routinely seek to evade detection infrastructure.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

@inproceedings {255344,
author = {Adam Oest and Yeganeh Safaei and Penghui Zhang and Brad Wardman and Kevin Tyers and Yan Shoshitaishvili and Adam Doup{\'e}},
title = {{PhishTime}: Continuous Longitudinal Measurement of the Effectiveness of Anti-phishing Blacklists},
booktitle = {29th USENIX Security Symposium (USENIX Security 20)},
year = {2020},
isbn = {978-1-939133-17-5},
pages = {379--396},
url = {},
publisher = {USENIX Association},
month = aug

Presentation Video