Standard Metrics and Scenarios for Usable Authentication

There is a constant flow of new authentication schemes proposed in the literature. In the past, most proposed schemes were not evaluated empirically, though in recent years there has been an increase in the number of authentication systems that have undergone a user study. Still, most of these user studies employ ad-hoc metrics (e.g., task completion time) and a unique scenario. Bonneau et al. included usability criteria in their heuristic evaluation of various types of web authentication mechanisms.

The use of ad hoc and disparate metrics and scenarios makes it difficult to compare the relative merit of these various proposals. This produces disjointed results that hinder our ability to make more rapid, scientific progress toward usable authentication systems. Based on our experience, we believe that the community would benefit significantly from the adoption of standard metrics and scenarios for use in the empirical evaluation of authentication schemes.