We used RUST to conduct usability study sessions with Windows CardSpace and Verisign Secure Letterhead. Each of the technologies approach the website authentication problem differently. CardSpace changes the login procedure, while Secure Letterhead provides the user with additional information to help them determine the website’s identity.
For recruitment we solicited participants on Craigslist, placed a listing in the Facebook Marketplace and posted fliers at Columbia University. The only requirement that we places on participants is that they had experience with web browsing.
CardSpace is an identity metasystem that manages a user’s online identities [5], replacing usernames and passwords. When a user needs to log in to a website, the user clicks on a button in the page content that begins the login process. At this point, CardSpace is launched and the website’s credentials are sent. When the CardSpace interface appears, the user selects the appropriate ”card” that represents their credentials. The user does not submit any personal information to the website. Instead, the exchange of credentials is handled completely within the CardSpace protocol once the ”card” representing the user’s identity is chosen by the user. Since the protocol is designed to reveal information only to the verified parties, tricking the user into giving away their credentials is less feasible and different attacks are required. As a result, we designed spoofs that focus on the user enterinig sensitive information directly to the website. One email stated CardSpace was down for scheduled maintenance and directed the user to a spoofed page to signup for temporary access. The user was then sent to a webpage requesting sensitive information. The other spoof redirected the user to a form when they attempted to login with CardSpace, stating the participant must register for an identity card by entering personal information. Since the user logged into the same institution’s website with a identity card previously, this should be alarming.
We recruited a total of 13 participants to evaluate CardSpace, 4 female and 9 male. Their ages ranged from 18-60 and each of them spend 20 or more hours per week on the Internet.
Of the eight tasks given in a session, the four tasks prior to the CardSpace instructional email directed the participant to: a real website with CardSpace, a spoofed site that asked the participant to register for a new identity card, a real website with CardSpace, and a spoofed website that stated CardSpace was down for maintenance. The four remaining tasks were in the same order and the first of the second set of four was also the instructional email.
In the first task, on the real website, 12 of the 13 participants reported some level of confusion. However, 6 participants stated in the post-study questionnaire that CardSpace was intuitive, despite commenting they were confused after the first task. Their comments after the first task include: ”It took me a little while to know where I was suppose to click”, ”I wasn’t asked to enter account info which was suspicious”, ”I understood the task, but the first thing I should have been asked is my account id and password”, ”when I clicked on the log-in tab I was a little confused, I am accustomed to seeing a user ID box and then some type of password, but what popped up were cards, there seems to be some serious security lapse”, and ”when I clicked login something irrelevant came up but when I hit ok I was logged in”.
In the first spoof, when participants were told to register for a new card, 11 participants completed the task without noticing anything suspicious. One person realized they had been tricked when they were redirected to the real site but this was after entering personal information. In the fourth task when the spoofed website stated CardSpace was “down for maintenance”, the same 11 participants fell for the spoof. The participant who realized they were spoofed previously recognized the spoof without entering any information. After reading the instructional email, two people reported they were still confused when completing the task that followed on the real site. By the fourth task, the second task on a real website, no one commented they were confused.
After reading the instructional email, 3 of the 13 participants did not fall for the spoof that occurred immediately after (a request to register for a new card). One participant was previously an identity theft victim and the other two participants cited the instructional email as their reason for not completing the task. The final spoof stated CardSpace was down for maintenance, and 3 people were not tricked by it. The identity theft victim was not tricked and one person cited the instructions. However, one person who cited the instructions in the previous spoof fell for the second spoof. The other person remarked the site didn’t look right in general, so they refused to complete the task. In the post-study questionnaire, four people said the CardSpace login procedure required slightly too much time and six reported the amount of time required by CardSpace was just right.
Overall, the participants were mostly confused that CardSpace took the place of a username and password. The spoof that stated CardSpace was down for maintenance tricked 11 of the 13 participants and after the instructional email the same spoof tricked 10 of the 13. The spoof that asked the participant to register for another card was successful on 11 of the 13 participants and after the instructional email successfully tricked 10 of the 13.
Secure letterhead assists the user by displaying security context information interactively and more prominently in the web browser’s primary interface. The implementation we tested was Firefox extension and displayed the logotype and certificate authority fields from an extended validation X.509 certificate [8, 9]. When the user navigates to a website that has an extended validation certificate, the logotype is displayed in the upper right hand corner of the browser next to the location bar. It extracts relevant fields from the SSL certificate, which can be difficult for users to find and understand. The fields are displayed as an interactive visual indicator. More certificate information is shown when user clicks on the logotype.
We conducted five sessions with Secure Letterhead, two female and three male. Their ages ranged from 18-50 and all reported to spend 20 or more hours on the Internet per week. Only one of the users demonstrated prior knowledge of phishing.
The four tasks prior to the instructional email that detailed how to use Secure Letterhead directed the user to a real site, a fake site where no additional information was displayed, a real site, and a fake site spoofing Secure Letterhead with CSS and HTML. The four remaining task emails were in the same order except the first of the second set was also the instructional email.
Since Secure Letterhead does not alter the login process, the first task went smoothly for the participants, however, no one noticed the presence of the logotype or tried to click on it. The first spoof, where no additional information was presented, tricked all 5 participants. One participant realized they had been tricked and wasn’t tricked again during the study. However, Secure Letterhead did not play a role in their realization; instead they realized they were tricked after logging in and being forwarded to the real login page. Again, the task on the real website went smoothly, but no one noticed the logotype. The second spoof, where the interface was recreated using CSS and HTML, was successful in tricking 4 of the 5 participants.
Every participant attempted to access the information in the logotype after reading the instructional email. One commented the instructions were incorrect and the logo was on the left, indicating they clearly misunderstood which logo the instructions referred to and mistook an image in the webpage content for the logotype. Another did not to complete the task and wrote they didn’t have a reason to trust the information it displayed. Another commented the instructions were too wordy, then during the task attempted to access the information by clicking on the logo in the page content. This behavior suggests users are unable to distinguish between page content and trusted indicators, which was also observed by Whalen et al. [15] and Dhamija et al. [6].
The first spoof after the instructional email, where no additional information was displayed, tricked 4 of the 5 participants. In the remaining three tasks, only two attempted to access the information. One of them attempted to access the information after being redirected to the real site because they had already entered their credentials on the illegitimate site. In other words, they checked the credentials at the wrong point of interaction and had already lost their password. The other one checked the logotype on the real website. The final spoof with the recreation of Secure Letterhead in content successfully tricked 4 of the 5 participants.
In the post-study questionnaire, 3 of the 5 participants reported they would remember to check for the logotype if they were about to do something important. 2 of the 5 participants stated they would not have figured out how to use Secure Letterhead without instructions. One person commented the information displayed did not seem useful, and two commented the information might be useful but may not be necessary.
Overall, it appeared as though the participants were unsure of how to interpret the information presented by Secure Letterhead, why it was important, or why they should trust it. These comments were made in the post-task questionnaire after receiving the instructional email and in the post-study questionnaire.