User Study: Comparison of Picture Passwords and Current Login Approaches

We have compared two authentication methods: regular password and picture password. We all use the regular passwords daily to authenticate to our services. Picture password is a similar concept where we define the password in terms of pictures on the screen instead of characters on a keyboard. We select a sequence of pictures from a set of thirty pictures.

The independent factor of this study is the ”authentication method.” In this case, the two authentication factors are ”regular password” and ”picture password .” Picture passwords can be either randomized picture order or non-randomized, and they can also have a theme or be single picture mode. This gives us a total of four possible picture password configurations for five possible interfaces. We have decided to test three authentication methods: Regular Password, Picture Password with an animal theme, and Picture Password with a single image. Hereafter, these methods may be referenced as classic, them, and single, respectively.

As our research focuses on finding the best-suited authentication factors for the common technology user, we conducted our study with various individuals representing a technology-using population. We will not focus on only including tech-savvy or individuals in cybersecurity fields. This, however, does not mean that our group of participants did not include such individuals. We aim to capture realistic measurements for a regular technology consumer and capture a diverse selection of male and female college-level educated individuals. Individuals were recruited through word of mouth at the University of Nevada, Reno campus.

We conducted this study with 12 participants; we discuss how these are divided in a future section.

We propose the following questions to be answered by this study:

H1. Picture passwords reduce password fatigue/stress.

H1.2 Users prefer to use picture passwords to picture passwords for simplicity.

H1.2.1 Users have an increased perception of increased security by using picture passwords, compared to regular passwords.

H1.3 Picture passwords with a single picture theme can reduce password fatigue.

H1.4 The users prefer Theme Picture Password theme mode interface.

H1.5 The users prefer Single Picture Password theme mode interface.

H2. Users can log in faster using picture passwords than regular passwords. We will measure account creation, login time, and error rate(described later).

H3. Users can retain more picture passwords than regular passwords. Test users on creating multiple accounts and see if adding one more account will make them lose retention of the previously learned account, and find that breaking point.

H4. Password reuse is reduced in picture passwords. We will test it by having them come up with multiple passwords in short succession and measure the similarity between passwords. How much do these passwords vary from the set? We will use a similarity score for their created set.

H5. Password sharing is significantly reduced with picture passwords. We share a password with the user and let them type it in, measuring their error rates.

H6. Picture passwords for Account creation are preferred.

H7. Users have a more straightforward use with picture passwords (frustration due to error); in other words, it reduces errors. We will check for error rates over tries. (error-rate/total-submits)

H8. On average more users remembered passwords within 20 minutes of making (after being asked to do other tasks) them with picture passwords (T2a,b).

H9. The amount of user interaction in the picture password interface is reduced (interaction score = clicks+keys). This could lead to the reduction of the perception of load.

H10. The number of mistakes (user identified) is reduced for picture passwords (number of passwords cleared or backspace used).

H11. The complexity of passwords is inherently higher with picture passwords with the only requirement being eight in length. Rank password complexity into the score and look into password shortcuts or patterns that make it inherently insecure.

In the task descriptions (below), if it describes a password, picture, or both, it means that the task will be repeated using a password, picture password, or both. This does not mean we will mix both passwords in the same task; it means users will have to execute the same task using different password interfaces. Each task was designed to correspond to one or more of the above hypotheses for testing purposes.

We will measure metrics for all account creation and login tasks. These metrics include time to successful completion time, number of times the participant used the system incorrectly, which will be measured in backspace usage, password clear button usage, and incorrect number of attempts. Other metrics include the number of buttons and clicks pressed. We will also use an algorithm to score the password sets entered by the user to check for similarity within the set.

1. (1)

   When participant agrees to the study, they will sign up for a time slot with the study coordinator.

2. (2)

   Two days before the study, an instructional document will be sent to participant, which they can review at their leisure.

3. (3)

   The participant meets up with the study coordinator at the research lab at the agreed time slot.

4. (4)

   They will receive a pre-study questionnaire that they will be requested to fill out (virtual document).

5. (5)

   Once complete, they will receive instructions on the study from the study coordinator on how to perform the tasks. The coordinator will also demonstrate how to use the interfaces.

6. (6)

   The participant will be asked if they would like to practice using the Picture Password interface. If they do, they will be given access to the demonstration interface previously shown (5 minutes maximum).

7. (7)

   Once the instructions are provided, the study begins.

8. (8)

   The participant will be provided an account creation interface. The interface will be different given the independent variable being tested.

9. (9)

   The participant must quickly create the account. (This account is not tied to anything and user will be told explicitly not to use any of his current account information for this study.)

10. (10)

    After account creation, the user will be instructed to log in using one of the interfaces being tested.

11. (11)

    Once they have successfully done so, they must perform the experiment again with different parameters.

12. (12)

    The study software will navigate the participants between each of the configurations and tasks during this sequence.

13. (13)

    Once the participants are done with all the tests above, the user study has concluded.

14. (14)

    The participants are then asked to provide feedback in a post-study questionnaire (virtual document). During this step, participants can provide quantitative and qualitative feedback on their experiment experience alongside suggestions for improvement.

15. (15)

    Once the post-study questionnaire is completed, the participant’s job is complete, and they may conclude their participation.

Here is a more extensive list of tasks that correspond to steps 7-10:

T1. Users will be asked to create an account (using either password, picture, or both). This will be an ongoing task that will test the user’s ability to recall passwords after some time or other tasks; Users will be prompted to remember them. Users will be asked to enter the password at the end of all other tasks. We will measure time, incorrect usage, clicks, button presses, backspaces, clears, and errors.

T2. Users will be asked to create a single password (either password, picture, or both). The user will then be asked to write it down as he was going to share it with someone (certain rules are set here). Next, the user will be asked to forget about the password. Towards the end of the session with the user, he will be asked to read his or her notes and log in. The same metrics will be measured as all other tasks.

T4. The participant will be asked to create an account (password, picture, or both), and metrics will be taken on this account creation task. The user will be asked to log in. The participant will be asked to log in five times using the same credentials (for each method). The same metrics will be measured as all other tasks. In this task, we test the account creation speed and log-in speeds more specifically.

T5. If the user is using picture password for T4. The picture password location will be randomized to test added security. The participant will be asked to log in another 5 times. This time the picture locations will be randomized and not the same as when they created them or previously logged in.

T6. Users will be asked to create an account and log in using specific password hardness/security requirements. We will then test the highest level of security so users can still successfully log in by increasingly asking them to increase their password security. The highest level of security will be a metric we will measure, as well as all other tasks.

T6. The participant will be asked to create a password (either password, picture, or both). We will then test for retention when multiple passwords are being memorized. We ask the participant to log in. We will then ask the user to create one more account and ask them to log in to all of the previously created accounts on this task. We will repeat the cycle of adding one more account until the user can no longer successfully log in to the accounts. The same metrics will be measured as all other tasks. In this case, the number of successfully retained accounts will also be another metric we will use.

T7. Predefined accounts will be created before user sessions are held. These will include a range of regular and picture passwords. Participants will be given a post-it or piece of paper with the password, which will simulate someone handing them the password. Participants will be asked to log in using that information. The same metrics will be measured as for all other tasks.

The entry and exit questionnaires are located in the appendix of this paper.

Users were asked to conduct account creation and login tasks as described in the previous section. This user study was conducted with IRB approval. We used a Dell Optiplex 7070 computer to conduct the user study. We installed Linux operating system. We also implemented a custom software interface that implements account creation, login and metric tracking for both regular passwords and picture passwords. The picture password interface implemented is described in the NIST NISTIR 7030 documentation standard (NIST, 2022). The setup can be seen in figure 1, and the interface samples can be seen in figures 2,3 and 4.

The user study was conducted in a quiet room side room of a campus research lab. This place was chosen because we have access to it exclusively. Therefore, we can conduct all the studies in a controlled and undisturbed manner.

Before the participants were convened to go through the study, we had two participants go through the study to find any possible improvement to the study workflow as a pilot demo. These two individuals were not part of the main study and were purely to test the instructions, software, and surveys before they were used with the rest of the participants. We also used these trial runs to adjust the session lengths, number of interfaces per user, and number of tasks.

• •

  A participant arrives at the lab; we greet them and guide them to the testing room where the computer is.

• •

  We briefed them. In this briefing, we asked the user to conduct the pre-study survey.

• •

  After a participant finished the survey, we let them know what we needed them to do in this study.

• •

  We instructed them to create accounts and login in a repeated way, and we need them to do it in a precise and concise manner.

• •

  The examiner asked the participant if he or she had any questions and answered them in a controlled manner.

• •

  We started by showing participants a picture password; we demonstrated what the picture password is and how it works.

• •

  We let the user do 5 minutes of practice with the picture password interface (all participants opted out).

• •

  We started the tasks as described in the task section. The user was asked to create accounts and login into the accounts using three different login methods. The software was self-guided with instructions.

• •

  After the participant is finished, the software prompted them to seek the examiner. The examiner handed the participant

• •

  the post-study survey, which they completed.

• •

  The examiner asked the participants if he or she had any questions.

• •

  The participants were thanked for their time and were dismissed.

We have gathered the following metrics (dependent variables) for all login and account creation tasks:

• •

  Total time (per action and task)

• •

  Error

• •

  Password retention

• •

  Password shareability

• •

  number of clicks

• •

  Key-presses

• •

  Number of Enter key presses

• •

  Number of Backspace key presses

• •

  Number of Clear (erasing password)

• •

  Number of skipped logins, tracks giving up

• •

  Total number of submissions

• •

  Number of incorrect submissions

• •

  Number of rounds for Task 6 (we can also use this to tack persistence)

• •

  Passwords and usernames used

We have additionally calculated the $errorrate$ as $incorrectlySubmitted/totalSubmitted$.

For non-parametric dependant variables, we collected the following using the questioners:

• •

  Stress/Fatigue levels

• •

  Perception of security

• •

  Preference of interface (in two forms)

• •

  Password

The data collected was more than the data we analyzed in the next section. Since this study studies a single independent variable with three levels, we used between subject groups with counterbalancing as shown in Table 1. We chose to use this method because we felt that the Picture Password was already at a disadvantage since Regular Password is used by all users in their normal lives.

For the remainder of this paper, Regular, Picture with theme pictures, and Picture with a single picture will be addressed as classic, theme, and single, respectively.

We have collected data to analyze regarding all of the hypothesis aforementioned, but due to time constraints, we will only look at results originating from data regarding the following hypothesis:

H1. Picture passwords reduce password fatigue/stress.

H1.2.1 Users have an increased perception of increased security by using picture passwords, compared to regular passwords.

H1.4 The users prefer Theme Picture Password theme mode interface.

H1.5 The users prefer Single Picture Password theme mode interface.

H2. Users can log in faster using picture passwords than regular passwords.

H7. Users have a more straightforward use with picture passwords (frustration due to error); in other words, it reduces errors. We will check for error rates over tries. (error-rate/total-submits)

In this analysis section, we only look at account creation time (average across tasks), logging time (Task 3), error rate across all tasks, and the results from the questionnaire.

The results are shown in Tables 2, 3,4, 5 and their corresponding histograms in Fig. 5, 6, 7. By observing each group’s means by data type, we can see that counterbalancing did not work (Discussed on the Discussion Section).

We conducted One-Way ANOVA test on the creation time, log-in time, and error rates.

The mean account creation time for Classic was 30.14, Theme was 42.77, and Single was 46.32. The difference was statistically significant ($F_{2,33}=0.88,p<0.5$).

By further investigating the data, as seen in Tables 6, 7, not all are normally distributed. This distribution has a major effect on the results and tests. We have removed some of the outliers and we can see in Tables 8, 9 that normality improves on both Classic and Single. We can also see that Theme data is heavily right-skewed. The difference was statistically significant ($F_{2,30}=3.13,p\approx 0.05$). We furthered by executing another ANOVA test, as seen on Tables 10, 11 on just Classic and Single. The difference was statistically significant ($F_{1,19}=0.35,ns$). From these tests, we can conclude that we fail to reject $H_{0}$ and fail to prove H2.

As seen in Fig. 3,12, Task 3 mean account log-in time for Classic was 6.22, Theme was 5.68, and Single 6.39. The difference was not statistically significant ($F_{2,33}=0.88,ns$). We can conclude that we fail to reject $H_{0}$ and fail to prove H2.

As seen in Tables. 4,13, the data does not follow a normal distribution; therefore, ANOVA is not a suitable test for this data. Meaning we fail to prove H7.

In this subsection, we further analyze data from Fig. 5.

We will start by looking into H1.2.1 perception of security. The average for both regular passwords and picture passwords is 3.42 on a scale of 1-5 (1=not safe, 5=most safe). We conducted Wilcoxon signed-ranks test (two-tailed). We conclude that with a $\mu 0=0$(no difference) the difference between classic and picture is not big enough to be statistically significant ($Z=-0.073,p=0.94,ns$). Meaning we can not reject the null hypothesis of them being equal. This means that users perceive both regular passwords and picture passwords with equal levels of security; the data also confirms this. Therefore, we reject H1.2.1.

Next, we will discuss results relating H1.4 and H1.5, preference for picture passwords over regular passwords. Initial observations of the data may lead to believe that users prefer Theme over Single or Classic as seen in Figures 8, 8, these results originate from four different questions. We conducted Friedman Test for Repeated-Measures and we concluded that it is not statistically significant with $p=.28$ ($\chi^{2}_{r}=2.54,ns$). Therefore we fail to reject null hypothesis and can not conclude H1.4 or H1.5.

. From the reported qualitative data and figures 10, 11, and 12 we can conclude the majority of users in this study would use picture passwords as MFA and would not use them instead of regular passwords. Users reported more password fatigue/stress from picture passwords. We have not conducted significant statistical tests on data for stress/fatigue, likelihood to use picture passwords as MFA, or likelihood to use picture passwords if available. Therefore, we can not say whether the results are true or false (in this case, we can not conclude H1)