Think-Aloud Verbalizations for Identifying User Experience Problems: Effects of Language Proficiency with Chinese Non-Native English Speakers

When using concurrent think-aloud (TA) protocols, participants verbalize their thought processes while working on the tasks at the same time. Depending on the types of prompts or interventions administered by the study moderator, there are three common types of concurrent TA protocols: the Classic Think-Aloud protocol (CTA), the Speech Communication protocol (SC), and the Interactive Think-Aloud protocol (ITA).

Classic Think-Aloud (CTA): CTA was established as a valid approach to studying human thinking processes by Ericsson and Simon (Ericsson and Simon, 1984) and later introduced into the fields of HCI and UX to study UX problems. Ericsson and Simon proposed a set of guidelines for conducting CTA: have a practice think-aloud session before the actual study session; use neutral instructions that do not direct participants to verbalize a specific type of thought process; keep prompts and interventions to a minimum by only reminding participants to “keep talking” if they fall into silence for a period.

Speech Communication protocol (SC): Realizing the unnaturalness of thinking aloud and helping to promote thinking aloud in usability testing, Boren and Ramey found that CTA can be hard to execute by UX evaluators in practice (Boren and Ramey, 2000) and proposed the SC protocol, which asked the moderator to play an active listener role by using tokens, such as “Em hmm”, “And now? …”, in addition to “keep talking”, to help participants think aloud.

Interactive Think-Aloud (ITA): In practice, it is not uncommon that the moderator actively asks participants questions during think-aloud usability test sessions to inquire about their opinions, explanations, or suggestions (Hertzum and Kristoffersen, 2018; Rubin and Chisnell, 2008; Dumas et al., 1999). This variation of the concurrent TA protocols was often referred to as interactive think-aloud (ITA).

Since prior studies showed the pros and cons of these three TA protocols (e.g.,  (Olmsted-Hawala et al., 2010b, c; Alhadreti and Mayhew, 2017)), we included all the three TA protocols to understand whether these TA protocols affect native and non-native English speakers’ verbalizations.

When conducting think-aloud studies in English, prior research often recruited participants of native or fluent speakers to minimize the potential affects of language proficiency. We conducted a literature review and identified 16 papers that studied think-aloud protocols (Andreasen et al., 2007; Bruun et al., 2009; Cooke, 2010; Elling et al., 2012; Hertzum et al., 2009; Zhao and McDonald, 2010; McDonald and Petrie, 2013; Chalil Madathil and Greenstein, 2011; Olmsted-Hawala et al., 2010a; Thompson et al., 2004; Haak et al., 2004; Alhadreti and Mayhew, 2017; Fan et al., 2020a, 2019; Hertzum et al., 2015; Krahmer and Ummelen, 2004). Eleven of these papers did not specify the language proficiency of their participants (Andreasen et al., 2007; Bruun et al., 2009; Cooke, 2010; Elling et al., 2012; Hertzum et al., 2009; Zhao and McDonald, 2010; McDonald and Petrie, 2013; Chalil Madathil and Greenstein, 2011; Olmsted-Hawala et al., 2010a; Thompson et al., 2004; Haak et al., 2004). The five papers left mentioned that their participants were either native speakers or competent in the language that they spoke in their studies (Alhadreti and Mayhew, 2017; Fan et al., 2020a, 2019; Hertzum et al., 2015; Krahmer and Ummelen, 2004). To the best of our knowledge, no studies have explicitly compared think-aloud verbalizations and UX problems of native and non-native speakers of a language. In this research, we sought to fill in this literature gap.

To understand what participants verbalize in think-aloud sessions, researchers coded participants’ verbalizations and identified different verbalization categories. In an early work, Bowers et al. identified five verbalization categories: Procedure, Explanation, Reading, Design, and Others (Bowers, 1990). Later, Cooke conducted a study using a website and identified five similar categories: Procedure, Reading, Observation, Explanation, and Others (Cooke, 2010). Procedure refers to participants’ verbalizations that describe “participants’ current or future actions”; Reading refers to participants’ verbalizations when they read any information (e.g., link labels, phrases, or sentences) from the test product; Observation refers to participants’ verbalizations when they make remarks or observations about the test product or about themselves; Explanation refers to participants’ verbalizations whey they explain their behaviors with the test product; Others refer to verbalizations that do not fit in the above four categories. These five categories of verbalizations were later confirmed in Elling et al.’s study, in which participants used more websites than Cooke’s study (Elling et al., 2012). Although later studies divided verbalizations into more categories (e.g.,  (Zhao and McDonald, 2010; Hertzum et al., 2015)), these categories could be mapped into the five-category scheme proposed in Cooke’s study (Cooke, 2010). Recently, Fan et al. also adopted Cooke’s five-category scheme to study the correlations between verbalization categories and UX problems among native English speakers (Fan et al., 2019). Following these prior studies, in this study, we also adopted Cooke’s categorization strategy to analyze our participants’ verbalizations and investigated how these verbalization categories indicate UX problems for both non-native and native English speaking groups.

We recruited 18 participants through social media platforms, word-of-mouth, and snowball sampling. All participants were undergraduate or graduate students in US universities except one who recently graduated. One group of participants (N=9) was native English speakers from the US (8) and Canada (1), and the other group of participants (N=9) was Chinese non-native English speakers who studied in US universities.

Table 1 shows the demographic information of the non-native English speakers. In addition to the standard English test scores (e.g., TOEFL, IELTS), the researchers informally assessed participants’ proficiency level based on their conversations with these participants using the Common European Frame of Reference for Languages (CEFR) guidelines (of Europe, 2021). The definitions of the related levels were as follows: B1: intermediate; B2: upper intermediate; C1: effective operational proficiency; C2: proficiency.

We chose three types of websites to increase variations of test products. One was an example of information-rich websites¹¹1https://airandspace.si.edu/ (web 1); one was an example of e-commerce websites²²2https://www.lazada.com.my/ (web 2 ); and the last one was an example of productivity-enhancement websites³³3https://basecamp.com/ (web 3).

Figure 1 shows the homepage of the museum website (Air and Space Museum) website. Figure 2 shows the homepage of the e-commerce website (Lazada). Figure 3 shows the homepage of the teamwork tool website (Basecamp). We chose these websites because 1) they represent common types of websites one would use in their daily life; 2) we conducted heuristic evaluation using both Nielsen heuristics (Nielsen, 1994b) and Norman principles (Norman, 2013) and found that all of these websites contained UX problems; and 3) none of the participants used these websites prior to the study.

We designed corresponding tasks that would require participants to use the features of the websites that contained UX problems. Table 2 shows the websites and the corresponding tasks used in the study.

We conducted the user study with participants online using Zoom. We screen- and audio-recorded study sessions. The study lasted for about an hour.

Upon signing the consent form, the moderator explained to participants that they would complete tasks on three websites and think aloud during the process. To help participants better understand how to think aloud, the moderator first explained verbally how to think aloud and then showed a think-aloud demo video provided by the Nielsen Norman group (NNGroup, 2014), which shows a user working on a website while thinking aloud at the same time. Afterward, the moderator asked participants to practice thinking aloud on a website, which was not one of the test websites, to complete a task. Next, the moderator asked participants to work on the three test websites to complete corresponding tasks in Table 2 while thinking aloud at the same time. For each website, participants used one of the three TA protocols (i.e., CTA, SC, ITA) as explained in Sec 2.1. The order of TA protocols and the test websites were counterbalanced using the Latin square design in Table 3.

When conducting CTA sessions, the moderator followed Ericcson and Simon’s guidelines and did not use any prompts and only reminded participants to “keep talking” if they fell into silence for more than 10 seconds. When conducting SC sessions, the moderator followed the guidelines put forward by Boren and Ramey (Boren and Ramey, 2000) and played an active listener’s role by saying phrases such as “Em, hmm”, “uh-huh”, “and now?”, and “keep talking” to encourage participants to think aloud. When conducting ITA sessions, the moderator actively probed participants using five types of prompts that were derived from prior studies (Alhadreti and Mayhew, 2017; Hertzum and Kristoffersen, 2018; Zhao and McDonald, 2010): Clarifying intentions (“What are you looking for?”), Seeking explanations (“Could you tell me why you did that?”), Seeking opinions (“What do you think of it?”), Seeking suggestions (“What redesign do you suggest?”), and Seeking user expectation (“what do you expect to be there?”).

After participants completed each task, they were asked to fill in the NASA Task Load Index (TLX) form to measure their perceived task load of completing the task while thinking aloud.

We first used an automatic transcribing tool, Otter.ai (Otter.ai, 2020), to transcribe session recordings and then manually checked the transcriptions to correct errors. After this process, we had all participants’ think-aloud verbalizations.

We followed a similar process used in the literature (Fan et al., 2019; Cooke, 2010; Elling et al., 2012; Zhao and McDonald, 2010) to review each think-aloud session recording and segment it into smaller segments based on pauses in the participant’s verbalizations and the semantics of the verbalizations. A segment could include sentences, phrases, or single words.

Next, for each segment, two UX researchers independently reviewed the corresponding verbalizations and assigned it a verbalization category label using Cooke’s five-category scheme (Cooke, 2010) as explained in Sec 2.3. This scheme was widely adopted by prior studies (Elling et al., 2012; Fan et al., 2019; Hertzum et al., 2015; Fan et al., 2021). The five verbalization categories were: Procedure, Reading, Observation, Explanation, and Others. Table 4 shows the categories, their definitions, and examples from our participants’ think-aloud verbalizations.

After each researcher finished assigning category labels for each segment independently, they reviewed their category labels together to gain a consensus on the labels. If there was a disagreement, they explained their rationales for their labels, discussed with each other, and consolidated the category labels.

For each verbalization segment, two UX researchers followed the same procedure of assigning verbalization category labels in Sec 4.1 to determine whether the user encountered a problem and to assign a binary problem label (0: no problem; 1: problem). Each segment with a problem label “1” represents “a moment in which a user encountered a problem” and was referred to as a “problem encounter.” Because different “problem encounters” might be caused by the same underlying UX problem, two UX researchers further reviewed these “problem encounters” and combined the problem encounters caused by the same underlying problem, which was referred to as an “actual problem”. As a result, the number of “actual problems” would be less or equal than the number of “problem encounters.”

Furthermore, two researchers followed the same procedure to assess the severity of each “actual problem” using Nielsen’s guidelines (Nielsen, 1994a). The definitions of the five severity levels are as follows: level 0 means “no usability problem”; level 1 means “cosmetic problem”; level 2 means “minor usability problem that should be given low priority”; level 3 means “major usability problem that should be given high priority”; level 4 means “usability catastrophe that imperative to fix before product can be released” (Nielsen, 1994a).

To understand how each verbalization category is indicative of UX problems, we counted the number of segments in each verbalization category (i.e., Procedure, Reading, Observation, Explanation, and Others) that were associated with a UX problem. We then grouped them by language groups and TA protocols. Table 10 and Table 11 showed the results of the number of segments in each verbalization category indicating UX problems for language groups and for three TA protocols respectively.

We discuss the effect of language groups on three aspects: the proportions of verbalization categories, UX problems, and the correlations between verbalization categories and UX problems.

Proportions of Verbalization Categories: Our analysis showed no significant difference in the proportion of each verbalization category for two language groups. Moreover, the proportions of five categories also followed a similar trend for two language groups. Observation and Procedure were the most frequently appeared two categories, followed by Explanation, Others, and Reading.

While there was no statistical significance, the non-native language group did seem to verbalize a relatively higher proportion of Procedure (i.e., what they were doing) and a relatively lower proportion of Reading (i.e., what they saw) than the native language group.

Because both Procedure and Reading were all level-1 and level-2 types of verbalizations according to Ericsson and Simon’s definition (Ericsson and Simon, 1984), we combined these two categories and found that the proportion of the two categories together was similar for the two language groups: 41.1% and 44.1% for native and non-native language groups. To better compare our results with prior work, we further combined the verbalizations of each category across all our participants. Table 14 shows the result. It showed that our study found similar proportions of Procedure and Reading, Observation, and Explanation as Elling et al. (Elling et al., 2012), Zhao et al. (Zhao et al., 2014), and Fan et al. (Fan et al., 2019), but there was a difference between ours and Cooke’s study (Cooke, 2010) or Fan et al.’s study (Fan et al., 2021). The difference might be due to differences in the test products, TA protocols used, and the participants. While Cooke only used CTA in their study (Cooke, 2010), we used three types of TA protocols. While we tested with websites only, Fan et al. (Fan et al., 2021) tested with both physical and digital products. Furthermore, while Fan et al. (Fan et al., 2021) focused on older adults, our participants were all young adults. Future work should conduct more-controlled studies with the same set of products, participants, and study procedures to better understand how test products and participants’ age and other backgrounds might affect think-aloud verbalizations.

UX Problems: Our analysis results show that the language groups did not significantly affect either the number of problem encounters or the number of actual problems. The only difference was in the identification of some low severity UX problems. The implication is that both native and non-native English participants are equally effective in helping locate common and severe UX problems in think-aloud usability testing.

Correlations: Results show that correlations between verbalization categories and UX problems followed similar trends for both native and non-native language groups. The categories ranged from the most to the least indicative of UX problems were: Observation, Procedure, Explanation, Others, and Reading.

Similarly, we discuss the effect of three TA protocols on the same three aspects: the proportions of verbalization categories, UX problems, and the correlations between verbalization categories and UX problems.

Proportions of Verbalization Categories: TA protocols did not have a significant effect on all categories except Explanation. ITA had significantly more proportions of Explanation than CTA and SC. This difference was likely because the moderator prompted the participants to verbalize more by asking for explanations, opinions, and suggestions when using the ITA protocol. In contrast, such prompting behavior was forbidden in both CTA and SC protocols.

UX Problems: TA protocols also did not have a significant effect on the number of actual problems. While TA protocols were shown to have a significant effect on the number of problem encounters, post-hoc analysis did not find a significant difference among TA protocols. This suggests that all TA protocols were equally effective in identifying UX problems.

Although there was no statistical significance, ITA found a higher number of problem encounters than CTA and SC, according to Table 9. To better understand this difference, we examined the severity of the problems found by each TA protocol. Four out of the five problems that were only found by ITS were all of the lowest severity level (i.e., cosmetic problems (Nielsen, 1994a)). Cosmetic problems were mostly related to none-essential and nice-to-have features, such as changing the cursor icon when hovering over a clickable element. In other words, all protocols were equally effective in identifying severe UX problems. Furthermore, this finding was also consistent with previous research, which also found CTA, SC, and ITA found a similar number and type of problems (Alhadreti and Mayhew, 2017).

Correlations: Correlations between Verbalization Categories and UX Problems followed similar trends for three types of TA protocols. The Observation category was the most indicative of problems of all categories. This was consistent with prior findings on CTA (Fan et al., 2019). One difference among the TA protocols was that Explanation in ITA seemed to be more indicative of problems than in CTA or SC. This difference was likely because ITA had a higher proportion of verbalizations in the Explanation category than CTA or SC.

In this research, we took a first step to uncover similarities and differences in think-aloud verbalizations, the UX problems, and their correlations between two English language groups (i.e., native and non-native English speakers) in three think-aloud protocols (i.e., CTA, SC, ITA). Based on the findings, we discuss three design implications (DIs).

DI1: None-native English speakers can be as effective as native English speakers to help identify UX problems in Think-Aloud Usability Testing. Our findings show that the verbalization patterns (i.e., verbalization categories and how they indicate UX problems) observed in native English speakers in prior research (Cooke, 2010; Elling et al., 2012; Zhao and McDonald, 2010; Fan et al., 2019, 2021) were mostly applicable to non-native English speakers whose English proficiency was on or above the intermediate level (of Europe, 2021). Thus, UX practitioners could consider to enroll English speakers to their think-aloud usability testing for identifying UX problems without needing to worry about whether they are native or non-native speakers as long as their English reaches an intermediate level (of Europe, 2021).

DI2: Three concurrent think-aloud protocols (i.e., CTA, SC, and ITA) are equally effective in identifying severe UX problems. Our results suggest that the three types of think-aloud protocols (i.e., CTA, SC, and ITA) do not significantly affect the number of UX problems. While ITA might be able to identify slightly more problems, these extra problems are often of low severity levels. In contrast, ITA requires significantly more effort from the study moderator. The moderator has to constantly probe participants with different types of questions, which increases her work load. As a result, we suggest UX practitioners stick with Ericcson and Simon’s classic think-aloud protocol (CTA) (Ericsson and Simon, 1984), which not only minimizes the effort from the moderator, who only needs to remind participants to “keep talking” if they fall into silence, but also is able to uncover a similar set of important UX problems as the other two TA protocols (i.e., SC, ITA).

DI3: Non-native English speakers’ verbalizations could be used to increase the amount of training data to build artificial intelligence (AI) models to detect UX problems automatically. Our results show that similar verbalization patterns indicating UX problems were found in both non-native English speakers and native English speakers. For example, as indicated in Sec 5.3, for both language groups, the most verbalized categories were Observation and Procedure, which were also more indicative of UX problems than other categories. Such patterns could be used to build AIs to help UX practitioners better identify UX problems. For example, Fan et al. showed that verbalization category label was an effective feature to train AIs to detect UX problems automatically along with other textual and acoustic features, such as sentiment, speech rate, pitch and loudness (Fan et al., 2020a). Meanwhile, verbalization category information could also be visualized to direct UX evaluators’ attention to segments of a TA usability test video that are more likely associated with a UX problem (Fan et al., 2020c).

One key challenge in building such AIs or AI-assisted tools is to gather a large amount of training data. This is even more challenging if UX practitioners have to focus on recruiting native English speakers due to the concerns of the effect of participants’ language proficiency on identifying UX problems. Indeed, outside of a few countries where English is the first language, it is almost impossible to recruit a sufficiently large number of native English speakers. Luckily, many countries around the world have a critical mass of regular non-native English speakers. Our research provides evidence that non-native English speakers’ TA verbalizations can be as effective as those of native speakers for identifying UX problems. Thus, the implication is that when UX practitioners recruit participants to collect a large amount of TA usability test videos, such as via online remote usability testing (e.g.,  (Thompson et al., 2004; Andreasen et al., 2007)), to train AI models, they could consider to recruit both native and non-native English speakers. This would significantly increase their chance and lower their cost of getting a sufficient amount of TA test sessions to train AIs.

Our research contributes to a first understanding of the effects of English language proficiency on think-aloud verbalizations and how TA verbalizations of native and non-native English speakers indicate UX problems. In this section, we would like highlight the limitations of our current work and discuss potential future work.

First, our study included a relatively small number of participants ($N=18$). It is essential to replicate the study with more participants of diverse backgrounds to further validate and extend the findings.

Second, we chose a subgroup of non-native English speakers—Chinese students who studied in US universities. Other non-native English speakers, such as ones from other countries, may think aloud differently in English. Further, our non-native English speakers had intermediate or higher language proficiency (Table 1). Non-native English speakers with lower proficiency might verbalize their thoughts differently too. Thus, it is worth investigating how cultural backgrounds and language proficiency of non-native English speakers might affect their thinking aloud in English.

Third, we focused on English language. Other language speakers might organize their thoughts differently. Slobin argued that language “is a subjective orientation to the world of human experience, and this orientation affects the ways in which we think while we are speaking” (Slobin, 1996). Indeed, research suggested that cultures and languages could affect users’ think-aloud processes (Hall et al., 2004; Clemmensen et al., 2009; Clemmensen, 2011). Thus, it is worth exploring whether subtle verbalization patterns are telltale signs of UX problems for other languages, for example, Chinese, Dutch and French to name a few. If similar verbalization patterns indeed exist in other languages, then such patterns could be utilized to inform the design of AIs or AI-assisted tools to support UX evaluators to uncover UX problems for products used by people who speak those languages.

Fourth, conducting TA usability testing relies on the moderator’s skills too, especially the interactive think aloud (ITA). Our study moderator had two years of UX experience in conducting TA sessions. It is possible that the moderator’s experience would affect how she prompts participants (e.g., what prompts to use and how frequently to prompt participants). Thus, it is worth exploring how the moderator’s experience might affect how participants think aloud.

Fifth, we selected three types of websites to increase the variations among the test products. However, it remains unknown whether and how products might affect participants’ thinking aloud processes. Similarly, task difficulty might influence how participants think aloud too. For example, participants might find it harder to verbalize their thoughts when working on a more difficult and demanding task. Thus, more research is warranted to understand how products and task difficulty might affect non-native English speakers’ verbalizations and their correlations with UX problems.

Lastly, we used three types of concurrent think-aloud protocols in the study. To reduce the potential drawbacks of concurrent protocols, retrospective think-aloud protocols (RTAs) are used in usability testing (Fan et al., 2020b). When using RTAs, participants complete the task first and then verbalize their thought processes while watching their session recording. Verbalizations in RTAs are shown to be a valid representative of participants’ thoughts (Guan et al., 2006). It is worth investigating how participants’ think-aloud verbalizations indicate UX problems in RTAs in the future.