Let’s Get Personal: Personal Questions Improve SocialBot Performance in the Alexa Prize

There has recently been an increased focus on creating dialogue systems capable of open-domain social dialogue konrad2021alquist ; finch2020emora ; gunrock2018 ; fang2018sounding . Unlike traditional systems, these conversational dialogue systems cannot assume domain restrictions or any specific information need, i.e., the only inherent goal is to converse with the user. While modern systems have access to more information and better tools, foundational components of natural human-human conversation remain elusive, i.e., intimacy and agency. Combined, these should increase engagement in the conversation.

Our conversational open-domain dialogue system, Athena harrison2020athena ; walker-etal-2021-athena , is a competitor in the Amazon Alexa Prize  gabriel2020further ; hu2021further , which gives real Amazon Echo users the ability to anonymously access and evaluate open-domain dialogue systems. These systems must interact socially, about any topic, with the goal of achieving a highly rated (>4 out of 5 average user rating) conversation that lasts longer than 20 minutes. Further detail on Athena’s technical components are available in the associated technical reports harrison2020athena ; walker-etal-2021-athena .

In general, there are three primary challenges with open-domain conversation:

1. P1:

   There are an infinite number of valid topics we must support. While some topics are common and have been explored in previous dialogue research, e.g., Movies and Music, other topics reflect a user’s special interest. Moreover, since Athena is a live system, there is an implicit expectation that Athena has up-to-the minute live data.

2. P2:

   We cannot rely on the user having an explicit goal or information need; they may just want to chat for entertainment.

3. P3:

   The conversation must be social. This means that we cannot artificially bloat the length of the conversation by telling twenty jokes in a row. Instead we must be responsive to a user’s interest in different topics.

In this paper we focus on personalization as a partial solution for all three of these core problems. We cannot remove topics from the pool of valid topics in P1, however by learning more about the user’s interests we can direct the conversation to areas of Athena’s knowledge. To accomplish this, we can start the conversation with personal questions that help Athena learn about the user. Whether or not the user has an explicit goal in mind for P2, we can share control of the conversation by using personal opinion questions to engage the user on a level beyond pure fact retrieval. This process will help direct users to content about which Athena has opinions. Personal questions are also a crucial part of human-human social conversation, as in P3. Most human-human conversations would quickly end if one of the conversational partners feel their interests are ignored. It’s vital, then, that Athena displays high levels of responsiveness to demonstrate her listening abilities and show understanding reis2011familiarity ; reis1996attachment . Current research has also found in dyadic conversations that a conversational partner was more well liked if they ask more follow-up questions huang2017doesnthurttoask : a similar strategy could increase the rapport between the user and Athena.

Existing work also shows the impact of experience with dialogue systems when engaging with a system like Athena bowden2019entertaining . Besides varying levels of experience, there are several user pools with different needs: there are users who have never interacted with Athena vs. repeat users, kids vs. adults, and users with varied backgrounds and interests. The only way to effectively cater to all of them is to personalize each conversation for each user. For example, the personal opinion questions we discuss below are labeled with age appropriateness, e.g., when talking about Movies with a child, Athena talks about cartoons like Frozen. Participation in the Alexa Prize allows us to evaluate our hypotheses: asking questions at the start of the conversation allows Athena to populate a personalized user model, users are willing to give highly informative feedback if invited, and A/B testing with live traffic shows a statistically significant positive impact on both perceived conversation quality and length when using personal opinion questions.

The remainder of this paper is organized as follows: Section 2 examines the current state of open-domain conversational systems with a specific emphasis on the Amazon Alexa Prize. Section 3 demonstrates Athena’s strategy of front-loading the conversation with personal questions to quickly build a personalized user model. Subsequently, Section 3.1 analyzes the resultant trends established across a large pool of real Alexa users while Section 3.2 explores the utility of asking the users directly for feedback and inviting personal questions from users. Then, in Section 4 we investigate two specific personal opinion question strategies (asking topic related Would You Rather choices and open-ended Hypothetical questions), for which we see a statistically significant improvement in both rating and length when evaluated at scale. In Section 5 we conclude.

Dialogue systems have been an area of interest for over 50 years weizenbaum1966eliza . Much previous work with dialogue systems has been centered around the explicit goal of completing a task, e.g., booking a flight, or booking a restaurant. Commonly, these systems anticipate an explicit goal or “information need” kiseleva2016predicting ; chuklin2015click ; Radlinski17 . Our goal of making a casual social dialogue system that is open domain is a very distinct task.

There have been many different approaches to open-domain dialogue ma2021unstructured . Recent work has seen an increased emphasis on trained end-to-end conversational systems sordoni2015neural ; vinyals2015neural ; mila2017 ; burtsev2018first . However, we believe that these models are not yet ready for real user interactions adiwardana2020meena ; zhang2020dialogpt ; roller2021recipes . Instead, many Alexa Prize systems take a hybrid approach of combining rules, retrieval, and generation to create a more robust system song2016twoAreBetterThanOne ; fedorenko2018avoiding ; zhou2020xiolice . These hybrid approaches closely resemble Athena’s design.

Other work that is focused on open-domain personalization has been focused on short exchanges yang2017personalizedResponseGenViaDomainAdaptation , or used Twitter and TV Script data as a source of utterances li2016persona . This is not adequate for Alexa Prize conversations bowden2018opendomain .

Athena’s unique environment, the Amazon Alexa Prize chandra2018conversational ; gabriel2020further , requires open-domain social chitchat with a spoken dialogue system. Over the last four years several competitive systems have emerged konrad2021alquist ; finch2021emora ; liang2020gunrock ; fang2018sounding . Personalization has become an increasingly important part of these social systems. IrisBot, Genuine2, and Caspr use repeat user detection to slightly alter some responses ahmadvand2018iris ; kinjal2021caspr ; rodriguezCantelar2021genuine . SoundingBoard associated their users with Big-5 personality traits and adapted the conversation to those traits  fang2018sounding . Tartan and Emora personalized conversation by using user-demographic information with handcrafted rules to affect follow-up content and ask an increased number of personal questions chen2020tartan ; finch2021emora . Alana used user profiles to track the interests of returning users and asked personal questions to build rapport alana2018 . Gunrock maintained user profiles which were used to both affect follow-up content and change sub-topic discussions according to gender, i.e., Fashion liang2020gunrock . Audrey trained a reinforcement learning model tasked with personalizing topic switching hong2020audrey . Proto uses an unspecified slot extraction process during the natural language understanding phase to extract attributes later used when personalizing responses saha2021proto , while Dream used Reddit data with the BART language model lewis2020bart to create a topic-based vectorization of Reddit user personas, which was subsequently used to give topic recommendation to real users baymurzina2021Dream . Alquist experimented with personalized sub-dialogue selection konrad2021alquist , while Athena reported results using a personalized topic promotion strategy walker-etal-2021-athena . This work is closely related to our own; however none of these systems evaluate and report the direct results of asking personal opinion questions, nor do they present the trends captured by their user models/personas.

We targeted two playful personal opinion question strategies (Would You Rather and Hypothetical questions) because playfulness is considered important when building rapport between conversational agents and users shani2022alexasnowman and humor has been shown to increase likeability and trust meyer2015understanding . Would You Rather questions have also been used in previous therapeutic settings to encourage conversation fields2009would .

Similar to when two humans are first meeting, it is challenging for Athena to break the ice with a new user; Athena has no idea where to direct the conversation in-order to engage the user. In this paper we argue that personalization increases the sense of intimacy and engagement. Therefore, Athena aims to build a user model as quickly as possible. To this end, Athena starts each new conversation with a carefully crafted introduction sequence that is explicitly designed to elicit user interests. Athena employs several different dialogue strategies to elicit this information. We ask the user for their name and learn about how the pandemic has impacted them. We also ask them about their hobbies, weekend activities, and vacation plans before inviting them to learn more about Athena by soliciting advice and inviting questions. By front-loading the conversation with these probing questions, we increase the chances of the user model picking up usable information.

In Figure 1 we provide an excerpt of a conversation and introductory sequence. During this conversation, the user model learns the user’s name, that their hobbies include swimming and chess (U6 and U13), that they are a student (U7), and that Hawaii is a travel destination of interest (U9). Some of this content will be immediately useful in the conversation, e.g., reaffirming their name and asking why they like Hawaii, while other pieces of information will be useful when initiating a new topic, e.g., talking about Sports because the user is a swimmer, or Board Games because of their interest in chess.

Athena supports 17 topics ranging from large general topics, e.g., Sports, to smaller niche topics. Each topic is associated with a set of referential expressions, for example, baseball, football, and Stephen Curry, all refer to the Sports topic. We also curated a gazetteer of $\sim$250 hobbies by analyzing the results of probing questions over several months. Each hobby is annotated with paraphrases, e.g., i like to paint, i like painting, and i painted when i was young will all match the painting hobby, and any associated topics when relevant, e.g., basketball and piano map to Sports and Music respectively.

Inviting an exchange of opinions about the current topic of discussion creates a more intimate experience that increases user satisfaction. We have curated a dataset of topic-annotated personal opinion questions, which are inter-weaved throughout the conversation bowden2019entertaining . The personal opinion questions are split into two different strategies, Would You Rather choices (WYR) and open-ended Hypothetical questions (HYP). We picked these strategies because playfulness and humor are important when building rapport and trust shani2022alexasnowman ; meyer2015understanding . These strategies have also been characterized as good conversation starters fields2009would and several lists of questions exist across the internet¹¹1E.g., https://conversationstartersworld.com contains several lists. The wouldyourather and hypotheticalsituation Reddit communities also have 316k and 102k subscribers respectively.. These sequences can also be characterized as a game²²2E.g., https://psycatgames.com/app/would-you-rather/.; gamifying our personal opinion question strategies slightly may have a desirable impact since “gaming” is a popular hobby among users, i.e., Figure 2. As we will discuss in Section 4.1, it is also straightforward to crowdsource the collection of question/answer pairs per topic, and the limited scope of answer choices, especially true for WYR, makes it easier to build NLU that can understand the user’s responses.

Figure 7 demonstrates a sample conversation using these strategies. In this example, Athena initiates the Dinosaurs topic in A1, and proceeds to interweave 6 different dinosaur themed dialogue strategies: A1-A2 gives dinosaur facts, A3-A4 talks about dinosaur preferences, A4-A5 focuses on dinosaur movies, A6-A7 goes through a dinosaur WYR sequence, A7-A8 talks about why dinosaurs are extinct, and A9-A10 goes through a HYP sequence. Athena’s two personal opinion strategies extend topical depth by four turns. The user also provides us with long thoughtful responses, which indicate their engagement walker2021IWSDSParadise , furthering the sense of rapport built between Athena and the user.

In this paper we proposed three fundamental challenges practitioners face when building open-domain dialogue systems: covering an intractable landscape of topics, users who aimlessly meander through the conversation, and the core requirement of sociability. We propose a partial solution to these three problems lies in personalizing the conversation to each user. We implement a system, Athena, and deploy it at scale in the Amazon Alexa Prize, yielding several interesting results. Firstly, we front-load the conversation with a sequence of probing personal questions to rapidly model the user. By qualifying several thousand real user conversations we’re able to understand the types of topics users are interested in along with common user hobbies - informing future practitioners where to concentrate their efforts. Secondly, we ask the user for candid advice on how to improve Athena and invite the user to ask questions. The user input is surprising informative, helping to identify areas of improvement and future development. Moreover, the user input indicates a strong desire for a more personable experience as evidenced by direct user requests, e.g., be more personable, and by the fact that most user questions ($>$ 90%) were personal questions about Athena’s life and opinions. This data further signals the possible benefits of asking the user candid questions during a conversation for future evaluations. Finally, we evaluated two personal opinion question strategies, Would You Rather choices and open-ended Hypothetical questions. A/B tests carried out over several days of live user traffic confirmed a statistically significant positive impact on both conversation rating and length when these questions were included in a conversation.