Proactive Interaction Framework for Intelligent Social Receptionist Robots

We denote the input image flow as $I^{(1)},I^{(2)},...,I^{(t)}$, where we use superscript $(t)$ to represent the $t$-th frame. In this module, we first extract the region of interests by YOLOv4 [20]. YOLOv4 is a powerful object-detection model that detects a wide range of classes of objects. In our case, we are mainly interested in 6 categories that may be connected to human identity and action recognition in our scenarios, including “person”, “backpack”, “handbag”, “suitcase”, “tie”, and “cell phone”, while other categories are neglected.

To reduce the computation cost, we re-use the output of the backbone of YOLOv4, the CSPDarkNet53 [21], as the feature extractor. As the bounding boxes are in different sizes, we apply RoIAlign pooling to normalize the sizes of feature maps, inspired by the MaskRCNN [22], as well as global average pooling (GAP) to reduce the feature representation dimension. Through this process we are able to represent each pedestrian $i$ in the image $I^{(t)}$ with a 512-dimensional feature vector $v^{(t)}_{i}$. We also add the embedding vector of 2D position information [23] and object classification ID, which gives the final visual token of ($\oplus$ denotes “concatenation” and EMB represents “embedding vector”)

$$e^{(t)}_{i}=v^{(t)}_{i}\oplus\textit{EMB}(\textit{POS}(i,t))\oplus\textit{EMB}(\textit{CLASS}(i,t)).$$

(1)

The action is denoted as a combination of an utterance ($u$), an emoji facial expression ($f$), and a body motion ($m$). We specify a static set of distinct actions $a_{k}=(u_{k},f_{k},m_{k}),k\in[1,K]$ by collecting the actions from the annotated dataset. To better represent the action in semantic space, we utilize an open-source natural language pre-training model ERNIE [12] to process each utterance into a vector. Then facial expression and the motion are limited to several pre-defined patterns, such as “smiling” and “hand-shaking” (Fig. 3). For each type of body motion and expression, we use an embedding vector. The representation of three modalities are concatenated and further processed by an additional feed-forward neural layer (FFN) to yield the representation for $k$-th action, denoted by

$$\phi(a_{k})=\textit{FFN}(\textit{ERNIE}(u_{k})\oplus\textit{EMB}(f_{k})\oplus\textit{EMB}(m_{k})).$$

(2)

To enable the decision model to reason over time, normally, we could try to track each individual and encode the trajectory. Unfortunately, tracking one individual requires registration of visual tokens across different frames, which can introduce additional error to the decision. To this end, instead of tracking each individual, we present a transformer decision model. Transformer is widely used in natural language processing [24] by allowing the tokens to fully interact with each other through self-attention. In our case, it is not only able to capture the trajectory of each pedestrian by attending over time, but also possible to capture the interaction between the target and other individuals, e.g., talking with another person, holding a cell phone, carrying a suitcase. Also, to preserve consistency for training and inference, for each visual token $e^{(t)}_{i}$ that we consider, we allow it to attend to $e^{(t^{\prime})}_{i}$ only when $t^{\prime}\leq t$, which implies masked self-attention in transformer blocks (MTRN for short). In our case, we use 6 transformer blocks.

For each frame $I^{(t)}$ received, we use visual token extractor to acquire a list of visual tokens $e^{(t)}_{1},e^{(t)}_{2},...$, from which we consider the top-$M$ ($M=20$ in the experiments) visual tokens. The priority of the selection goes as follows: the “person” class is selected in the first place; the larger the bounding box, the higher priority; in case there are less than $M$ visual tokens, we add padding terms. In every time step, we use the latest $N$ frames as the input, which gives $M\times N$ tokens in one sequence. For each token, we further add an embedding vector of the relative frame ID $\textit{E}_{[i]}$. The encoding process can be represented by

	$\displaystyle o^{(t-N+1)}_{1},$	$\displaystyle...,o^{(t-N+1)}_{M},...,o^{(t)}_{M}=\textit{MTRN}($
		$\displaystyle e^{(t-N+1)}_{1}\oplus\textit{E}_{[1]},...,e^{(t-N+1)}_{M}\oplus\textit{E}_{[1]},$
		$\displaystyle e^{(t-N+2)}_{1}\oplus\textit{E}_{[2]},...,e^{(t-N+2)}_{M}\oplus\textit{E}_{[2]},$
		$\displaystyle,...,$
		$\displaystyle e^{(t)}_{1}\oplus\textit{E}_{[N]},...,e^{(t)}_{M}\oplus\textit{E}_{[N]})$		(3)

Starting from Eq. Transformer-based Decision Model, the prediction of the decision model is three-fold: 1. decide whether to initiate an interaction, which depend on $\hat{y}^{(t)}$; 2. predict the target to interact, which is dependent on $\hat{h}^{(t)}_{i}$; 3. select the action to be taken, which is dependent on $\hat{p}^{(t)}(a_{k})$. We first take the max-pooling of all the positions in the last frame to represent the scenario, which is defined as

$$o^{(t)}=\textit{MaxPooling}(o^{(t)}_{1},o^{(t)}_{2},\ldots,o^{(t)}_{M}),$$

(4)

we then specify the three-fold output of the decision model as follows:

	$\displaystyle\hat{y}^{(t)}=$	$\displaystyle\sigma(W_{y}\cdot o^{(t)}+b_{y})$		(5)
	$\displaystyle\hat{h}^{(t)}_{i}=$	$\displaystyle\sigma(W_{h}\cdot o^{(t)}_{i}+b_{h}))$		(6)
	$\displaystyle\hat{p}^{(t)}(a_{1}),...,p^{(t)}(a_{k})=$	$\displaystyle Softmax(\phi(a_{1})\cdot o^{(t)},...,$
		$\displaystyle\phi(a_{k})\cdot o^{(t)})$		(7)

Correspondingly, the annotation in training data include three parts: interaction trigger $y^{(t)}=0/1$, where $y^{(t)}=1$ indicate a proper opportunity to initiate the interaction; interaction target indicator $h_{i}^{(t)}=0/1$, where $h_{i}^{(t)}=1$ implies that the visual token is one of the target to be interacted; the action selection $\bm{I}^{(t)}_{a}$ which is a one-hot vector of length $K$. We use cross-entropy loss $\mathcal{L}_{\textit{ce}}$ for each of them, giving the loss function of Eq. Transformer-based Decision Model.

	$\displaystyle\mathcal{L}=$	$\displaystyle\sum_{t}\mathcal{L}_{\textit{ce}}(y^{(t)},\hat{y}^{(t)})+\sum_{t}y^{(t)}\cdot\mathcal{L}_{\textit{ce}}(\bm{I}^{(t)}_{a},\bm{\hat{p}}^{(t)})$
		$\displaystyle+\sum_{t}y^{(t)}\sum_{i}\mathcal{L}_{\textit{ce}}(h^{(t)}_{i},\hat{h}^{(t)}_{i})$		(8)

For inference, we use weighted sampling on $\bm{\hat{p}}^{(t)}$ for action selection; for target filtering, we first remove the visual tokens of classes other than “person”, then we use a threshold $H$, such that $h^{(t)}_{i}>H$ indicate a valid target. At executing the actions, the robot turns towards the centroid of all the valid targets. We argue that turning to the interaction target is essential to the user experience.

Hours-long videos were collected from two working Xiaodu robots (Fig. 3) in the office lobbies, and then they were annotated and processed into 5-second-long video clips as the training and test set. The office lobbies for our data collection are significantly different in the light condition. As shown in Fig. 3 (b) and (c), the B-lobby has a worse light condition than the A-lobby because of the backlighting environment. This makes the vision-based approaches more challenging in the B-lobby. After collecting hours-long videos from these two lobbies, we used the following method to preprocess, annotate, and post-process them:

1. 1.

   The raw videos were preprocessed by a multiple object tracking model [25], which marks the bounding boxes of tracked persons with a unique ID for each. The experts label suitable targets for initiating interactions with track IDs.

2. 2.

   Experts watched the preprocessed videos and selected the suitable timestamps for initiating interactions. For each selected timestamp, the expert annotated the tracking IDs of targeting interaction objects, chose the facial expression and the body motion from a list that the Xiaodu robot permits, and filled the textbox with a proper utterance for greeting.

3. 3.

   Annotated videos were post-processed into 5-second-long video clips in which positive examples were from experts’ annotations, and negative examples were randomly sampled from segments without annotations then filtered out in-interaction segments.

Statistically, for the training set, we collected and labeled 3900 positive cases (i.e., trigger label is 1) with 1000+ unique multi-modal actions (triplet of utterance to speak, emoji to display, and body action to execute), in which 1720 cases are from B-lobby, and 2180 cases are from A-lobby. For the test set, from 12-hour videos, we collected and labeled 74 positive cases in A-lobby and 91 positive cases in B-lobby. However, when evaluating the performance on the test set, we used the full 135k+ negative cases instead of sampling to verify the model sufficiently.

To evaluate the proactive HRI framework, it is desirable to verify whether the system predicts the correct timestamp to initiate interaction and whether the action selection is proper. However, although each positive case in the test set has a labeled action, we found that multiple actions can be suitable for one scenario. Therefore, evaluating the action selection is extremely challenging. In this part, we evaluated the capability of predicting proper triggering time only. The user experience research in the next part will evaluate whether the selected action is proper.

We used the state-of-the-art model called R(2+1)D [13] in a similar video action recognition task as our baseline. The R(2+1)D model was pre-trained on a large video dataset from social media. We fine-tuned it on our dataset in an end-to-end manner to predict the multi-modal action ID. To enable a fair comparison between our Transformer setting and R(2+1)D, we add a “NULL” action in the action set, such that selecting the null action is equal to not triggering an initiation. For R(2+1)D, the output is a $(K+1)$-way classification on the action space. In this way, the R(2+1)D can predict the triggering time without predicting $\hat{y}^{(t)}$. For our model, we also add a “NULL” action in the action set, this leads to three different inference modes:

• •

  Trigger-Only By removing the null action, We rely only on $\hat{y}^{(t)}$ to decide whether to initiate interaction.

• •

  Actor-Only By setting $\hat{y}^{(t)}=1$ identically, we rely only on $\bm{\hat{p}}^{(t)}$ to select the null action to serve as the trigger.

• •

  Trigger-Actor Initiation is only triggered when we both have $\hat{y}^{(t)}>H$ and avoid selecting the null action.

To find out the contribution of different elements in the visual token, we designed ablation studies by removing the RoIAlign pooling feature ($v^{(t)}_{i}$), the position embedding, and the classification embedding in turn. We reported the performance of successful proactive interaction (proactively and appropriately interact with the user) in terms of precision and recall. The performance of the trigger is related to the threshold, so we include two more metrics: average precision (AP) and average recall (AR) for better evaluation. Furthermore, we use the F1 score to demonstrate overall performance. All the results are shown in Table I. Note that we select the threshold with the maximum F1 score to report the precision and recall. In the Trigger-Actor mode, we also use this threshold for the trigger. As we can see, the results show that the proposed transformer model outperforms the state-of-the-art end-to-end R(2+1)D model. Besides, the ablation study shows that the most important factor is the information from the RoIAlign pooling feature ($v^{(t)}_{i}$). Without this information, the framework suffers the largest F1 score drop.

Because the data were collected under a passive HRI system, the distribution of the test set was different from that of the online experiments. To figure out whether the proposed framework performs well on a real robot, we implement this user experience research on a real robot in the A and B lobbies, comparing the existing wake word-based interaction system of the Xiaodu robot. On the Jetson AGX Xavier, the proposed TFVT-HRI model runs at 6.25fps, whereas the R(2+1)D model runs at 1.89fps. Since the R(2+1)D model suffers from 3x computation latency, and its performance is worse than the proposed framework, we do not deploy it on the real robot for this user experience research.

Participants. We recruited 30 new employees who have never interacted with the Xiaodu robot, but 26 of them used other similar AI devices (they share the same wake word “X” to start an interaction). Among the participants, there are 16 male and 14 female; their ages range from 21 to 35 years old (M=27.1, SD=3.84). All participants reported normal or corrected to normal vision and normal hearing. All participants volunteered to participate in the study and agreed to make audio and video recordings of the research process. At the end of the study, all participants were given appropriate compensation.

Design. The experiment adopted a between-subjects design. Participants were randomly assigned to one of the two groups, the experimental group and the control group. In the experimental group, the robot was deployed with the proposed framework and would proactively initiate interaction. In the control group, the robot was waiting reactively for the wake word.

The dependent variables included the objective factor (success rate) and subjective factors (emotions, attitudes). For emotions, we utilize the Self-Assessment Manikin (SAM) technique [26] because of its high correlation with psychological response [27] and focus on the metrics of valence and arousal. For attitudes, the participants are asked to evaluate the level of overall comfort, naturalness, friendliness, and intelligence using a 7-point Likert questionnaire (from 1-7, higher scores indicated stronger agreement). The present study only focused on the initiating process of HRI. Participants were also instructed to give ratings based on their experience of initiating interaction.

A success case means that the participant use at least one function of the Xiaodu robot. For example, after deploying the proposed framework, the robot may greet then suggest one function to the participant. If the participant noticed it and responded, this is a success case. For the existing wake word-based HRI, the participant should try to wake, the robot then following text instruction on the screen to make a success interaction case. A failure case means that the participant neither responds to the proactive calling of the robot nor wakes the robot and follows the instruction.

For the objective metric, success rate, 100% of participants had success interactions when using the proposed framework. Whereas the success rate was 80% on the existing wake word-based HRI system. For emotions, the results of descriptive statistics are shown in Table II. Independent-Samples T Test indicated that there was no significant difference in valence ($t(28)=1.218$, $p=0.233>0.05$) and arousal ($t(28)=1.906$, $p=0.067>0.05$) We combined the scatter diagram of the valence-arousal plain and affect words coordinates in Fig. 4. It informed us that two groups both have a positive emotion, EXCITED. The Levene’s Test indicated that all the attitudes variables except for the intelligence score met the assumptions of variance homogeneity at $p>0.05$. Thus, we used the Independent-Samples T Test for Overall Comfort Level, Naturalness, Friendliness scores, and T’ Test for intelligence score. Compared to the control group, the experimental group reported significantly higher scores in Overall Comfort Level ($t(28)=2.141$, $p=0.041<0.05$), Naturalness ($t(28)=2.354$, $p=0.026<0.05$), Friendliness ($t(28)=2.705$, $p=0.012<0.05$), and Intelligence ($t^{\prime}(24.679)=2.225$, $p=0.035<0.05$).

In summary, the proposed framework has a higher success rate of initiating an interaction than the existing wake word-based HRI system of the Xiaodu robot due to that the proposed framework can capture the intention of the participants and trigger multi-modal interaction signals (language, facial expression, and body language). However, no significant difference was found in the emotions of the two groups. The robot’s characters may have some positive impacts on users’ emotions, such as its cute appearance (Fig. 3), and child-like voices. However, the proposed framework leads to significant differences in feelings of overall comfort, naturalness, friendliness, and intelligence.

Some interaction cases are shown in Fig. 5. As we can see in Fig. 5 (a), when a group of visitors passed by, the Xiaodu robot raised its hand and greeted these people using voice. Here we emphasize that the Xiaodu robot started this action when the visitors were in the near field, matching the prior studies based on Hall’s proxemics theory [15]. In the second case (Fig. 5 (b)), a visitor were taking photos with the Xiaodu robot. The robot proactively blinked its eyes as responses. The third case (Fig. 5 (c)) shows that the Xiaodu robot successfully led the girl to a depth interaction. Overall, the TFVT-HRI framework substantially learned certain knowledge of social rules from the experts’ demonstrations.