To Ask or Not To Ask: Human-in-the-loop Contextual Bandits
with Applications in Robot-Assisted Feeding

We design a user study (Fig. 3) to capture how different query types affect an individual’s querying workload in robotic caregiving scenarios. In such scenarios, a feeding system may request various assistance types from the user, such as semantic labels, bounding boxes for food items, or explanations for failed acquisition attempts. These response types can have varying impacts on workload [\citeconsecutivecui2021understanding, koppol2021interaction].

For instance, a semantic label response may impose less workload than an open-ended response. In addition, the end-user of the system and the caregiver may be busy with other tasks, such as watching TV while eating. We conduct an online user study that captures the above factors to determine how an individual’s querying workload changes in response to requests for assistance from an autonomous system.

Participants perform a distraction task while periodically receiving queries from a “robot,” simulating realistic caregiving situations where users might be engaged in other activities. The robot queries represent realistic assistance requests during bite acquisition. We vary four independent variables hypothesized to impact querying workload:

1. 1.

   Robot query difficulty ($d_{t}$): 2 options – easy or hard.

2. 2.

   Distraction task difficulty ($dist_{t}$): 3 options – no distraction task, an easy numeric addition task, or a hard video transcription task.

3. 3.

   Time interval between queries ($\Delta T$): 2 options – either 1 minute or 2 minutes.

4. 4.

   Response type ($resp_{t}$): 3 options – (a) a multiple-choice question (MCQ) asking for a semantic label, (b) asking the user to draw a bounding box (BB) around a food item, or (c) an open-ended (OE) question asking the user to explain why an acquisition attempt failed.

Each participant experiences 12 study conditions, one for each of the 2 settings of $d_{t}$, 3 settings of $dist_{t}$, and 2 settings of $\Delta T$, with the entire study taking roughly 1.5 hr. During each condition, lasting 5.5 minutes, participants engage in distraction tasks and respond to robot queries (with varying response types $resp_{t}$) as shown in Fig. 3. We measure self-reported workload (our dependent variable) after each robot query and at the end of each condition using 5 modified NASA-TLX subscales [\citeconsecutivehart1988development, hertzum2021reference] (mental/physical, temporal, performance, effort, frustration), each measured on a 5-point Likert scale (see Appendix for exact question wording).

We collect data from 14 users (7 male, 7 female; ages 27-50) with mobility limitations resulting from a diverse range of medical conditions: spinal muscular atrophy (SMA), quadriplegia, arthrogryposis, cerebral palsy, Ullrich congenital muscular dystrophy, schizencephaly with spastic quadriplegia, spinal cord injury (including C2 and C3 quadriplegia). We additionally collect data from 3 occupational therapists (OTs), (3 female; ages 23-26), trained to simulate right/left hemiplegia (stroke). The OTs positioned one side of their body in full shoulder adduction, and elbow flexion to allow only 30 degrees of functional elbow joint range. Finally, we collect data from 89 users without mobility limitations (40 male, 49 female; ages 19-68). The population size for users with limitations is larger than the median ($n=6$) in studies of assistive robots [\citeconsecutivemankoff2010disability, nanavati2023physically].

We consider three distinct datasets under which to train the workload model: $D_{1}$, which only includes the data from the 89 users without mobility limitations; $D_{2}$, which only includes the data from the 14 users with mobility limitations and 3 OTs; and $D_{1,2}$, which includes both $D_{1}$ and $D_{2}$. We convert the modified NASA-TLX responses into a single querying workload score by normalizing responses to 3 of the questions, and taking a weighted average (weight details in Appendix). We find a statistically significant difference in mean querying workload between $D_{1}$ ($0.36\pm 0.29$) and $D_{2}$ ($0.40\pm 0.28$), as described in the Appendix.

We develop a predictive model of querying workload based on the user study data (Sec. IV-A). Our models operate in discrete time, where the time variable $t$ refers to an integer number of timesteps since the beginning of a condition, where each timestep corresponds to a fixed time spacing $\Delta t$ (set to $10\text{s}$ in our experiments). The model has the form $WL_{t}=f(WL_{0},Q_{t};\theta)$, where $WL_{0}$ is the initial workload, $Q_{t}=\{(d_{t^{\prime}},resp_{t^{\prime}},dist_{t^{\prime}})\}_{t^{\prime}=1}^{t}$ is the history of previous queries (where $d_{t}$, $resp_{t}$, and $dist_{t}$ are the query variables defined in Sec. IV-A), and $\theta$ are the model parameters.

Linear discrete-time models. To capture the dependency between the $WL_{t}$ and $Q_{t}$, we use a linear predictive workload model inspired by Granger causality models [\citeconsecutivegranger1969investigating,liang2023randomization]:

where $\mathbf{w_{i}}$ represents the effect of the query asked $i$ timesteps in the past on the workload at the current time $t$, $H$ is the history length, and $w_{0}$ is a bias term. We train the model using linear regression, where we convert $d_{t}$, $resp_{t}$, and $dist_{t}$ into features using one-hot encodings, and generate a training pair for each robot task query in our study (see Appendix for more details about the Granger models).

Model selection. We train models $f_{1}$, $f_{2}$, and $f_{1,2}$, corresponding to the datasets $D_{1}$, $D_{2}$, and $D_{1,2}$, respectively (Sec. IV-B). ¹¹1When training these models, our assumptions about the time taken for users to complete the modified TLX surveys, particularly for users without mobility limitations, are in the Appendix.. The models $f_{i,sim}$ and $f_{i,real}$ indicate models learned for the simulation experiments and the real-world user study, respectively. For our simulation experiments, we learn 3 separate models $f_{1,sim}$, $f_{2,sim}$, and $f_{1,2,sim}$ (details in the Appendix). For our real-world user study, we only use 2 of the 3 real models: $f_{1,real}$ for the users without limitations and $f_{1,2,real}$ for the users with limitations. In our setting, $f_{1,real}$ and $f_{1,2,real}$ are models chosen based on cross-validation median test MSE on $D_{1}$ and $D_{1,2}$, respectively (full model selection details in Appendix). We also train a model $f_{2,real}$ (cross-validation score: $0.066\pm 0.022$), which has a higher mean and standard deviation MSE than $f_{1,2,real}$ (cross-validation score: $0.058\pm 0.010$). This suggests that training only on $D_{2}$ leads to a higher-variance, less accurate model, compared to training on $D_{1,2}$. Thus, it is still helpful to consider the data from users without limitations ($D_{1}$) when training our workload models.

We simulate food interaction using a dataset of bite-sized food items on plates, including 16 food types [4], each with 30 trials for the 6 robot actions $a_{r}$. We first draw a random image of a given food type from the dataset and generate the corresponding context $x$ using SPANet. When the policy selects a robot action $a_{r}$, we sample a binary task reward $r$ from a Bernoulli distribution with success probability $s_{x}(x,a_{r})$. If $r=1$, we declare that the bandit has converged, and move to a new food type at the next timestep. If $r=0$, we draw a new random image of the same food type from the dataset until we exceed the maximum number of attempts $N_{att}=10$. When the policy selects a query action $a_{q}$, the bandit policy repeatedly executes the optimal action $a^{*}(x)$ until convergence, or the limit $N_{att}$ is reached.

Metrics. We compute 5 objective metrics to evaluate the tradeoff between task performance and querying workload:

• •

  The mean task reward $r_{task,avg}=\frac{1}{T}\sum_{t=1}^{T}r_{task}(x_{t},a_{t})$, measuring the efficiency of successful food acquisition.

• •

  The episodic change in workload $\Delta WL=WL_{T}-WL_{0}$.

• •

  Our surrogate objective $M_{wt}=w_{task}r_{task,avg}-(1-w_{task})\Delta WL$, where $w_{task}$ represents a user-specific preference for the task performance/workload tradeoff.

• •

  The number of timesteps taken to converge $t_{conv}$, which provides additional insights into task performance.

• •

  The fraction of food items for which we queried $f_{q}$, which provides an insight into querying workload.

Evaluating querying workload in simulation. In our simulation environment, we use the learned workload model $f$ for both LinUCB-QG action-selection, as well as computing the evaluation metrics $\Delta WL$ and $M_{wt}$. During the rollout of a particular bandit policy, we evolve the workload model state $WL_{t}$ according to the learned model when we query. We consider the three dataset scenarios and corresponding workload models $f_{1,sim}$, $f_{2,sim}$, and $f_{1,2,sim}$ (Sec. IV-C), and set $WL_{0}=0.5$ (a median initial workload).

Pretrain, validation, and test sets. In our experiments, we partition the set of 16 food types into a pretraining set (for the contextual bandit and for training SPANet), a validation set (for tuning the querying algorithm hyperparameters), and a test set (for metric evaluation). Our validation set (cantaloupe, grape) and test set (banana, carrot) include food items with varying material properties. Our pretraining set includes the remaining 12 food types in the food dataset. We use the validation set to select the hyperparameter that maximizes the weighted metric $M_{wt}$ given a $w_{task}$ setting.

Results. We investigate how LinUCB-QG balances the tradeoff between task reward and querying workload. Table I compares the four algorithms for a setting with a slight preference for maximizing task performance over minimizing workload ($w_{task}=0.7$). Across all three workload data settings, LinUCB-QG has a higher mean task reward $r_{task,avg}$ than LinUCB and LinUCB-ExpDecay, and a mean weighted metric $M_{wt}$ that is better than or competitive with the other methods (within the uncertainty of the simulation), suggesting that it achieves the best overall tradeoff between task reward and querying workload. Additionally, the querying fraction $f_{q}$ for LinUCB-QG increases from $D_{2}$ to $D_{1,2}$ to $D_{1}$, suggesting that LinUCB-QG is sensitive to the higher workload predictions with care recipient and OT data, while still trading off task reward and workload.

Experiment: generalization on $D_{2}$. We run an additional experiment to ablate $D_{2}$, where at test time we use either $f_{1,sim}$ or $f_{1,2,sim}$ in LinUCB-QG action-selection, but use $f_{2,sim}$ for metric evaluation ($w_{task}=0.4$). We find that $M_{wt}$ is lower in the first setting ($0.269\pm 0.040$) compared to the second setting ($0.373\pm 0.108$), suggesting that including $D_{2}$ is crucial to achieve the best trade-off between task reward and workload for users with mobility limitations. See Appendix for more details and comparison metrics.

To evaluate the real-world performance-workload tradeoff of LinUCB-QG, we conduct a user study with 19 users: 18 users without mobility limitations (8 male, 10 female; ages 19-31; 66% with prior robot interaction experience), and one 46-year old user with Multiple Sclerosis since they were 19. We investigate whether LinUCB-QG improves task performance compared to LinUCB, while minimizing workload compared to Always-Query. This study was approved by the IRB at Cornell University (Protocol #IRB0010477).

Study setup. We use a Kinova Gen-3 6-DoF robot arm with a custom feeding tool (details in Appendix). For each method, we present users with a plate of 3 food items (banana, carrot, cantaloupe), with diverse characteristics: cantaloupe works with all robot actions; carrots require sufficient penetration force and fork tines perpendicular to the food major axis; bananas are soft, requiring tilted-angled actions [5]. We set $N_{att}=3$ for a reasonable study duration. The user provides feedback using a speech-to-text interface when queried, without any additional distraction tasks. Users evaluate each of the three methods in a counterbalanced order. We measure workload before/after each method by asking 5 modified NASA-TLX subscale questions: mental/physical, temporal, performance, effort, and frustration (question wording in Appendix). We conduct 2 repetitions of the 3 food items for all 3 methods (18 total trials).

Metrics. We define 7 subjective and 3 objective metrics to compare the methods. 4 subjective metrics correspond to the modified post-method NASA-TLX questions: Post-Mental Workload, Post-Temporal Workload, Post-Performance, and Post-Effort Workload. The other 3 subjective metrics measure workload changes during each method: Delta-Mental Workload, Delta-Temporal Workload, and Delta-Querying Workload (the change in querying workload score, weighting function in Appendix). The 3 objective metrics are mean task reward per plate ($r_{task,avg}$), mean successes per plate ($n_{success}$), and mean query timesteps per plate ($n_{q}$).

Overall results. Among the three methods, LinUCB-QG offers the most balanced approach to human-in-the-loop bite acquisition. It is more efficient than LinUCB and imposes less querying workload than Always-Query. While LinUCB typically picks up the carrot or cantaloupe within 1-2 timesteps but struggles with the banana, Always-Query successfully picks up food in the first timestep by always querying the user, leading to higher workload. Our method, LinUCB-QG, selectively asks for help with the banana and acts autonomously for the other foods in most trials, balancing task performance with querying workload.

(1) Task success results: Fig. 4 (right) shows that LinUCB-QG achieves higher objective ($r_{task,avg}$) and subjective (Post-Performance) success ratings versus LinUCB, our autonomous baseline, showing greater efficiency. (2) Querying workload results: As shown in Fig. 4 (left, top), LinUCB-QG has lower subjective querying workload scores (mental, temporal, effort) compared to Always-Query, indicating that selective querying reduces workload. This is the case for both the post-method metrics (Post) and the workload change metrics (Delta). All results are statistically significant (Wilcoxon paired signed-rank test, $\alpha=0.05$). See Appendix for full comparisons across all metrics.

Results: user with mobility limitations. We highlight observations from the user with mobility limitations. First, they provided a higher mean subjective success rating for LinUCB-QG than for LinUCB, aligning with the aggregate results. However, the user provided consistently low physical/mental, temporal and effort ratings across all methods. Potential reasons for this include the relative simplicity of providing feedback in our setting, and the fact that this particular user frequently requests assistance in daily life, reducing their querying workload. The user also commented that relying on external physiological signals (such as EEG) to estimate workload would add stress (due to the additional hardware required), reinforcing the benefit of non-intrusive workload models for human-in-the-loop querying algorithms.

Discussion. Future work could incorporate different query types in the real-world study, beyond the best acquisition action, such as the other query types in our workload model. In our setting, task complexity depends on physical food properties such as compliance or shape. We did not explicitly focus on such variables when modeling workload, although query difficulty $d_{i}$ does correlate with task complexity. The NASA-TLX scale has known limitations, such as subjectivity [38], over-emphasizing task difficulty [39], and workload score calculation problems [\citeconsecutivevirtanen2022weight, bolton2023mathematical], meaning that the modified NASA-TLX scores used to train our workload models are inherently noisy. We mitigate this by focusing on relative workload changes when evaluating how well our algorithms balance performance and workload. Additionally, we do not take measurements between queries in our workload dataset, so our workload models interpolate between queries using their learned weights, leading to rollouts that are not always intuitive (e.g. due to assigning greater weight to past queries). Future directions include predictive modeling with objective, non-intrusive workload measurements, and extensions to other assistive tasks where performance must be balanced with workload, using domain-agnostic workload features.

We provide the full algorithmic description for LinUCB-ExpDecay in Algorithm 2.