The Interface Usage Skills Test: An Open Source Tool for Quantitative Evaluation in Real Time for Clinicians and Researchers

In the domain of functional rehabilitation, outcome measures span a wide range from kinesthetic and neurophysiological [4] measures of patients to global outcomes in terms of overall function and community reintegration [5].

The Wheelchair Skills Test (WST) is the state-of-the-art in clinical evaluation of an individual’s ability to drive a PW, and is an intermediate-level outcome that lies between the two extremes cited above [6]. This measure consists of various tasks—including ascending and descending slopes, and navigating through doorways—and for each task, an observing therapist chooses a capacity and confidence based on the completion of the task and subjective safety. A score of 0 indicates failure to complete the task, 1 indicates that the user had some difficulties completing the task, and 2 indicates that they accomplished the task without any difficulty. This measure does not capture details of the exact difficulties the person experienced in completing specific tasks, such as control smoothness. Although the WST is a powerful assessment tool, its delivery is subject to clinician training [7]. Powered Mobility Clinical Driving Assessment (PMCDA) is another assessment tool that is also observational [8].

Currently, there are no assessments that consider how an individual completes the skill in terms of objective measures of safety, such as distance to barriers, speed, or smoothness profiles [9]. Furthermore, in the assistive robotics domain, there is no standard way to assess user skill in operating a teleoperaton interface. The most common performance measures used to evaluate the efficacy of assistive robotic systems include task completion time and tracking errors, or subjective questionnaires that do not assess objective user skill [10, 11, 12].

All of these assessments require a specific physical setup and, more importantly, do not provide a quantitative analysis of interface usage, but rather qualitative observations on wheelchair driving skill. We propose that by looking at one level of abstraction—the interface operation—we can better identify areas of control deficit that will lead to improving overall wheelchair driving skill.

To our knowledge, there is no standard assessment tool for characterizing interface usage. Research studies of interface characterization have focused mainly on the neuromuscular and physical response of the human during manual control to study human sensing and response characteristics [13, 14, 15].

In the field of assistive technology, clinicians have been surveyed on the usefulness and adequacy of powered wheelchair control interfaces for their patients [16]. Their results provide subjective evidence regarding steering and maneuvering difficulty, and for a need to integrate robot autonomy into conventional powered wheelchairs. However, their results do not provide quantitative information on interface usage, which could be exploited by assistive autonomy. Novel interface technologies have been developed for those with severe motor impairments, including an isometric joystick [17]. The authors compared task completion time and accuracy of the novel interface to a conventional hand-held joystick within a control population but not an end-user population. In another work, the authors introduce a novel interface and compare user precision and performance between the target spinal cord injured (SCI) group and an uninjured control group, but do not compare the performance against commercially available interfaces [18].

In the human-robot interaction domain, a study investigated the influence of video game usage on human-robot team performance [19]. The authors classify interfaces based on their inputs and outputs and use the information to create a framework for systematically evaluating interfaces in the Human Robot Interaction (HRI) domain. Prior work in remote teleoperation has also studied the effect of time delay and communication channel degradation on the quality of teleoperation and manual control [20].

In this work, we contribute an assessment tool for the quantitative characterization of interface usage.

The assessment consists of a series of tasks that evaluate qualities of the human input including speed, precision, and stability: all necessary criteria for a human to be able to properly command an assistive machine. This is done through two distinct tasks: a command following task and trajectory following task. Tasks are designed in a simulated environment so that uncertainties from real world dynamics do not corrupt the interface usage performance measures.

Individual user profiles can be made and the tasks can be selected via a menu (Fig. 2(a)). Each task can be reconfigured as described in Section III-C.

A multitude of variables impact the human’s interface usage characteristics while operating an assistive machine. We group these variables into four distinct categories [22]:

• •

  Operational Variables. Factors such as the dynamics of the device they are controlling (e.g., rear-wheel vs mid-wheel drive wheelchair), the mechanics of the control interface, as well as what information is available to the human and how.

• •

  Environmental Variables. Factors such as temperature, whether the task is indoors or outdoors, or noise.

• •

  Internal Variables. Factors such as internal motivation, training and skill, fatigue, and stress that affect human internal states.

• •

  Procedural Variables. Features of the experimental design, such as the instructions for the given task and order of task presentation.

The effect of procedural variables is minimized in the Interface Skills Test by standardizing the instructions for each task within the app. To control for the remaining variables, the clinician, test taker, or test giver can input information using questionnaires prior to and after a given task, as described in the following section.

The assessment tool is designed with a variety of configurable input settings through a user-friendly GUI. The configurable measures consist of information relating to operational variables, environmental variables, and the human’s internal variables, as well as configurable settings pertaining to how the tasks are presented to the human test taker.

The controlled covariates for documenting the human’s internal state are collected through a series of Likert-type questionnaires administered directly within the app. Documenting these variables is not necessary for running the assessment, but it is important and recommended to keep track of these as they may inform larger trends in the human’s interface usage skill characteristics.

• •

  Fatigue: Measured using the Fatigue Scale, which is an 11-item Likert-type questionnaire [23].

• •

  Motivation: Measured using the Intrinsic Motivation Inventory (IMI) [24].

• •

  Workload: Measured using the raw NASA-TLX which is a shortened version of this assessment tool [25].

• •

  Stimulant consumption: Text entry question.

• •

  Confidence: A 5-point rating scale question on how confident the human is in their ability to use the interfacing device.

• •

  Stress: Measured via the Perceived Stress Questionnaire [26].

Other control variables can also be documented that monitor environmental and operational variables:

• •

  The interface used during the test.

• •

  How often the interface is used daily by the patient.

Independent variables used within the command following tasks able to be reconfigured by the clinician and test-taker include (Fig. 2(b)):

• •

  Set of target control commands. This also includes the choice of selecting only directions or magnitudes of the commands. The default is the four cardinal and four inter-cardinal angles.

• •

  Number of times each target command is prompted. The default is set to 20.

• •

  Range of time each target prompt is displayed. This is the amount of time each command prompt is visible and the time the user has to respond. The default range is between 1-2 seconds.

One of the main contributions of our work is that the assessment is calculated using strict closed-form equations, and that the results do not depend on qualitative observations of the test taker. Additionally, outcome measures are calculated while the assessment is being administered, and results are available immediately following the conclusion of the assessment. A summary of the performance statistics is available immediately after the conclusion of a test trial, accessed via the GUI as seen in Figure 2.

The outcome measures available immediately after the command following task include:

• •

  Average response delay: The average of all time differences between target command prompts and the first instance when the patient issues the correct command.

  $$\centering\begin{split}\qquad\qquad\qquad t_{d}&=\frac{1}{M}\sum_{m=1}^{M}\tilde{t}^{m}_{*}\\ \text{where}\quad\tilde{t}^{m}_{*}=&\operatorname*{arg\,min}_{t\in(0,T^{m}]}\{~{}\big{|}\textbf{u}^{t}_{h}-\hat{\textbf{u}}^{m}|<\epsilon~{}\}\end{split}\@add@centering$$

  $M$ is the total number of target command prompts, and $\hat{\textbf{u}}^{m}$ is the $m^{th}$ target command prompt which lasts for a duration of $T^{m}$. $\textbf{u}^{t}_{h}$ is the patient command at time $t$, and $\tilde{t}^{m}_{*}$ the time when this command first comes within tolerance $\epsilon\pm 5\degree$ of the target prompt. The clock restarts for each target command prompt.^†††When the dimensionality of $\hat{\textbf{u}}$ is greater than 1, the $arctan(\textbf{u}^{t}_{h},\hat{\textbf{u}}^{m})$ operator is used to compute the difference $~{}\big{|}\textbf{u}^{t}_{h}-\hat{\textbf{u}}^{m}|$.

• •

  Average successful response percent: The percentage of command prompts to which the patient is able to respond successfully.

  $$\begin{split}&\qquad\qquad\qquad r_{p}=100\cdot\frac{1}{M}{\sum_{m=1}^{M}I^{m}}\\ I^{m}&=\begin{cases}1,&\text{if $\exists\enskip t\in(0,T^{m}]\enskip s.t.~{}|\textbf{u}^{t}_{h}-\hat{\textbf{u}}^{m}|<\epsilon$}\\ 0,&\text{otherwise}.\end{cases}\end{split}$$

  where $I^{m}$ is a tracking index.

• •

  Average settling time: The time it takes until the patient continually issues the prompted command within an allowable tolerance of $\epsilon$.

  $$\begin{split}&\qquad\qquad t_{s}=\frac{1}{M}\sum_{m=1}^{M}t^{m}_{*}\\ \text{where}~{}~{}~{}t^{m}_{*}&=\operatorname*{arg\,min}_{t\in(0,T^{m}]}\{~{}\big{|}\textbf{u}^{k}_{h}-\hat{\textbf{u}}^{m}|<\epsilon~{}\forall~{}k\in[t,T^{m}]~{}\}\end{split}$$

  $t^{m}_{*}$ is the time from which the human command remains within tolerance.

• •

  Initial response accuracy: How close on average the first within-tolerance response is to the target prompt.

  $$a_{r}=\frac{1}{M}\sum_{m=1}^{M}[1-|\tilde{\textbf{u}}^{m}_{*}-\hat{\textbf{u}}^{m}|]$$

  where $\tilde{\textbf{u}}^{m}_{*}$ is the patient command at time $\tilde{t}^{m}_{*}$.

• •

  Average settled accuracy: How close on average the within-tolerance response is to the target command after having settled.

  $$a_{s}=\frac{1}{M\cdot(T^{m}-t^{m}_{*})}\sum_{m=1}^{M}\sum_{t=t^{m}_{*}}^{T^{m}}[1-|\textbf{u}^{t}-\hat{\textbf{u}}^{m}|]$$

  where ${\textbf{u}}^{t}$ is the within-tolerance response once settled and until the end of the trial $[t^{m}_{*},T^{m}]$.

Additionally, the detailed raw data used to calculate the summary statistics for each test condition are stored as an SQL file that can be accessed for further analysis at any time.

In terms of practicality, we designed our assessment tool to require minimal equipment which reduces cost, space requirements, and set-up time. The only equipment needed are the person’s own electric assistive machine (e.g., powered wheelchair), a tablet or any device able to run the assessment application, and our interfacing device (Sec. IV) if the interface is not already Bluetooth capable.

In terms of usability, we find that the full assessment can be completed in one session lasting between 30-60 minutes with the default settings, and without taxing the patient or the experimenter. The duration can be adjusted to be longer based on the configurable independent variables, as deemed appropriate by the experimenter. Simple and clear instructions for each task are contained within the app itself, so that the tester can easily administer the assessment. There have been no adverse incidents with the assessment tool as all tasks are simulation-based. The start and end of each test are also clearly defined.

Our motivation was to design an assessment tool that produces outcome measures that are repeatable, consistent, precise, and immune to test-taker bias. The outcome measures

are calculated automatically, which preserves scoring reliability across various test takers and sessions for a single patient profile. For example, to calculate the total percentage of time the simulated wheelchair is outside of the allowed bounds during the trajectory following task, we use definitive geometries (Fig. 3) with boundaries designed to account for human field-of-view limitations.