HI-GVF: Shared Control based on Human-Influenced Guiding Vector Fields for Human-multi-robot Cooperation

To derive the guiding vector field, we first suppose that the desired path can be described by an implicit function:

where $\phi:\mathbb{R}^{2}\to\mathbb{R}$ is twice continuously differentiable, and $\xi=(x,y)\in\mathbb{R}^{2}$ represents the two-dimensional position. Then a guiding vector field proposed in [17] to solve the path-following problem is defined by:

where $E$ is the $90^{\circ}$ rotation matrix $\left[\begin{smallmatrix}0&-1\\ 1&0\end{smallmatrix}\right]$, $\gamma\in\{1,-1\}$ is used to specify the propagation direction along the desired path, $k$ is an adjustable parameter. The first term of (2) is the tangential component, enabling the robot to move along the desired path, while the second term of (2) is the orthogonal component, assisting the robot to move closer to the desired path. Therefore, intuitively, the guiding vector field can guide the robot to move toward and along the desired path at the same time.

When the desired path is occluded by obstacles, a robot needs to deviate from the desired path to avoid obstacles, and then return to the desired path. To address this problem, we design a smooth composite vector field, consisting of vector fields generated by the desired path and obstacles and integrated with two bump functions [18].

Finding and prioritizing the most threatening fire source to extinguish is the primary task for the MRS. Each robot prioritizes the fire to be extinguished based on its perception of the fire situation, and generates the robot intention $\boldsymbol{v}_{t}^{i}$, which points to the largest fire source observed by the $i$-th robot.

Suppose there are $N$ homogeneous omnidirectional robots, with positions denoted by $\boldsymbol{p}_{r_{i}}\in\mathbb{R}^{2},i\in\mathbb{Z}_{1}^{N}$, $M$ fire sources, with positions denoted by $\boldsymbol{p}_{s_{i}}\in\mathbb{R}^{2},i\in\mathbb{Z}_{1}^{M}$, $P$ circle obstacles, with positions denoted by $\boldsymbol{p}_{o_{i}}\in\mathbb{R}^{2},i\in\mathbb{Z}_{1}^{P}$, and $Q$ barriers, with positions denoted by $\boldsymbol{p}_{b_{i}}\in\mathbb{R}^{2},i\in\mathbb{Z}_{1}^{Q}$, in the two-dimensional space. In this paper, the observed severity of the $j$-th fire source by the $i$-th robot is described by the observed fire value $\theta(\boldsymbol{p}_{r_{i}},\boldsymbol{p}_{s_{j}})\geq 0$. This value increases if the fire becomes more severe and decreases if obstacles obstruct the robot’s observation. It is worth emphasizing that, in practice, the observed fire value is directly measured by the onboard sensors. However, for simplicity, we use a circle centered at the fire source to represent the severity of the fire; the more severe the fire, the larger the radius. Then the observed fire value is approximated as the angle calculated by subtracting the angle of view obscured by the obstacle ($\alpha$ in Fig. 7) from the angle of view of the fire ($\beta$ in Fig. 7) when there are no obstacles. Note that if $\alpha>\beta$, i.e., the fire source is totally obstructed by obstacles, then the observed fire value is zero. Fig. 7 graphically illustrates how the fire value is determined in an environment with fire sources, robots and obstacles. Therefore, the $i$-th robot can select the most severe fire source as its prioritized target $s^{i}$, which is defined as:

In some cases, robots may not choose the same target. For example, in Fig. 7, two robots choose ➀ as their prioritized target while one robot has a different target: ➁. Then, similar to method (2), we define a vector field for the $i$-th robot based on the target $s^{i}$:

where $\phi_{s^{i}}(\xi)=|\xi-\boldsymbol{p}_{s^{i}}|^{2}-{R_{s^{i}}}^{2}$, and $\boldsymbol{p}_{s^{i}}$, $R_{s^{i}}$ represent the position and radius of the prioritized target fire source $s^{i}$, respectively. Note that the tangential component is removed and only the orthogonal component remains. Therefore, following this vector field (11), the robot would move directly towards the fire target.

As some robots have different intentions, it may cause them to be separated from the team for too long or too far away, thus breaking away from the formation and resulting in a failure to work together to accomplish the mission. To address this problem, we adopt the distance-based formation control method to maintain the formation of MRS while moving.

Then, using (2), we can define a composite vector field for the $i$-th robot based on the neighbors of the $i$-th robot:

where $d_{ij}$ represents the desired distance between the $i$-th and $j$-th robot, $\phi_{r_{j}}(\xi)=\|\xi-\boldsymbol{p}_{r_{j}}\|^{2}-d_{ij}^{2}$, $\hat{(\cdot)}$ is the normalization notation (i.e., it normalizes the gradient $\nabla\phi_{r_{j}}$), and $\mathcal{N}_{i}$ represents the neighbor set of the $i$-th robot²²2If a robot has a (bidirectional) communication link with the $i$-th robot, then it is a neighbor of the $i$-th robot. For example, when a communication range $R_{c}$ is set, other robots within the range are neighbors of $i$-th robot. Human operators can also initialize the communication connection topology among robots before the task execution.. Due to the constraints of formation control, robots with greater robot intention will dominate when each robot moves to its selected target. Therefore, the target selection of the entire MRS will eventually become consistent.

Then, similar to (9), the VF induced by the target (i.e., the robot intention), and the VF for formation control are defined respectively as follows:

In the upper layer of the framework, we use an intention field model to allow the human operator to control one robot and to propagate the intention among the robots, thus affecting the whole system. As stated in the preceding sections, $\boldsymbol{v}^{t}_{i}\in\mathbb{R}^{2}$ obtained by (13) represents the robot intention, and $\boldsymbol{v}^{h}_{i}\in\mathbb{R}^{2}$ obtained by (9) represents the human intention received by the human-influenced robot. In each control cycle, the $i$-th robot updates the shared intention $\boldsymbol{v}_{i}^{s}$ at time step $k+1$ as follows³³3For simplicity, the notation of the shared intention $\boldsymbol{v}_{i}^{s}(\xi(k+1))$ at time step $k+1$ is simplified as $\boldsymbol{v}_{i}^{s}(k+1)$; the same simplification is used for other notation. :

where $\boldsymbol{v}_{i}^{s}(0)=\boldsymbol{0}$, $\omega_{0},\omega_{1},\omega_{2},\omega_{3}\in\mathbb{R}_{+}$ are weighted coefficients that regulate the degree of influence of the robot intention, the neighbor intention, and the human intention in the update process, and $\Psi_{i}$ is an indicator function that $\Psi_{i}=1$ holds when the $i$-th robot has received the human intention $\boldsymbol{v}_{h}^{i}$ and $\Psi_{i}=0$ otherwise, and $\Phi(\cdot)$ is a dead zone function defined as:

where $\epsilon\geq 0$. The intention field model can help the human operator to control effectively by abstracting the human input as an intention field value and fusing it with the robot intention. The shared intention can be viewed as a virtual control intention generated by robot intentions and human intentions. Assuming that the desired robot motion speed is $\mathcal{C}>0$, the final shared intention can be represented as

The model has the following properties:

• 1)

  Local similarity. If the $i$-th robot is close to the $j$-th robot, then $\boldsymbol{v}_{s}^{i}$ and $\boldsymbol{v}_{s}^{j}$ have similar values.

• 2)

  Space separation. If the $i$-th robot is far from the $j$-th robot, then variation in $\boldsymbol{v}_{s}^{i}$ has little impact on $\boldsymbol{v}_{s}^{j}$.

Local similarity allows nearby robots to be controlled simultaneously by a human operator, while space separation enables different parts of the MRS to be controlled separately. With the intention field model, each robot updates itself under the influence of its neighbors and the human operator. When one robot receives the human intention, the inter-robot network implicitly propagates that intention to each robot. This result is fundamentally different from the “leader-follower” model, in which followers just react passively to the actions of their neighbors. The stability analysis of the intention field can be referred to in the Appendix-A.

In the lower layer of the framework, a consensus network is used to blend the shared intention generated by the upper layer and VF of formation control generated by the local controller. We can obtain the final velocity $\boldsymbol{v}^{i}$ as follows:

where $\lambda_{i}=\lambda(\|\boldsymbol{v}_{i}^{s}\|,\|\boldsymbol{v}_{i}^{f}\|)$, $\boldsymbol{v}_{i}^{s}$, $\boldsymbol{v}_{i}^{f}$ and $\hat{\boldsymbol{v}}_{i}^{s}$ are obtained by (15), (14) and (17), respectively.

The weighting function $\lambda:\mathbb{R}^{+}\times\mathbb{R}^{+}\to[0,1]$ need to satisfy the following conditions:

• 1)

  $\lambda$ is continuous, non-negative, and $\lambda(0,0)=0$.

• 2)

  Let $f_{b}(a)=\lambda(a,b)$, then $f_{b}(a)$ is strictly increasing.

• 3)

  Let $f_{a}(b)=\lambda(a,b)$, then $f_{a}(b)$ is strictly decreasing.

• 4)

  $\lim_{a\to\infty}f_{b}(a)=1$ and $\lim_{b\to\infty}f_{a}(b)=0$

We design a function that satisfies these conditions

where $k_{s},k_{f}\in R^{+}$ are constants.

The final velocity $\boldsymbol{v}_{i}$ integrates all the VFs so that with suitable policy-blending, the MRS can maintain its formation and avoid obstacles while reaching the target. The stability analysis of the consensus network can be referred to in the Appendix-B.

Although formation control can keep robots at a distance from each other to some extent, it does not guarantee that robots will not collide with each other at all. Therefore, we still need to consider collision avoidance within the formation. By first defining a set of safe states, we use the control barrier function (CBF) to formally guarantee forward invariance of the desired set, i.e., if the system starts in the safe set, it stays in the safe set. A CBF is similar to a Control Lyapunov Function in that they ensure certain properties of the system without the need to explicitly compute the forward reachable set.

For $i,j\in\mathbb{Z}_{1}^{N},i\neq j$, we define a function

where $R_{r}>0$ is the safe distance among robots. The function $\mathfrak{h}_{ij}$ reflects whether two robots maintain a safe distance. Therefore, the pairwise safe set can be defined as

Using $\mathfrak{h}_{ij}$, we can define a barrier function

We use the optimization method to adjust the final velocity $\boldsymbol{v}_{i}$ so that the new velocity $\hat{\boldsymbol{v}}_{i}$ enables collision-avoidance. Specifically, the new velocity $\hat{\boldsymbol{v}}_{i}$ should subject to the constraint $\dot{B}_{ij}\leq\alpha/B_{ij}$, where $\alpha>0$ is a constant. Using (22), we can obtain $\dot{B}_{ij}=-\dot{\mathfrak{h}}_{ij}/\mathfrak{h}_{ij}^{2}=-2(\boldsymbol{p}_{r_{i}}-\boldsymbol{p}_{r_{j}})^{\top}(\hat{\boldsymbol{v}}_{i}-\hat{\boldsymbol{v}}_{j})/\mathfrak{h}_{ij}^{2}$, so the constraint $\dot{B}_{ij}\leq\alpha/B_{ij}$ can be rewritten as

In order to ensure that the main control objectives are met to the greatest extent possible, the collision avoidance strategy should modify the target velocity as little as possible. Therefore, we calculate the new velocity $\hat{\boldsymbol{v}}=(\hat{\boldsymbol{v}}_{1},\cdots,\hat{\boldsymbol{v}}_{N})\in\mathbb{R}^{2N}$ by the quadratic program (QP) as follows:

This optimization problem is centralized as it requires the positions and desired velocities of all robots. Further, we can improve it to a distributed form for each robot $i\in\mathbb{Z}_{1}^{N}$:

Therefore, robot $i$ only considers the positions of its neighbors, and does not need to get the desired velocity of other robots.

Note that the safety barrier certificates can also deal with static or moving obstacles by treating obstacles as agents with no inputs. Similar to the method of collision avoidance among robots, the robot needs to maintain a safe distance between obstacles. Therefore, a function can be designed to ensure that the robot $i$ does not collide with the obstacle $\boldsymbol{o}_{k}$

where $R_{o}$ is the safe distance between the robot and the obstacle.

Similar to (23), we can obtain the constraint of robot avoiding obstacles:

where $\beta>0$ is a constant. Then, the new velocity $\hat{\boldsymbol{v}}_{i}$ can be calculated by the QP as follows:

It is worth noting that the collision avoidance behavior may affect the convergence of the robot formation, and even deadlocks may occur at certain positions (the deadlock problem is still an intractable challenge at this point), resulting in robots not being able to move. Thus, to ensure that multi-robot cooperative tasks and collision avoidance behavior are feasible, we assume that the required parameters have been properly designed, that the initial Euclidean distances between the robots are all greater than the safe distance $R_{r}$, and that the initial distances between robots and obstacles are all greater than the safe distance $R_{o}$. When the robots are unable to move due to the deadlock problem, we temporarily disengage the robots from the deadlock position by increasing the human intent.

We design a human-computer interface based on QT for simulation experiments as shown in Fig. 8. The operator can arbitrarily initialize the robot position and connection, modify parameters, etc. We set up the following three types of experiments to verify the effectiveness of the algorithm.

As we mentioned earlier, we assume that in a scenario of human-multi-robot collaboration, humans obtain a global view through the UAV, thus guiding the robots to accomplish the task together using multiple human-robot interactions. We briefly overview the brain-computer interfaces, myoelectric wristbands, and eye-tracking devices used.

Based on our self-designed human-multi-robot interaction system, we conduct physical robotics experiments. All participants were provided with written informed consent to participate in the experiment in accordance with the Declaration of Helsinki. We consider the following two scenarios: (1) The desired path is given in advance, and the robot moves along the path motion and deviates from the path when the target is found. (2) While the robot is executing the task, a human intervenes to change the execution sequence.

Consider the robot as a first-order model, i.e., $\dot{\boldsymbol{x}}_{i}=\boldsymbol{v}_{i}$. Equation (15) can be rewritten as follows:

For the whole robot formation, the state can be represented as $\boldsymbol{x}=[\boldsymbol{x}_{1}^{\top},\cdots,\boldsymbol{x}_{N}^{\top}]^{\top}\in\mathbb{R}^{2N}$. Let $\Psi=\text{diag}({\Psi_{i}})\otimes I_{2}\in\mathbb{R}^{2N\times 2N}$, $\Phi(\boldsymbol{x})=[\phi(\boldsymbol{x}_{1})^{\top},\cdots,\phi(\boldsymbol{x}_{N})^{\top}]^{\top}$, $\boldsymbol{v}_{s}=[{\boldsymbol{v}^{s}_{1}}^{\top},\cdots,{\boldsymbol{v}^{s}_{N}}^{\top}]^{\top}$, $\boldsymbol{v}_{t}=[{\boldsymbol{v}^{t}_{1}}^{\top},\cdots,{\boldsymbol{v}^{t}_{N}}^{\top}]^{\top}$, $\boldsymbol{v}_{h}=[{\boldsymbol{v}^{h}_{1}}^{\top},\cdots,{\boldsymbol{v}^{h}_{N}}^{\top}]^{\top}$, where $\otimes$ is the Kronecker product. Then, (32) becomes

where $\widetilde{D}=D\otimes I_{2}$, D is the incidence matrix. We use the Lyapunov method for stability analysis of the intention field model.

According to the consensus theory [30], the system dynamics of the consensus network can be represented as

where $\boldsymbol{x}=[\boldsymbol{x}_{1}^{\top},\cdots,\boldsymbol{x}_{N}^{\top}]^{\top}\in\mathbb{R}^{2N}$ is the state set; $\Delta=[\Delta_{1}^{\top},\cdots,\Delta_{N}^{\top}]$, $\Delta_{i}=\boldsymbol{v}_{i}-\boldsymbol{v}_{i}^{f}=\lambda_{i}(\hat{\boldsymbol{v}}_{i}^{s}-\boldsymbol{v}_{i}^{f})$; $\widetilde{L}=L\otimes I_{2}$, $L$ is the Laplacian matrix; $\boldsymbol{e}=\widetilde{D}^{\top}\boldsymbol{x}$ is the consensus error. And we can easily obtain Lemma 1.

For the designed $\lambda$ function in (19), we have Lemma 2.

Combining Theorem 1 and Theorem 2, we obtain corollary: