Planning-Assisted Context-Sensitive Autonomous Shepherding of Dispersed Robotic Swarms in Obstacle-Cluttered Environments

The success of robotic swarm shepherding relies on the modelling of sheep flocking behaviour and the design of sheepdog control strategies. The rules of BOIDS [22] [23] [24] are the most common sheep modelling method, where separation, cohesion and alignment of sheep are considered. To improve robustness when herding larger flocks, Harrison et al.[25] viewed the flock as an abstracted deformable shape, while Hu et al. [26] used adaptive protocols and artificial potential filed methods to model the sheep flocking behaviour.

The shepherding field of research has focused more on the design of sheepdog control strategies. As a representative work of rule-based shepherding methods, Strömbom et al’s shepherding algorithm [7] laid the foundations for many other shepherding methods, such as the modulation model in [8] and the Reinforcement Learning (RL) approach in [27]. Strömbom et al. [7] (described in section 3) simulated two typical sheepdog behaviours (collecting the dispersed sheep, and driving the aggregated sheep swarm to a specific location). They found that the mission completion time increases and the success rate decreases as the number of agents in the swarm increases.

A coordination algorithm was designed in [26] to employ multiple robotic sheepdogs to herd two flocks of sheep, which consists of 20 and 30 sheep, respectively. It was observed that the proposed algorithm could not handle the shepherding of a large flock. El-Fiqi et al. [13] investigated the influence of some key factors (e.g., the density of obstacles and the initial spatial distribution of sheep) on the complexity of shepherding and identified the limitations of reactive shepherding. It was suggested that an increase in the density of obstacles and the sheep’s initial level of dispersion in the environment escalate problem complexity and reduce mission success rate.

Learning-based methods have also been studied [10] [9] [28]. Go et al. [27] extended Strömbom et al.’s model by applying RL for learning the sheepdog’s behaviour policy. Hussein et al. [14] decomposed the shepherding problem into two sub-problems: learning to push an agent from a location to the destination and selecting whether to collect scattered agents or drive the largest flock to the destination. They aimed to reduce the problem’s complexity and proposed a curriculum-based RL to accelerate the learning process. However, the investigation of the swarm shepherding problem with multiple sub-swarms randomly dispersed in the obstacle-cluttered environment is still limited, and an efficient way to address this problem is lacking.

Mission planning approaches such as path planning algorithms have been well investigated and applied for mobile robots. The planning sub-problems (e.g., path planning, route planning, task assignment) involved in mission planning are defined, and the promising applications for swarm shepherding are discussed in [18]. Long et al. [1] also suggested that the sheepdog should take charge of high-level planning, such as path planning and task allocation for completing complex shepherding tasks. Some research on the applications of planning approaches for swarm shepherding exist [11] [12]. For instance, Lien and Pratt [2] presented a computer-human interactive motion planning method to address the shepherding problem. They observed that the planner lacks efficiency when the flock separates into several sub-groups. To modify the shepherding model in environments with obstacles, Elsayed et al. [20] presented a 2-stage differential evolution-based path planning algorithm that optimises the path for the sheepdog and sheep. They demonstrated that the path planning algorithm could reduce the time to complete the shepherding task.

TSP is a well-known NP-hard route planning problem that aims to find the route with the optimal cost for a salesperson to visit each city exactly once and returns to the initial city, given a set of cities and the travelling cost between each pair of cities [19]. TSP is a generalisation of or can be applied to many real-world problems, such as vehicle routing problem (VRP) [29], multi-robot task allocation [30], multi-regional coverage path planning [31], transportation and delivery [32]. Significant research has been conducted on TSP [33] [34]. Some effective approaches for solving the TSP include EC [35] [36] and swarm-intelligence algorithm such as Ant Colony Optimisation (ACO) [37], which has demonstrated its ability to solve TSP in multiple studies [21] [38] [39]. Many variants of TSP, such as Multiple TSP (MTSP) and Dynamic TSP (DTSP) [35], exist. For example, when there are multiple salespersons, the problem is called MTSP and can be further classified as single-depot and multi-depot based on where salespersons depart from [40]. Transformation methods have been used to convert a complex TSP problem to a classic single TSP where general and efficient TSP solvers exist [41]. Shepherding problems, especially with multiple sub-swarms, share some similarities with TSP. For example, there are some swarm locations (‘cities’) required to be visited by some agents (sheepdogs/salespersons) in both problems. However, to the best of our knowledge, TSP has not been applied to the robotic shepherding problem before.

Given a highly dispersed sheep swarm $\Pi$ in an environment, the first step in the planning-assisted shepherding framework is to group the dispersed sheep into some sub-swarms and locate the Local Centre of Mass (LCM) of each sub-swarm. The set of sheep sub-swarms is denoted as

where $Q$ is the number of sub-swarms and the sub-swarm $\phi_{q}$ subjects to $\bigcup_{q=1}^{Q}\phi_{q}=\Pi$, $\bigcap_{q=1}^{Q}\phi_{q}=\emptyset$ and $\phi_{q}\neq\emptyset,\ \forall q\in\{1,...,Q\}$. A sheep is assigned to a sub-swarm $\phi_{q}$ if it is within the cohesion range $R_{\Lambda}$ from any sheep of this sub-swarm. The LCM of $\phi_{q}$ at time step $t$ is calculated as

where $N_{s}^{q}$ is the number of sheep in the sub-swarm and $P_{\pi_{l}^{q}}^{t}$ is the position of the $l_{th}$ sheep grouped in $\phi_{q}$ at $t$. The LCM of each $\phi_{q}$ is regarded as a target location, which the sheepdog should visit.

After obtaining LCMs $\{LCM_{1}^{t},...,LCM_{q}^{t},...,LCM_{Q}^{t}\}$ of sub-swarms, the swarm shepherding problem can be transformed into a variant of TSP. This section discusses how to transform the single-sheepdog shepherding and bi-sheepdog shepherding problems to a single TSP and presents the mathematical formulation of the shepherding-transformed single TSP. Subsequently, the general TSP solver (presented in Section 5.1) can be employed to find the optimal push sequence of sheep sub-swarms to guide the sheepdog(s)’ behaviours.

Given the sequenced sub-swarms $\{\phi_{1}^{\prime},...,\phi_{q}^{\prime},...,\phi_{Q}^{\prime}\}$ and the corresponding LCMs $\{LCM_{1}^{\prime},...,LCM_{q}^{\prime},...,LCM_{Q}^{\prime}\}$, the mission of the sheepdog can be regarded as a set of sequential sub-tasks, i.e., pushing $\phi_{q}^{\prime}$ from $LCM_{q}^{\prime}$ to $LCM_{q+1}^{\prime}$, $\forall\leavevmode\nobreak\ q\in\{1,...,Q\}$. Path planning is crucial for both sheepdog(s) and sheep swarm to reduce detours and mission completion time. Next we present the mathematical formulation of path planning and discuss how to integrate the path planning algorithm into shepherding based on a proposed classification of sheepdog moving mode.

Based on the discussion above, the overall planning-assisted shepherding framework is presented in Algorithm 2, where $Mode=0$ represents the no-interaction mode while $Mode=1$ represents the interaction mode. Before the real-time shepherding, the offline planner obtains the grouping and sequencing results (lines 2 and 3). Then the sheepdog starts with the no-interaction mode for reaching the driving point of $\phi_{1}^{\prime}$.

During the shepherding process, the sheepdog switches between the no-interaction and the interaction mode based on context and to complete the set of sequenced sub-tasks, i.e., pushing the sheep sub-swarms $\phi_{q}^{\prime}$ to $LCM_{q+1}^{\prime}$, $\forall\leavevmode\nobreak\ q\in\{1,...,Q\}$. An adaptive switching approach is designed based on the real-time evaluation of the shepherding progress and is integrated into the shepherding framework. As presented in line 5-9 of Algorithm 2, the switching from the interaction mode to the no-interaction mode happens when $\phi_{q}^{\prime}$ encounters $\phi_{q+1}^{\prime}$, indicating that the sheepdog just successfully pushed $\phi_{q}^{\prime}$ to $LCM_{q+1}^{\prime}$ to merge with $\phi_{q+1}^{\prime}$ and is in preparation for the next sub-task by moving to the driving point of $\phi_{q+1}^{\prime}$. Then once the driving point of $\phi_{q+1}^{\prime}$ is reached in the no-interaction mode, the sheepdog switches to the interaction mode (line 27-29), which means that the sheepdog starts pushing the sub-swarm $\phi_{q+1}^{\prime}$. This process continues until all the sheep reach the goal area $LCM_{Q+1}^{\prime}$ or the pre-defined maximum time steps $T$ is reached. Figure 3 shows the corresponding flowchart.

Given the sheep swarm $\Pi$, a cohesion range-based grouping method as presented in Algorithm 3 is used to obtain the set of sheep sub-swarms and calculate the LCMs. Then MMAS [21], a well-known ACO algorithm, is introduced to address the shepherding-transformed TSP for getting the optimal push sequence of sub-swarms due to its outstanding performance in addressing TSP. ACO is inspired from the real ant colonies’  foraging behaviour. During the foraging process, ants deposited pheromone trails on the return routes if they find food sources. It enables other ants to find optimal paths to food sources by getting information from pheromone trails.

The core components of MMAS are solution construction and pheromone update. To find the optimal visiting sequence of sheep sub-swarms, the solution in MMAS is constructed by selecting sub-goals (regarded as a solution component $c_{q}^{q^{\prime}}$) one by one based on pheromone values $\tau_{qq^{\prime}}$ and the values $\eta(c_{q}^{q^{\prime}})$ of edges to form the complete solution, which indicates the visiting sequencing. Pheromone values $\tau_{qq^{\prime}}$ are updated every generation based on the quality of constructed solutions and the evaporation of existing pheromones. The values of $\eta(c_{q}^{q^{\prime}})$ are defined as $\eta(c_{q}^{q^{\prime}})=1/C_{qq^{\prime}}$ where $C_{qq^{\prime}}$ is the travelling cost from $LCM_{q}^{t}$ to $LCM_{q}^{\prime t}$. In MMAS, only the best ant is used to update the pheromone and the pheromone values are limited in the predefined ranges $[\tau_{min},\tau_{max}]$. The implementation of MMAS for finding the optimal push sequence of sub-swarms is described as follows:

Step 1: Initialise the parameters of MMAS, including the ant colony size $N_{s}$, two parameters $\alpha$ and $\gamma$, evaporation rate $\rho$, pheromone values of each edge $\tau_{qq^{\prime}}=\tau_{max}$, the values of each edge $\eta(c_{q}^{q^{\prime}})=1/C_{qq^{\prime}}$, the best solution $s^{*}=$ NULL;

Step 2: Construct new ant solutions:

Step 2.1: Start with a partial solution $s_{p}=LCM_{0}^{t}$;

Step 2.2: For each $LCM_{q^{\prime}}^{t},q^{\prime}\in\{1,2,...,Q\}$, calculate the probability $p(c_{q}^{q^{\prime}}|s_{p})$ of moving from the current location $LCM_{q}^{t}$ to $LCM_{q^{\prime}}^{t}$ based on the following Equation;

where $N(s_{p})$ is a set of available solution components for the current partial solution $s_{p}$;

Step 2.3: Select the next solution component based on the probability $p(c_{q}^{q^{\prime}}|s_{p})$ and add the selected $LCM_{q^{\prime}}^{t}$ to the current partial solution $s_{p}$;

Step 2.4: If all $LCM_{q^{\prime}}^{t},q^{\prime}\in\{1,2,...,Q\}$ are visited, add $LCM_{Q+1}^{t}$ to $s_{p}$ and go to Step 2.5; otherwise, go to Step 2.2;

Step 2.5: If $N$ new solutions are constructed, go to Step 3; otherwise, go to Step 2.1;

Step 3: Calculate the cost $F(s)$ and record the best solution $s*$ found with the lowest cost;

Step 4: Update the pheromone values according to:

where $\Delta\tau_{qq^{\prime}}^{best}=1/F(s*)$;

Step 5: If the termination condition is met, output the best solution which represents the optimal travelling sequence; otherwise, go to Step 2.

As discussed in Sections 4.3 and 4.4, path planning is crucial for reducing detours of both sheepdogs and sheep swarm, and is invoked during the online shepherding process. This section presents a two-layer path planning algorithm A*-PP, where the first layer, A*, finds the path with optimal cost, and the second layer, post-processing, eliminates the redundant waypoints in the path.

A* [44] is a well-known node-based path search algorithm that searches in a landscape represented by graphs. A* starts from the specific start node of the search graph and expands the nodes on candidate paths by adding one node at a time until it reaches the goal node in the graph. To decide which node on the candidate paths to be extended next, A* employs an evaluation function $f(n)$ which can be calculated as Equation (24) to estimate the cost of the path going through node $n$.

where $g(n)$ is the cost of the optimal path from the start node to the current node $n$ and $h(n)$ is the heuristic function for estimating the cost from the current node $n$ to the goal node. In this paper, $g(n)$ is calculated as following:

where $C\_L(n)$ and $C\_Th(n)$ are the length cost and threat cost. $h(n)$ is calculated as the straight line distance from the current node $n$ to the goal node, which is permissible to guarantee A* returns the optimal path.

The pseudo-code of A* is given in Algorithm 4. $Open$ is the set of nodes that can be considered for expansion. $Closed$ is the set of nodes that have been expanded, which makes sure that each node can be travelled at most once. $c(n,m)$ is the cost from the node $n$ to node $m$. $Parent(m)$ is to record the path with the lowest cost. A* starts the search from the initial point $W_{init}$. At each iteration of the main loop, A* selects the node $n$ with the lowest $f(n)$ from $Open$ and removes it from $Open$ to $Closed$. Then A* checks the neighbours of $n$ to insert feasible neighbour nodes $m$ into $Open$ if $m$ is not in $Open$, or update $f(m)$ if $m$ is already in $Open$ and $g(m)+h(m)$ is better than the old $f(m)$. The loop continues until the node with the lowest $f(n)$ is the goal point $W_{goal}$ or $Open$is empty, meaning no feasible path exists.

However, the original path obtained by A* usually contains many waypoints and the sub-path between two waypoints might be taking an unnecessary detour in some cases where a straight line can connect these two waypoints with no obstacle collision. Therefore, a path post-processing method, line of sight path pruning [45], is introduced to remove some redundant waypoints on the path to further reduce the path cost. The pseudo-code of the path post-processing is presented in Algorithm 5. The core of this process is to replace the original sub-path between two waypoints with a straight line if the straight line does not collide with any obstacles, meaning that the waypoints on the original sub-path, except the start point and end point, will be removed.

To evaluate the planning-assisted shepherding model and the hierarchical mission planning algorithm, experiments are conducted on a set of synthetic shepherding problems with different levels of complexity. Table 1 presents the details of the 20 benchmark problems, showing the environment size (mostly $100\times 100$), the number of sheep $N$ (20, 50, 100) and if obstacles are contained in each case. The benchmark set consists of three groups, the obstacle-free group, the obstacle-contained group with small swarms and the obstacle-contained group with large swarms. The cases in each group have an increasing level of complexity. Fig. 4 shows the visualised initialisation of each case with red dots representing the sheep, red asterisks representing sheepdogs, black areas denoting obstacles and a blue circle representing the goal area. Cases 1-6 are obstacle-free environments with an increasing level of complexity by varying the environment size, $N$, the goal location and the swarm initialisation. Cases 7-20 are obstacle-contained environments where the density of obstacles further impacts the problem’s complexity. The initialisation in cases 11 and 18 is based on randomly distributed sheep individuals, while the initialisation in other cases is based on randomly distributed sub-swarms.

For each problem instance, the experiments are conducted 20 times to capture the statistical behaviour. The number of the maximum time steps for each run is set to $T=300+20*N$. Three metrics are recorded to evaluate shepherding performance, including 1) SR: the success rate, i.e., number of times the shepherding mission was completed out of 20 runs; 2) No. of steps: the number of time steps consumed to complete the shepherding mission, and 3) path length: the total moving distance of the sheepdog. The mean and standard deviation of only the successful runs are presented for the no. of steps and path length.

The parameters in MMAS are set according to [21]. To be specific, the maximum number of iteration is set to 600; the population size is set to the problem dimension; $\alpha=1$, $\beta=2$, $\rho=0.98$, $\tau_{min}=1/D$, $\tau_{max}=1$. The number of neighbours in the A* search algorithm is set to 8, and the scaling factor is 1. Table 2 presents the setting of most parameters involved in the shepherding model by referring to [7]. The newly introduced parameter $R_{th}$ and the cost weights $\alpha_{1},\ \alpha_{2}$ are analysed in the following.