Efficient Spatial Representation and Routing of Deformable One-Dimensional Objects for Manipulation

Objects such as wires, cables, ropes, threads, and surgical sutures can be found in many industrial, surgical, construction, and everyday settings. In the literature, they are commonly called Deformable One-dimensional Objects (DOOs) or Deformable Linear Objects (DLOs). Automation of tasks involving rigid bodies had been extensively studied in industrial robotics; however, the need for further automation of manual tasks is forcing robotics applications to move towards working with DOOs, raising research interest in various robotics areas [1, 2, 3].

A crucial part of many robotics applications involving DOOs is routing [2, 4, 5]. Routing, also known as route planning, finds a viable path to change the initial state of a DOO to a goal state. Path in this context depends on the application and may mean a path in Euclidean space, a trajectory in the configuration space of the DOO, or a series of actions performed on the DOO.

Several representation methods have been used to capture the state of a DOO in route planning and manipulation problems. These representations range from spring-mass models [6] to linear combination of curves [7, 8], and fixed-length segments [9, 10].

Planning methods for routing DOOs for manipulation can be categorized into computational algorithms and approaches based on learning.

Computational routing methods are generally sampling-based approaches. These methods sample the space and use search methods such as Probabilistic Roadmaps (PRM) and Rapidly-exploring Random Trees (RRTs) for finding the route from the initial DOO state to the final state. Guo et al. [4] propose the RRT-BwC (Bi-direction with Constrain) planning algorithm to plan for aircraft cables assembly in narrow cabins with obstacles. Their method is based on the geometric formulation of the objects and the bi-directional RRT search to route in the high dimensional planning space. Amato et al. [5] find an approximate route by pre-computing a global roadmap using a variant of PRM, then refine the route by constrained sampling and applying adaptive forward dynamics. Moll and Kavraki [7] propose DOO planning using minimal-energy curves and a sampling-based planning method such as PRM. Roussel et al. [11] use quasi-static and dynamic models coupled with sampling-based methods to plan for an elastic rod manipulation. Koo et al. [12] and Ma et al. [13] apply RRT search-based approaches for routing and manipulation of DOOs. For all these methods, the solution can be statistically guaranteed, but they suffer from the curse of dimensionality, and in complex routing cases, the time and space complexity of the algorithms may make the algorithms slow and infeasible for many practical scenarios.

On the other hand, the learning-based methods primarily consist of learning-from-demonstration approaches, potentially working with any DOO type for any task. However, they are not generalizable and quickly fail when the experiment setup conditions even slightly deviate from the learning data [14, 15, 16].

Our contribution is proposing a novel approach to solving the DOO routing problem that is suitable for both offline and online routing due to its efficiency and speed. This approach relies on the geometrical decomposition of the task space into convex regions. It uses this discretized space to describe the DOO’s configuration using a compact sequence, which is simplified from the original 3-D continuous space description. Unlike the existing routing methods that have to find a solution by exploring a high-dimensional space, our new spatial representation allows utilizing a high-speed dynamic programming sequence matching method that reduces the planning delay to near zero, making it suitable for online planning for routing and manipulation tasks.

In the next section, we will describe the problem and our assumptions. Section III presents our spatial representation method. Section IV proposes our routing approach based on the described spatial representation method. Section V shows our experiments and presents our results. Finally, Section VI discusses the proposed approach and presents the roadmap for further improvements.

In the rest of this manuscript, we refer to the area where the task is taking place as work region. We assume that the exact positioning of a deformable one-dimensional object is only important inside the work region, and the details of its positioning outside this region are ignored. Additionally, we presume that the work region falls within the workspace of the manipulator robot, and the robot can access all of the work regions. Moreover, we assume that the work region is a "free space" with different components occupying some of its space. All the components’ positions, shapes, and dimensions are presumed to be known.

The exact positioning of a DOO in the work region is called its state, which contains the exact places in space occupied by a DOO.

Given the initial and desired states of a DOO in a work region, routing is finding the sequence of actions required to perform on the DOO to change its state from the initial state to the final state.

The actual DOO state is continuous and needs to be converted to a discrete spatial representation for robotics and computer applications. Common different ways of such representation include sampling equidistant points along the DOO’s medial axis, fixed-length B-splines, and fixed-length cylinders connected by spherical joints [4, 9, 17].

We propose to use a more efficient spatial representation based on the convex decomposition of the work region, combined with a fast sequence matching algorithm to solve the routing problem. Our proposed method is completely independent of the DOO dynamics and tries to embed the dynamics effects in the state representation. We relax the problem assuming that the slack of the DOO is not important. In other words, we assume that the application working with the DOO is not affected if the DOO has some extra slack in any area of the work region. For example, suppose a cable is not laying straight in a region and has a rather significant bending in a region. In that case, the extra bending is considered slack, and our algorithm does not consider it. We define slack as extra curves in DOO that do not pass around any components or anchor points; the slack can virtually be eliminated by creating tension in the DOO.

The following sections describe our proposed spatial representation and the routing method for DOOs.

Let us introduce a graph $G_{s}$ (called a spatial representation graph) to model the spatial representation of a deformable one-dimensional object passing through a work region.

The work region should be decomposed into convex subspaces to generate the vertices $V_{s}$ of the spatial representation graph $G_{s}$. These subspaces are called convex polygons in 2-D and convex polytopes in 3-D spaces. Each of these subspaces is a vertex in the graph $G_{s}$. If the work region is not enclosed (i.e., if a portion of DOO can lie outside the work region), a new vertex is added to $V_{s}$ to represent the "outside" region. There are many exact and approximate approaches for convex decomposition, and each can be used for this work [18, 19, 20, 21]. Figure 2 illustrates the convex decomposition on an example circuit board. Each of the components on the board is a node of the convex polygons generated from the board’s layout.

The generated convex regions allow efficiently defining subspaces in both 2-D and 3-D. It is desirable to represent the subspace only in 2-D when possible for simplicity. Much of the workspace in the finish line of industrial robotics is on a tabletop which can be approximated as a 2.5-D space. Meaning two dimensions are far more significant than the third dimension. There are specific scenarios where a 3-D work region can be simplified as a 2-D region with additional 3-D "tunnel"-like components such as bridges, passes, and tunnels. To allow the 2-D representation for these work regions, we can add a vertex to $V_{s}$ for each of the entrances of these components. Figure 3 shows all the vertices constructed from the example board of Figure 2(a) with the yellow dots representing the vertices of the entrances of the tunnel components.

Once all the vertices $V_{s}$ are defined, the edges $E_{s}$ for the spatial representation graph $G_{s}$ can be computed using the following rules:

• •

  Vertices from the neighbor convex regions (convex regions sharing a side) are connected with an edge.

• •

  Convex regions with a side not shared with any other convex region (i.e., convex regions surrounding the work region) are connected to the outside vertex.

• •

  Vertices for entrances of a tunnel component are connected to each other.

• •

  Each vertex for the tunnel entrances is connected to the vertex of the convex (or outside) region that it is lying on.

Figure 4 shows the spatial representation graph $G_{s}$ constructed for the circuit board of Figure 2(a).

Without the loss of generality, we assume that the size of $V_{s}$ is $n+1$, with the vertices numbered from $-1$ to $n-1$, and $-1$ reserved for the outside vertex.

Having computed the graph $G_{s}$, a DOO lying in the work region or passing through it can be represented by an ordered sequence $C$ of the vertex numbers it is passing through. We call this sequence representing the DOO as configuration of DOO. Note that if the DOO is bidirectional (does not have a pre-assigned head and tail), it can have two sequences for the same configuration that are reverse of each other. For example, the configuration of the DOO drawn on the circuit board in Figure 5 is $C=(-1,1,27,28,11,4,1,6,4,15,6,9,-1)$ or its reverse. Note that if the graph $G_{s}$ is computed correctly, every two consecutive vertices in $C$ should have an edge in $E_{s}$.

Based on the assumptions of the problem (see Section II), the extra slack of a DOO in each region is not encoded into its configuration. However, the extra slack is encoded if the DOO goes through some neighboring regions and comes back (e.g., if the DOO "touches" the neighbor convex region while passing through a convex region). Such instances are encoded as palindrome subsequences (i.e., subsequences that are the same if read backward or forward). Removing such subsequences may be desirable depending on the application and simplifies the configuration $C$ of a DOO in the work region.

We should note that the idea of convex decomposition in planning has been explored in other contexts before [22, 23, 24]. However, it is used differently here for defining the DOO configuration rather than planning itself.

Assume that the current configuration $C_{0}$ of a deformable one-dimensional object in a work region is provided along with the desired goal configuration $C_{g}$ of the DOO.

The problem is to route the DOO in the work region from the current configuration $C_{0}$ to the goal configuration $C_{g}$. A naive solution to the routing problem is to completely undo $C_{0}$ into a "free" DOO, then apply $C_{g}$ configuration by passing through all the vertices in $C_{g}$. However, this solution is inefficient and requires the maximum number of manipulative actions. A more efficient approach is to keep the matching areas between the current and goal configurations and only manipulate what is necessary to reduce the number of manipulative actions.

We propose utilizing the sequence matching algorithms to minimize the number of actions required to change from $C_{0}$ into $C_{g}$. Let us assume that the manipulator supports two motion primitives: 1) pick a DOO at a specific point, and 2) place the picked DOO at a specific point in the work region. Then the following actions on the configuration sequences can be applied:

• •

  Replacing the $i^{\text{th}}$ element $s_{i}$ in $C_{0}$ with the $j^{\text{th}}$ element $g_{j}$ in $C_{g}$: Pick the DOO where it is passing through vertex $s_{i}$ and place it at vertex $g_{j}$.

• •

  Removing the $i^{\text{th}}$ element $s_{i}$ in $C_{0}$ that does not correspond to an element in $C_{g}$: Remove the DOO from region $s_{i}$.

• •

  Inserting the $j^{\text{th}}$ element $g_{j}$ in $C_{g}$ that does not correspond to an element in $C_{0}$: Adding (i.e., stretching) the DOO to region $g_{j}$.

With these three actions, we propose modifying the well-known Levenshtein sequence distance algorithm [25] to obtain the manipulation actions required for routing.

The original Levenshtein algorithm computes the minimum required edits (i.e., replacement, deletion, and insertion) to convert the initial sequence to the final sequence. To return this minimum distance, Levenshtein’s dynamic programming method computes a matrix that retains the minimum number of edits required for converting the first $i$ elements of the initial sequence to the first $j$ elements of the final sequence. While the algorithm itself only computes the minimum number of edits, the types of edits can be extracted by backtracing this matrix once the algorithm is finished. Note that these edits are not unique, and backtracing will only output only one of the feasible solutions with the minimum number of actions.

To use the Levenshtein algorithm for the routing problem, the following modifications are required:

1. 1.

   When comparing two elements $s_{i}$ and $g_{i}$, they match if they are the same vertex number (i.e., $s_{i}=g_{i}$). However, if they are both $-1$ (the "outside" vertex), they only match if there is a common neighbor for them in the sequence (e.g., if $s_{i-1}=g_{i+1}$). In other words, the two outside regions are considered the same only if they are next to the same vertices. That same vertex may occur before or after $-1$.

2. 2.

   The cost for each action is set to $1$. However, for a tunnel-like component, the action cost of either of the operations depends on how many vertices come before and after it. In other words, for the $i^{\text{th}}$ element in the sequence of size $n$, the cost will be $2\times\min(i-1,n-i)+1$. For example, to remove the DOO from a tunnel-like region, it must free either the start of the DOO or the end of the DOO and put everything back again, bypassing the tunnel.

If the DOO is bi-directional, the algorithm should be repeated with one of the sequences reversed to get the least number of actions. Then, backtracing can give the actions needed to perform on the DOO to change its configuration from $C_{0}$ to $C_{g}$. The time and space complexities of the algorithm are $\mathcal{O}(nm)$, where $n$ and $m$ are the lengths of the current configuration ($|C_{0}|$) and goal ($|C_{g}|$) configuration sequences. Figure 6 shows the routing actions for a DOO to get from its current configuration to the goal configuration.

For realizing the computed actions, a single manipulator can act as below:

• •

  For replacement of element $s_{i}$ in $C_{0}$ with element $g_{j}$ in $C_{g}$: Pick the DOO where it is passing through vertex $s_{i}$ and place it at vertex $g_{j}$.

• •

  Removing element $s_{i}$ in $C_{0}$: Pick the DOO where it is passing through vertex $s_{i}$, and place it at the point where the goal DOO crosses from $g_{j-1}$ to $g_{j}$.

• •

  Inserting element $g_{j}$ in $C_{g}$: Pick the DOO where it is crossing $s_{i-1}$ into $s_{i}$, and place it at (i.e., stretch it to) vertex $g_{j}$.

The details on how these actions are implemented depend on the motion primitives of the robot and the environment.

We call the point on the DOO where the picking happens as picking point and the points where the two sequences match (i.e., no action is required) as fixed points. If more than one manipulator is available, the second manipulator can grab the closest fixed point before the picking point, and the third manipulator can grab to the closest fixed point after the picking point to prevent these points from moving.

The proposed routing algorithm does not incorporate the DOO dynamics and, therefore, cannot understand the result of the actions taken by the manipulator on the whole DOO configuration. To mitigate the lack of dynamics knowledge, routing and manipulation action can be performed iteratively until the current configuration matches the goal configuration. A single routing and then a manipulative action is performed at each iteration to get the DOO closer to the goal configuration.

To test the proposed method, we implemented it for a routing and manipulation task. The task includes a single-arm manipulating a cable on the circuit board of Figure 2(a) to change its current configuration to a goal configuration. This board is originally designed for task #3 of the Assembly Performance Metrics and Test Methods by the National Institute of Standards and Technology (NIST) to measure the capability of robotics systems for performing advanced manipulation on cables [26] and later adopted for the Robotic Grasping and Manipulation Competition in IROS 2020.

We performed many simulation experiments and several experiments with different settings on our robot. Each experiment included several iterations of routing and a manipulative action (i.e., pick and place actions) until the cable configuration had matched the given goal configuration.

We performed 200 simulation experiments, where we randomly placed a 0.3-0.5 $m$ cable on the 0.38 $m$ NIST board and randomly (in 170 tests) or manually (in 30 tests) placed a cable of the same length on the board as the goal configuration. The average number of actions over all the experiments was 4.34, and the maximum number of actions was 9. The processing time for each routing step was less than 1 $ms$ for all the experiments. Figure 6(b) illustrates an iteration of our simulated routing experiments.

For the robot experiments, we manually placed the cables on the NIST board. We considered automated layout detection for this board using real-time rectangle and ellipse detection [27] methods, but used manual specification by the user to save the implementation time. At each step of the planning, we utilized the DOO detection algorithm proposed in [9] to automatically detect the cable and extract its configuration on the circuit board. Then the goal configuration was manually given to the system. Our experiments showed that the method could also extend to real systems. Figure 1 shows the routing and manipulation experiments using our Universal Robots UR3 arm robot.

We presented a novel method for the spatial representation and routing of a deformable one-dimensional object that is efficient and fast. The proposed method decomposes the work region into convex polygons and polytopes. Then it uses the decomposition to encode the configuration of a DOO in this work region. The resulting configuration is a sequence of the regions the DOO passes from, effectively simplifying the routing algorithm to a modified sequence matching method with a quadratic time and space complexity. The iteration of our routing algorithm with manipulative actions can accomplish the desired routing and manipulation tasks. The low planning time and overhead makes it ideal for offline as well as online planning problems for routing and manipulation. Our experiments showed that the method could correctly plan the manipulation actions and achieve goal configurations of the DOO from various initial configurations.

The proposed approach is still in its infancy and can be extended further to cover many real-world tasks that are currently being addressed using slower and less efficient methods such as sampling-based planners. In its current iteration, this method can be used in tasks where the environment can be divided to separate convex regions and for a manipulator with two primitive actions: pick a point on the DOO, and place it in a specific point. The algorithm can be easily extended to include more manipulator motion primitives, such as wrapping the DOO around a component or passing through loops to create knots.

Finally, the proposed routing and manipulation algorithm ignores the dynamics of a cable. Although the routing algorithm itself can work well independent from the dynamics, in our real-world tests, we realized that considering the dynamics during manipulative tasks can help the system with performing the pick/place tasks. Additionally, further incorporating simple dynamics into the routing method’s cost calculation in the future can reduce the number of actions (i.e., iterations) required for performing a routing/manipulation task by more accurately predicting the result of the manipulation task at each iteration.

The authors would like to thank Wenzhao Lian for his support and help with robot experiments and implementations.