An Easy-to-use Scalable Framework for Parallel Recursive Backtracking

Parallel computation is becoming increasingly important as performance levels out in terms of delivering parallelism within a single processor due to Moore’s law. This paradigm shift means that to attain speed-up, software that implements algorithms that can run in parallel on multiple cores is required. Today we have a growing list of supercomputers with tremendous processing power. Some of these systems include more than a million computing cores and can achieve up to 30 Petaflop/s. The constant increase in the number of cores per supercomputer motivates the development of parallel algorithms that can efficiently utilize such processing infrastructures. Unfortunately, migrating known serial algorithms to exploit parallelism while maintaining scalability is not a straightforward job. The overheads introduced by parallelism are very often hard to evaluate, and fair load balancing is possible only when accurate estimates of task “hardness” or “weight” can be calculated on-the-fly. Providing such estimates usually requires problem-specific knowledge, rendering the techniques developed for problem $A$ useless when trying to parallelize an algorithm for problem $B$.

As it is not likely that polynomial-time algorithms can be found for NP-hard problems, the search for fast deterministic algorithms could benefit greatly from the processing capabilities of supercomputers. Researchers working in the area of exact algorithms have developed algorithms yielding lower and lower running times [1, 2, 3, 4, 5]. However the major focus has been on improving the worst-case behavior of algorithms (i.e. improving the best known asymptotic bound on the running time of algorithms). The practical aspects of the possibility of exploiting parallel infrastructures has received much less attention.

Most existing exact algorithms for NP-hard graph problems follow the well-known branch-and-reduce paradigm. A branch-and-reduce algorithm searches the complete solution space of a given problem for an optimal solution. Simple enumeration is normally prohibitively expensive due to the exponentially increasing number of potential solutions. To prune parts of the solution space, an algorithm uses reduction rules derived from bounds on the function to be optimized and the value of the current best solution. The reader is referred to Woeginger’s excellent survey paper on exact algorithms for further detail [6]. At the implementation level, branch-and-reduce algorithms translate to search-tree-based recursive backtracking algorithms. The search tree size usually grows exponentially with either the size of the input instance $n$ or, when the problem is fixed-parameter tractable, some integer parameter $k$ [7]. Nevertheless, search trees are good candidates for parallel decomposition. Given $c$ cores, a brute-force parallel solution would divide a search tree into $c$ subtrees and assign each subtree to a separate core for sequential processing. One might hope to thus reduce the overall running time by a factor of $c$. However, this intuitive approach suffers from several drawbacks, including the obvious lack of load balancing.

Even though our focus is on NP-hard graph problems, we note that recursive backtracking is a widely-used technique for solving a very long list of practical problems. This justifies the need for a general framework to simplify the migration from serial to parallel. One example of a successful parallel framework for solving a different kind of problem is MapReduce [8]. The success of the MapReduce model can be attributed to its simplicity, transparency, and scalability, all of which are properties essential for any efficient parallel framework. In this paper, we propose a simple, lightweight, scalable framework for transforming almost any recursive backtracking algorithm into a parallel one with minimal communication overhead and a load balancing strategy that is implicit, i.e., does not require any problem-specific knowledge. The key idea behind the framework is the use of an indexed search-tree approach that is oblivious to the problem being solved. To test our framework, we use it to implement parallel exact algorithms for the well-known Vertex Cover and Dominating Set problems. Experimental results show that, on sufficiently hard instances, we obtain linear speedups for at least 32,768 cores.

Typically, a recursive backtracking algorithm exhaustively explores a search tree $T$ using depth-first search traversal and backtracking at the leaves of $T$. Each node of $T$ (a search-node) can be denoted by $N_{d,p}$ for $d$ the depth of $N_{d,p}$ in $T$ and $p$ the position of $N_{d,p}$ in the left-to-right ordering of all search-nodes at depth $d$. The root of $T$ is thus $N_{0,0}$. We use $T(N_{d,p})$ to denote the subtree rooted at node $N_{d,p}$. We say $T$ has branching factor $b$ if every search-node has at most $b$ children. An example of a generic serial recursive backtracking algorithm, Serial-RB, is given in Figure 1. The goal of this paper is to provide a framework to transform Serial-RB into an efficient parallel algorithm with as little effort as possible. For ease of presentation, we make the following assumptions:

• -

  Serial-RB is solving an NP-hard optimization problem (i.e. minimization or maximization) where each solution appears in a leaf of the search tree.

• -

  The global variable $best\_so\_far$ stores the best solution found so far.

• -

  The IsSolution($N_{d,p}$) function returns true only if $N_{d,p}$ is a solution which is “better” than $best\_so\_far$.

• -

  The search tree explored by Serial-RB is binary (i.e. every search-node has at most two children).

In a parallel environment, we denote by ${\cal C}=\{C_{0},C_{1},\ldots,C_{c}\}$ the set of available computing cores. The rank of $C_{i}$ is equal to $i$ and $|{\cal C}|=c$. We use the terms worker and core interchangeably to refer to some $C_{i}$ participating in a parallel computation. Each search-node in $T$ corresponds to a task, where tasks are exchanged between cores using some specified encoding. We use $E(N_{d,p})$ to denote the encoding of $N_{d,p}$. When search-node $N_{d,p}$ is assigned to $C_{i}$, we say $N_{d,p}$ is the main task of $C_{i}$. Any task in $T(N_{d,p})$ sent from $C_{i}$ to some $C_{j}$, $i\neq j$, is a subtask for $C_{i}$ and becomes the main task of $C_{j}$. The weight of a task, $w(N_{d,p})$, is a numerical value indicating the estimated completion time for $N_{d,p}$ relative to other tasks. That is, when $w(N_{d,p})>w(N_{d^{\prime},p^{\prime}})$, we expect the exploration of $T(N_{d,p})$ to require more computational time than $T(N_{d^{\prime},p^{\prime}})$. Task weight plays a crucial role in the design of efficient dynamic load-balancing strategies [9, 10, 11]. Without any problem-specific knowledge, the “best” indicator of the weight of $N_{d,p}$ is nothing but $d$ since estimating the size of $T(N_{d,p})$ is almost impossible. We capture this notion by setting $w(N_{d,p})={1\over{d+1}}$. We say a task is heavy if it has weight close to $1$ and light otherwise.

From the standpoint of high-performance computing, practical parallel exact algorithms for hard problems mean one thing: unbounded scalability. The seemingly straight-forward parallel nature of search-tree decomposition is deceiving: previous work has shown that attaining scalability is far from easy [12, 13, 14]. To the best of our knowledge, the most efficient existing parallel algorithms that solve problems similar to those we consider were only able to scale to less than a few thousand (or only a few hundred) cores [10, 11, 15]. One of our main motivations was to solve extremely hard instances of the Vertex Cover problem such as the 60-cell graph [16]. In earlier work, we first attempted to tackle the problem by improving the efficiency of our serial algorithm [17]. Alas, some instances remained unsolved and some required several days of execution before we could obtain a solution. The next obvious step was to attempt a parallel implementation. As we encountered scalability issues, it became clear that solving such instances in an “acceptable” amount of time would require a scalable algorithm that can effectively utilize much more than the 1,024-core limit we attained in previous work [11].

We discuss the lessons we have learned and what we believe to be the main reasons of such poor scalability in Section III. In Section IV, we present the main concepts and strategies we use to address these challenges under our parallel framework. Finally, implementation details and experimental results are covered in Sections V and VI, respectively.

In this section, we present our framework by showing how to incrementally transform Serial-RB into a parallel algorithm. First, we discuss indexed search trees and their use in a generic and compact task-encoding scheme. As a byproduct of this encoding, we show how we can efficiently guarantee the extraction of the heaviest unprocessed task for dynamic load balancing. We provide pseudocode to illustrate the simplicity of transforming serial algorithms to parallel ones. The end result is a parallel algorithm, Parallel-RB, which consists of two main procedures: Parallel-RB-Iterator and Parallel-RB-Solver.

We tested our framework with parallel implementations of algorithms for the well-known Vertex Cover and Dominating Set problems.

Both problems have received considerable attention in the areas of exact and fixed parameter algorithms because of their close relations to many other problems in different application domains [20]. The sequential algorithm for the parameterized version of Vertex Cover having the fastest known worst-case behavior runs in ${\mathcal{O}}(kn+1.2738^{k})$ time [3]. We converted this to an optimized version by introducing simple modifications and excluding complex processing rules that require heavy maintenance operations. For Dominating Set, we implemented the algorithm of [4] where the problem is solved by a reduction to Minimum Set Cover. We used the hybrid graph data-structure [17] which was specifically designed for recursive backtracking algorithms that combines the advantages of the two classical adjacency-list and adjacency-matrix representations of graphs with very efficient implicit backtracking operations.

Our input consists of a graph $G=(V,E)$ where $|V|=n$, $|E|=m$, and each vertex is given an identifier between $0$ and $n-1$. The search tree for each algorithm is binary and the actual implementations closely follow the Parallel-RB algorithm. At every search-node, a vertex $v$ of highest degree is selected deterministically. Vertex selection has to be deterministic to meet the requirements of the framework. To break ties when multiple vertices have the same degree, we always pick the vertex with the smallest identifier. For Vertex Cover, the left branch adds $v$ to the solution and the right branch adds all the neighbors of $v$ to the solution. For Dominating Set, the left branch is identical but the right branch forces $v$ to be out of any solution. The ConvertIndex function is straightforward as the added, deleted, or discarded vertices can be retraced by iterating through the index and applying any appropriate reduction rules along the way. Every time a smaller solution is found, the size is broadcasted to all participants to avoid exploring branches that cannot lead to any improvements.

All our code relies on the Message Passing Interface (MPI) [21] and uses the standard C language with no other dependencies. Computations were performed on the BGQ supercomputer at the SciNet HPC Consortium¹¹1SciNet is funded by the Canada Foundation for Innovation under the auspices of Compute Canada; the Government of Ontario; Ontario Research Fund - Research Excellence; and the University of Toronto. [22]. The BGQ production system is a $3^{rd}$ generation Blue Gene IBM supercomputer built around a system-on-a-chip compute node. There are 2,048 compute nodes each having a 16 core 1.6GHz PowerPC based CPU with 16GB of RAM. When running jobs on 32,768 cores, each core is allocated 1GB of RAM. Each core also has four “hardware threads” which can keep the different parts of each core busy at the same time. It is therefore possible to run jobs on $65,536$ and $131,072$ cores at the cost of reducing available RAM per core to 500MB and 250MB, respectively. We could run experiments using this many cores only when the input graph was relatively small and, due to the fact that multiple cores were forced to share (memory and CPU) resources, we noticed a slight decrease in performance.

The Parallel-Vertex-Cover algorithm was tested on four input graphs.

• -

  p_hat700-1.clq: is a graph on $700$ vertices and $234,234$ edges with a minimum vertex cover of size $635$

• -

  p_hat1000-2.clq is a graph on $1,000$ vertices and $244,799$ edges with a minimum vertex cover of size $946$

• -

  frb30-15-1.mis is a graph on $450$ vertices and $17,827$ edges with a minimum vertex cover of size $420$

• -

  60-cell is a graph on $300$ vertices and $600$ edges with a minimum vertex cover of size $190$

The first two instances were obtained from the classical Center for Discrete Mathematics and Theoretical Computer Science (DIMACS) benchmark suite (http://dimacs.rutgers.edu/Challenges/). The frb30-15-1.mis graph is a notoriously hard instance for which the exact size of a solution was only known so far from a theoretical perspective. To the best of our knowledge, this paper is the first to experimentally solve it; more information on this instance can be found in [23]. Lastly, the 60-cell graph is a 4-regular graph (every vertex has exactly $4$ neighbors) with applications in chemistry [16]. Prior to this work, we solved the 60-cell using a serial algorithm which ran for almost a full week [17]. The high regularity of the graph makes it very hard to apply any pruning rules resulting in an almost exhaustive enumeration of all feasible solutions. For the Parallel-Dominating-Set algorithm we generated two random instances 201x1500.ds and 251x6000.ds where $n$x$m$.ds denotes a graph on $n$ vertices and $m$ edges. Both instances could not be solved by our serial algorithm when limited to 24 hours.

All of our experiments were limited by the system to a maximum of 24 hours per job. To evaluate the performance of our communication model and dynamic load balancing strategy, we collected two statistics from each run: $T_{S}$ and $T_{R}$. $T_{S}$ denotes the average number of tasks received (and hence solved) by each core while $T_{R}$ denotes the average number of tasks requested by each core. In Table I, we give the running times of the Parallel-Vertex-Cover algorithm for every instance while varying the total number of cores, $|C|$, from $2$ to $131,072$ (we only ran experiments on 65,536 or 131,072 cores when the graph was small enough to fit in memory or when the running time exceeded 10 minutes on 32,768 cores).

The values of $T_{S}$ and $T_{R}$ are also provided. Similar results for the Parallel-Dominating-Set algorithm are given in Table II. Due to space limitations, we omit some of the entries in the tables and show the overall behaviors in the chart of Figure 9. In almost all cases, the algorithms achieve near linear (super-linear for the 60-cell graph) speedup on at least 32,768 cores. We were not able to gather enough experimental data to characterize the behavior of the framework on 65,536 and 131,072 cores but results on 201x1500.ds and frb30-15-1.mis suggest a 10 percent decrease in performance. As noted earlier, cores were forced to share resources whenever $|C|$ was greater than 32,768. Further work is needed to determine whether this can explain the performance decrease or if other factors came into play. We also note that harder instances are required to fairly test scalability on a larger number of cores since the ones we considered were all solved in just a few minutes using at most 32,768 cores. In Figure 10, we plot the different values of $T_{S}$ and $T_{R}$ for a representative subset of our experiments. This chart reveals the inherent difficulty of dynamic load balancing. As $|C|$ increases, the gap between $T_{S}$ and $T_{R}$ grows larger and larger. Any efficient dynamic load-balancing strategy has to control the growth of this gap (e.g. keep it linear) for a chance to achieve unbounded scalability. The largest gap we obtained was on the frb30-15-1.mis instance using 131,072 cores. The gap was approximately 300,000. Given the number of cores and the amount of time ($>$ 10 minutes) spent on the computation, the number suggests that each core requested an average of 2.5 tasks from every other core. One possible improvement which we are currently investigating is to modify our virtual topology to a graph-like structure of bounded degree. The framework currently assumes a fully connected topology after initialization (i.e. any two cores can communicate) which explains the correlation between $|C|$ and $|T_{S}-T_{R}|$. By bounding the degrees in the virtual topology, we hope to make this gap weakly dependent on $|C|$.

Combining indexed search-trees with a decentralized communication model, we have showed how any serial recursive backtracking algorithm, with some ordered branching, can be modified to run in parallel. Some of the key advantages of our framework are:

• -

  The migration from serial to parallel entails very little additional coding. Implementing each of our parallel algorithms took less than two days and around 300 extra lines of code.

• -

  It completely eliminates the need for task-buffers and the overhead they introduce (Section III-B).

• -

  The input of both the serial and parallel implementations are identical. Running the parallel algorithms requires no additional input from the user (assuming every core has access to $r$ and $c$). Most parallel algorithms in the literature require some parameters such as task-buffer size. Selecting the best parameters could vary depending on the instance being solved.

• -

  Experimental results have showed that our implicit load-balancing strategy, joined with the concise task-encoding scheme, can achieve linear (sometimes super-linear) speedup with scalability on at least $32,768$ cores. We hope to test our framework on a larger system in the near future to determine the maximum number of cores it can support.

• -

  Although not typical of parallel algorithms, when using the indexing method and the ConvertIndex function, it becomes reasonably straightforward to support join-leave (i.e. cores leaving the computation after solving a fixed number of tasks) or checkpointing capabilities (i.e. by forcing every core to write its $current\_idx$ to some file).

We believe there is still plenty of room for improving the framework at the risk of losing some of its simplicity. One area we have started examining is the virtual topology. A “smarter” topology could further reduce the communication overhead (e.g. the gap between $T_{S}$ and $T_{R}$) and increase the overall performance. Another candidate is the GetNextParent function which can be modified to probe a fixed number of cores before selecting which to “help” next. Finally, we intend to investigate the possibility of developing the framework into a library, similar to [14] and [13], which will provide users with built-in functions for parallelizing recursive backtracking algorithms.

The authors would like to thank Chris Loken and the SciNet team for providing access to the BGQ production system and for their support throughout the experiments.