HordeSat: A Massively Parallel Portfolio SAT Solver

Since we can only access the core solvers via the interface defined above, our only tools for diversification are setting phases using the setPhase method and calling the solver specific diversify method.

The setPhase method allows us to partition the search space in a semi-explicit fashion. An explicit search space splitting into disjoint subspaces is usually done by imposing phase restrictions instead of just recommending them. The explicit approach is used in parallel solvers utilizing guiding paths [9] and dynamic work stealing [20].

We have implemented and tested the following diversification procedures based on literal phase recommendations.

• •

  Random. Each variable gets a phase recommendation for each core solver randomly. Note that this is different from selecting a random phase each time a decision is made for a variable in the CDCL procedure.

• •

  Sparse. Each variable gets a random phase recommendation on exactly one of the host solvers in the entire portfolio. For the other solvers no phase recommendation is made for the given variable.

• •

  Sparse Random. For each core solver each variable gets a random phase recommendation with a probability of $(\#solvers)^{-1}$, where $\#solvers$ is the total number of core solvers in the portfolio.

Each of these can be used in conjunction with the diversify method whose behavior is defined by the core solvers. As already mentioned we use Lingeling and MiniSat as core solvers. In case of MiniSat, we implemented the diversify method by only setting the random seed. For Lingeling we copied the diversification algorithm from Plingeling [4], which is the multi-threaded version of Lingeling based on the portfolio approach and the winner of the parallel application track of the 2014 SAT Competition [3]. In this algorithm 16 different parameters of Lingeling are used for diversification.

The clause sharing in our portfolio happens periodically in rounds. Each round a fixed sized (1500 integers in the implementation) message containing the literals of the shared clauses is exchanged by all the MPI processes in an all-to-all fashion. This is implemented by using the MPI_Allgather [12] collective communication routine defined by the MPI standard.

Each process prepares the message by collecting the learned clauses from its core solvers. The clauses are filtered to remove duplicates. The fixed sized message buffer is filled up with the clauses, shorter clauses are preferred. Clauses that did not fit are discarded. If the buffer is not filled up to its full capacity then one of the core solvers of the process is requested to increase its clause production by calling the increaseClauseProduction method.

The detection of duplicate clauses is implemented by using Bloom filters [7]. A Bloom filter is a space-efficient probabilistic set data structure that allows false-positive matches, which in our case means that some clauses might be considered to be duplicates even if they are not. The usage of Bloom filters requires a set of hash functions that map clauses to integers. We use the following hash function which ensures that permuting the literals of a clause does not change its hash value.

where $i>0$ is a parameter we are free to choose, $C$ is a clause, $\oplus$ denotes bitwise exclusive-or, and $\mathit{primes}$ is an array of large prime numbers. Literals are interpreted as integers in the usual way, i.e., $x_{j}$ as $j$ and $\overline{x}_{j}$ as $-j$.

Each MPI process maintains one Bloom filter $g_{x}$ for each of its core solvers $x$ and an additional global one $g$. When a core solver $x$ exports a learned clause $C$, the following steps are taken.

• •

  Clause $C$ is added to $g_{x}$.

• •

  If $C\not\in g$, $C$ is added to $g$ as well as into a data structure $e$ for export.

• •

  If several core solvers concurrently try to access $e$, only one will succeed and the new clauses of the other core solvers are ignored. This way, we avoid contention at the shared resource $e$ and rather ignore some clauses.

After the global exchange of learned clauses, the incoming clauses need to be filtered for duplicates and distributed to the core solvers. The first task is done by using the global Bloom filter $g$. For the second task we utilize the thread local filters $g_{x}$ to ensure that each of them receives only new clauses.

All the Bloom filters are periodically reset, which allows the repeated sharing of clauses after some time. Our initial experiments showed that this approach is more beneficial than maintaining a strict “no duplicate clauses allowed”-policy.

Overall, there are three reasons why a clause offered by a core solver can get discarded. One is that it was duplicate or wrongly considered to be duplicate due to the probabilistic nature of Bloom filters. Second is that another core solver was adding its clause to the data structure for global export at the same time. The last reason is that it did not fit into the fixed size message sent to the other MPI processes. Although important learned clauses might get lost, we believe that this relaxed approach is still beneficial since it allows a simpler and more efficient implementation of clause sharing.

We investigated the individual influence of clause sharing and diversification on the performance of our portfolio. In the case of application benchmarks we obtained the unsurprising result that both diversification and clause sharing are highly beneficial for satisfiable as well as unsatisfiable instances. However, for random 3-SAT problems the results are more interesting.

By looking at the cactus plots in Figure 2 we can observe that clause sharing is essential for unsatisfiable instances while not significant and even slightly detrimental for satisfiable problems. On the other hand, diversification has only a small benefit for unsatisfiable instances. This observation is related to a more general question of intensification vs diversification in parallel SAT solving [13].

For the experiments presented in Figure 2 we used sparse diversification combined with the diversify method, which in this case copies the behavior of Plingeling. It is important to note that some diversification arises due to the non-deterministic nature of Lingeling, even when we do not invoke it explicitly by using the setPhase or diversify methods.

In parallel processing, one usually wants good scalability in the sense that the speedup over the best sequential algorithm goes up near linearly with the number of processors. Measuring scalability in a reliable and meaningful way is difficult for SAT solving since running times are highly nondeterministic. Hence, we need careful experiments on a large benchmark set chosen in an unbiased way. We therefore use the application benchmarks of the 2011 and 2014 Sat Competitions. Our sequential reference is Lingeling which won the most recent (2014) competition. We ran experiments using 1,2,4,…,512 processes with four threads each, each cluster nodes runs 4 processes. The results are summarized in Figure 3 using cactus plots. We can observe that increased parallelism is always beneficial for the 2011 benchmarks. In the case of all the benchmarks the benefits beyond 32 nodes are not apparent.

From a cactus plot it is not easy to see whether the additional performance is a reasonable return on the invested hardware resources. Therefore Table 1 summarizes that information in several ways in order to quantify the overall scalability of HordeSat on the union of the 2011 and 2013 benchmarks. We compute speedups for all the instances solved by the parallel solver. For instances not solved by Lingeling within its time limit $T=50\,000s$ we generously assume that it would solve them if given $T+\epsilon$ seconds and use the runtime of $T$ for speedup calculation. Column 4 gives the average of these values. We observe considerable superlinear speedups on average for all the configurations tried. However, this average is not a very robust measure since it is highly dependent on a few very large speedups that might be just luck. In Column 5 we show the total speedup, which is the sum of sequential runtimes divided by the sum of parallel runtimes and Column 6 contains the median speedup.

Nevertheless, these figures treat HordeSat unfairly since most instances are actually too easy for investing a lot of hardware. Indeed, in parallel computing, it is usual to analyze the performance on many processors using weak scaling where one increases the amount of work involved in the considered instances proportionally to the number of processors. Therefore in columns 7–9 we restrict ourselves to those instances where Lingeling needs at least 10$p$ seconds where $p$ is the number of core solvers used by HordeSat. The average speedup gets considerably larger as well as the total speedup, especially for the large configurations. The median speedup also increases but remains slightly sublinear. Figure 4 shows the distribution of speedups for these instances.

Another way to measure speedup robustly is to compare the times needed to solve a given number of instances. Let $T_{1}$ ($T_{p}$) denote the per instance time limits of the sequential (parallel) solver (50 000s (1 000s) in our case). Let $n_{1}$ ($n_{p}$) denote the number of instances solved by the sequential (parallel) solver within time $T_{1}$ ($T_{p}$). If $n_{1}\geq n_{p}$ ($n_{1}<n_{p}$) let $T_{1}^{\prime}$ ($T_{p}^{\prime}$) denote the smallest time limit for the sequential (parallel) solver such that it solves $n_{p}$ ($n_{1}$) instances within the time limit $T_{1}^{\prime}$ ($T_{p}^{\prime}$). We define the count based speedup (CBS) as

The CBS scales almost linearly up to 512 cores and stagnates afterward. We are not sure whether this indicates a scalability limit of HordeSat or rather reflects a lack of sufficiently difficult instances – in our collection, there are only 65 eligible instances.

The most similar parallel SAT solver to our portfolio is the state-of-the-art solver Plingeling [4]. Plingeling is the winner of the parallel track of the 2014 SAT Competition. Both solvers are portfolio-based, they are using Lingeling and even some diversification code is shared. The main differences are in the clause sharing algorithms and that Plingeling does not run on clusters only single computers. For this reason we can compare the two solvers only on a single node. The results of this comparison on the benchmark problems of the 2011 and 2014 SAT Competition are displayed in Figure 5. Speedup values are given in Table 1.

Both solvers significantly outperform Lingeling. The performance of HordeSat and Plingeling is almost indistinguishable when running with 8 cores, while on 16 cores HordeSat gets slightly ahead of Plingeling.

An important next step is to work on the scalability of HordeSat for 1024 cores and beyond. This will certainly involve more adaptive clause exchange strategies. Even for single node configurations, low level performance improvements when using modern machines with dozens of cores seem possible. We also would like to investigate what benefits can be gained by having a tighter integration of core solvers by extending the interface. Including other kinds of (not necessarily CDCL – based) core solvers might also bring improvements.

When considering massively parallel SAT solving we probably have to move to even more difficult instances to make that meaningful. When this also means larger instances, memory consumption may be an issue when running many instances of a SAT solver on a many-core machine. Here it might be interesting to explore opportunities for sharing data structures for multiple SAT solvers or to decompose problems into smaller subproblems by recognizing their structure.