Solving QSAT problems with neural MCTS

The PUCT (Predictor + UCT) algorithm implemented in AlphaZero (?; ?) is essentially a neural MCTS algorithm which uses PUCB Predictor + UCB (?) as its confidence upper bound (?) and uses the neural prediction $P_{\phi}(a|s)$ as the predictor. The algorithm is running through multiple searching iterations to decide the optimal action for the current state. During each iteration, there are 4 phases:

1. 1.

   SELECT: At the beginning of each iteration, the algorithm selects a path from the root (current game state) to a leaf (either a terminal state or an unvisited state) in the tree according to the PUCB (see (?) for a detailed explanation for terms used in the formula). Specifically, suppose the root is $s_{0}$, we have ¹¹1Theoretically, the exploratory term should be $\sqrt{\frac{\sum_{a^{\prime}}N(s_{i-1},a^{\prime})}{N(s_{i-1},a)+1}}$, however, the AlphaZero used the variant $\frac{\sqrt{\sum_{a^{\prime}}N(s_{i-1},a^{\prime})}}{N(s_{i-1},a)+1}$ without any explanation. We tried both in our implementation, and it turns out that the AlphaZero one performs much better.:

   $$a_{i}=\operatorname*{arg\,max}_{a}\left[Q(s_{i},a)+cP_{\phi}(a|s_{i})\frac{\sqrt{\sum_{a^{\prime}}N(s_{i},a^{\prime})}}{N(s_{i},a)+1}\right]$$

   $$Q(s_{i},a)=\frac{W(s_{i},a)}{N(s_{i},a)+1}$$

   $$s_{i+1}=move(s_{i},a_{i})$$

2. 2.

   EXPAND: Once the select phase ends at a non-terminal leaf, the leaf will be fully expanded and marked as an internal node of the current tree. All its children nodes will be considered as leaf nodes during the next iteration of selection.

3. 3.

   ROLL-OUT: Normally, starting from the expanded leaf node chosen from previous phases, the MCTS algorithm uses a random policy to roll out the rest of the game (?). The algorithm simulates the actions of each player randomly until it arrives at a terminal state, which means the game has ended. The algorithm then uses the outcome of the game as the result evaluation for the expanded leaf node.

   However, a random roll-out usually becomes time-consuming when the tree is deep. A neural MCTS algorithm, instead, uses a neural network $V_{\phi}$ to predict the result evaluation so that the algorithm saves the time on rolling out.

4. 4.

   BACKUP: This is the last phase of an iteration where the algorithm recursively backs-up the result evaluation in the tree edges. Specifically, suppose the path found in the Select phase is $\{(s_{0},a_{0}),(s_{1},a_{1}),...(s_{l-1},a_{l-1}),(s_{l},\_)\}$. then for each edge $(s_{i},a_{i})$ in the path, we update the statistics as:

   $$W^{new}(s_{i},a_{i})=W^{old}(s_{i},a_{i})+V_{\phi}(s_{l})$$

   $$N^{new}(s_{i},a_{i})=N^{old}(s_{i},a_{i})+1$$

   However, in practice, considering a Laplace smoothing in the expression of Q, the following updates are actually applied:

   $$Q^{new}(s_{i},a_{i})=\frac{Q^{old}(s_{i},a_{i})\times N^{old}(s_{i},a_{i})+V_{\phi}(s_{l})}{N^{old}(s_{i},a_{i})+1}$$

   $$N^{new}(s_{i},a_{i})=N^{old}(s_{i},a_{i})+1$$

   Once the given number of iterations has been reached, the algorithm returns a vector of action probabilities of the current state (root $s_{0}$). And each action probability is computed as $\pi(a|s_{0})=\frac{N(s_{0},a)}{\sum_{a^{\prime}}N(s_{0},a^{\prime})}$. The real action played by the neural MCTS is then sampled from the action probability vector $\pi$. In this way, neural MCTS simulates the action for each player alternately until the game ends. This process is called neural MCTS simulation, which is the core of self-play.

A quantified Boolean formula (QBF) is a formula in the following form:

Where $x_{i}$ are distinct boolean variables. The sequence of quantifiers and variables is called the prefix of a QBF. The propositional formula $\Phi$ is called the matrix of a QBF, which only uses the variables in $\{x_{i}\}$. A QBF can evaluate to either true or false since there are no free variables, and it is solvable only if it evaluates to true, otherwise, it is unsolvable. The problem of determining the truthfulness of a QBF is called QSAT problem, which is known to be PSPACE complete.

A QSAT problem can be seen as a game between two players: the existential player (the Proponent (P)) who assigns values to the existentially quantified variables, and the universal player (the Opponent (OP)) who assigns values to the universally quantified variables. The two players make moves by assigning values to the variables alternately following the sequence of quantifiers in the prefix. The existential player (P) wins if the formula evaluates to True and the universal player (OP) wins if it evaluates to False.

In this work, QBFs are encoded as graphs, and a Gated Graph Neural Network (GGNN) (?; ?) is applied to embed the QBFs into the neural MCTS framework. Notice that the GGNN is not the only option and there are alternatives (?; ?), we choose GGNN for the sake of its easy implementation.

The forward pass of the GGNN can be described as following:

where $e$ is the edge type in a multigraph, $A_{e}$ is the edge-weight matrix to be learned during the training, $h_{v}^{t}$ is the hidden representation of node $v$ at message passing iteration $t$, and $m_{v}^{t}$ is called the message from node $v$ at iteration $t$. $R$ is called the read-out which aggregates information from each node to generate a global feature target (notice that $\sigma$ means the sigmoid activation function, $f$ and $g$ are MLPs, and $\odot$ means element-wise product).

The message passing process iterates for a given $T$ times, during each iteration, each node $v$ computes its message using the hidden representation from the neighbor nodes $N(v)$. After that, a Gated Recurrent Unit (GRU) is used to update the hidden representation of the node $v$. The message passing process allows each node’s hidden representation to capture the global structure information of the entire input graph. Finally, the read-out process $R$ is applied to all the nodes to compute the global target of the input graph. GGNN is invariant to graph isomorphism, which is well-suited to capture the symmetry properties among the QBFs.

Although the QSAT problem has a simple syntactic structure, symmetries induced by the semantics of propositional logic should not be ignored (?). The fact that symmetric QBFs are equivalent can improve learning efficiency. In this work, we specially designed a graph encoding of the QBFs, which helps us catch those symmetries through graph isomorphism.

After using Tseyting transformation to re-write $\Phi$ in conjunctive normal form (CNF), a QBF is represented as an undirected multigraph (Fig. 1) with two nodes for every literal and its negation, and one node for every clause. There are four types of edges in this multigraph: 1. E2A edge, an edge between every consecutive existential literal and universal literal; 2. A2E edge, an edge between every consecutive universal literal and existential literal; 3. L2C edge, an edge between every literal and every clause it appears in; 4. reflexive edge, and an edge between each pair of literal and its negation.

The reason behind such a design are three aspects: 1. the sequential information of the prefix is essential to identify the solution of a QBF. Even if two QBFs have the same matrix $\Phi$, a different variable sequence in the prefix might lead to a massive difference in the solution. Therefore, we use the E2A edges and A2E edges to track such sequential information. 2. In a QBF, variables only show as positive literals in the prefix; however, they can be both positive and negative in the matrix $\Phi$. Hence we naturally represent any variable as two nodes, meaning a pair of two complementary literals. 3. Since any literal and its complement are coupled, we use a reflexive edge to capture such entanglement.

In our design, the policy-evaluation neural network of the neural MCTS becomes two GGNNs (see section 3.3), one for each player. The reason why we use two independent neural networks instead of one is that the QSAT game is asymmetric in terms of the winning condition. As we have introduced in the section 3.2, P wins the game if and only if the QBF evaluates to true, while OP wins the game if and only if the QBF evaluates to false. On the other hand, when it comes to the outcome of GGNN for two consecutive moves by different players, we noticed that the hidden representation sometimes has no significant difference between the two players. Hence the GGNN becomes confused on the input graphs. This issue can be resolved only by separating the neural networks, so that both of the players can learn and progress mutually and consistently.

Another fact to notice is that we treat every single QSAT problem as an independent game. During the self-play phase, the neural MCTS algorithm (section 3.1) simulates the move for each player based on the player’s GGNN. The neural MCTS takes in the current game state (the QBF graph) and uses the current player’s GGNN to do the selection and rollout. After a certain number (25 in our case) of iterations, neural MCTS will return the action probability distribution for the current state. The player will sample her next move from this distribution. The simulation alternates between the two players until the game ends, where the game outcome will be evaluated and stored for the training phase. To call the neural network, the hidden representation $h_{v}^{0}$ of each node $v$ is initialized with the type of the node. Specifically, for an existential literal node, the hidden representation is $[1,0,0,...,0]$; for a universal literal node, the hidden representation is $[0,1,0,...,0]$; and for a CNF clause node, the hidden representation is $[0,0,1,...,0]$ . Notice that we use $0$’s to pad the vector to a given length. Another fact to notice is that there are two read-out task ($P_{\phi}$ and $V_{\phi}$). Hence we use two different sets of aggregation MLPs for each of the task:

After each self-play simulation, we store the game trace of each player separately as a set of tuples in the form of ($s$, $\pi$, $v$), where $s$ is the game state (the QBF graph), $\pi$ is the action probability distribution generated by neural MCTS based on current state, and $v$ is the game result in the perspective of current player based on the game outcome. We run such a simulation several times (in our case, ten times) to retrieve enough training data. After that, we train the GGNN independently for each of the players using those training data collected during self-play. After training, we use the newly trained GGNNs to play against each other for 20 rounds and collect the performance data for evaluation and analysis, and this is called the arena phase.

The hyperparameters are set as follows: the number of searching iteration for neural MCTS is set to 25, and the number of simulation is set to 100; the message passing time $T$ is set to 10 for the GGNN; the size of the hidden representation of the GGNN is set to 128.

Considering the capacity and computation power of our machine, we generate 20 random QBFs (10 solvable and 10 unsolvable) which have 51 nodes (the prefix has 21 quantified variables, and the matrix has 9 clauses. So there are 42 literal nodes and 9 clause nodes.) after encoding as graphs. Each QBF is regarded as a single game to be played and learned by the neural MCTS. We run the learning iteration (i.e., self-play, training, and arena) for 32 epochs, and collect the performance data in the arena phase during each iteration.

To measure the performance of the algorithm, we use two metrics: the local correctness ratio and the global correctness ratio. We compute the local correctness ratio of the two players during the arena phase where the two players compete with each other for 20 rounds. The action is locally correct if it preserves a winning position. It is straightforward to check the local correctness of actions using a QSAT solver: GhostQ (?). We collect the local correctness ratio of the two players after each round of competing in arena phase. Then we take the average value of their local correctness ratio as the performance measurement for the current training iteration.

Since the two neural networks might be inductively biased to each other, the locally correct solution could be incorrect. To see whether the neural MCTS learns the correct solution, we measure the global correctness ratio by test the algorithm with an enumerator. To be specific, if a QBF is satisfactory, then we enumerate all possible move for the OP (the universal player) and use the enumerator to play against the P’s neural network. Vice-versa for the unsatisfactory QBF. Theoretically, OP’s neural network fails to solve the QBF if there is any chance that the P’s enumerator can win the game. We count the number of winning games for each player and use it to measure the global correctness ratio. A 100% global correctness not only means the neural MCTS has found the correct solution, but also a fully support (represented as a winning strategy encoded in the neural network) to that solution. On the other hand, a non-100% global correctness can be treated as a measure of approximation of the algorithm.

Our experiment shows that the algorithm can correctly determine the truthfulness of all 20 test QBFs. We notice that, for a solvable QBF, the existential player can quickly dominate the game and win at most of the times, and vice-versa for the universal player in an unsolvable case. The result indicates that for a solvable/ an unsolvable QSAT problem, the existential/ universal player has a higher chance to win the corresponding QSAT game against the universal/ existential player.

We also measured the algorithm’s global correctness ratio for all test cases, and we noticed an asymmetry between the solvable and unsolvable cases. To be specific, we computed the average global correctness ratio (AGC) for all solvable and unsolvable QBFs respectively, and it turns out that the AGC for solvable cases is 87%, while the AGC for unsolvable cases is 85%. This fact indicates that neural MCTS can still be an adequate approximator to QSAT problem, even if it cannot derive a 100% correct strategy.

In this section, for the illustration purpose, we show the experiment results for a solvable QSAT and an unsolvable QSAT (described in Fig. 2 and Fig. 3 where, due to limited space, we only show the matrix of the QBF). One can see, in Fig. 2, the local correctness ratio of the existential player (P) soars high after the first epoch; while in Fig. 3, the local correctness ratio of the universal player (OP) increases rapidly. Even though there are fluctuations, one of the player always dominate the game, this phenomenon is treated as an indicator to the truthfulness of the QSAT. Also, notice that the curves in the unsolvable case wave more violently than the ones in the solvable case. This fact means that even though the player can dominate the game, dominating an unsolvable QSAT game might be harder than a solvable one. In terms of global correctness ratio, both of them got 100% correctness, that means the neural MCTS not only makes the correct decision but also constructively support its decision.

One of the known issues of self-play is that the two players will always mutually bias their strategy to fit with the other’s one through exploiting their experiences. This mutual inductive bias facilitates the learning process of the players when they are at the same pace. However, once the learning speeds are unbalanced, the mutual inductive bias foils the improvement of players’ performance by stagnating their strategies in a local optimum. To understand this issue, one can think about a game between an expert and a newbie. Since the expert can easily find a strategy to win against the newbie, the newbie will always lose the game. And because there is no positive feedback at all, the newbie will build a biased belief that there is no way to win the game. Such a belief can be strengthened during self-play, and finally, it leads to some fixed losing strategy. While on the other side, since the opponent is not so challenging, the expert will also stay with the current strategy without any attempt to improve it.

Nevertheless, we notice that neural MCTS is resilient to mutual inductive bias. Whenever the learning paces are unbalanced, the weaker players decisions become indifferent (i.e., no matter what moves she takes, she will always lose the game). On the other hand, neural MCTS pushes those indifferent actions into a uniform distribution, hence to encourage exploration by making random moves. Consequently, neural MCTS adaptively balance the exploration and exploitation, thus jumping out of the local optimal.

Neural MCTS is capable of handling a large state space (?). Such an algorithm must search only a small portion of the state space and make the decisions from those limited observations. To measure the state space coverage, we recorded the number of states accessed during the experiment, In each QSAT game, we count the total number of states accessed during each game in self-play, and we compute the 10-game moving average of state accessed for all self-plays (Fig. 4). This result indicates an implicit adaptive pruning mechanism behind the neural MCTS algorithm, which can be regarded as a justification for its capability of handling large state spaces.

Our test cases are restricted to a limited size. Because QSAT is known to be PSPACE complete, verifying the correctness of the algorithm is time-consuming. In our experiment, there are 10 to 11 moves for each player. Hence to verify the correctness of the algorithm, it roughly takes $2^{10}$ to $2^{11}$ tests. And the verification time increases exponentially with the number of variables in the QBF.

On the other hand, the strategy learned by the neural MCTS algorithm is implicitly encoded inside the neural network, and there is no way to extract such a strategy so that they can be explicitly verified by any more efficient approaches from the formal method. Therefore, using an enumerator to verify the correctness is inevitable for the time being. As a result, even though neural MCTS can handle a deep game tree hence a large number of variables, it is still hard or even impossible to verify the learning outcome.