Multi-Grained Specifications for Distributed System Model Checking and Verification

We started by writing TLA⁺ specifications for the Zab protocol and the ZooKeeper system (its realization of the Zab protocol), respectively. Both are now part of the official TLA⁺ specifications of the ZooKeeper projects.

Despite that the system specification effectively describes the ZooKeeper system, it still omits certain implementation details. We present three common patterns of model-code gaps which in our experience is important to consider as they often induce tricky and error-prone implementation. Overlooking them would allow bugs to escape from model checking and reduce the confidence of verification results.

The prevalence of model-code gaps in existing specifications indicates the need to further model important, fine-grained behavior like non-atomic updates and concurrency for verifying code implementation (which is uncommon in existing TLA⁺ specifications). However, doing so would significantly increase state space, resulting in state-space explosion. Currently, using TLC to model check the system specification of ZooKeeper (§2.1.2) cannot finish in ten days with a standard configuration (three nodes, three transactions, up to three node crashes, and up to three network partitions) (maric2017cutoff, ). How to balance the granularity of the specification and the scalability of model checking is a key challenge.

For a given specification, we write its fine-grained counterpart by modeling low-level code behavior. For a target action in the original specification, we rewrite the enabling conditions based on code logic and next-state updates of the action. We focus on three patterns of model-code gaps (§2.2):

• •

  Atomicity. We rewrite an action that is not guaranteed to be atomically executed at the code level. We split the action into multiple separate actions in the specification and set up their enabling conditions accordingly.

• •

  Concurrency. To model concurrency, we separate state transitions, which are executed by different threads, into different actions as per the executing threads. Inter-thread communications (e.g., message queues for local thread messages) are also modeled in the specification.

• •

  Missing state transitions. We focus on enhancing state transitions with existing variables in the target action. If the enabling condition includes other dependent variables that are missed, we add them to the specification.

We decide atomic blocks based on how threads/nodes communicate, following prior work (hawblitzel2015ironfleet, ). An atomic block starts with reading the external state, performing internal computations, and ends with writing to the external state. The results of internal computations are invisible to other threads/nodes, thus can be safely folded in an atomic block. An example atomic block starts with receiving a message from the network, and ends with putting another message in a queue to be handled by another thread.

When model checking a target module with the fine-grained specification, the other modules are coarsened to reduce state space and avoid state-space explosion. To ensure verification safety, the coarsening must follow the interaction preserving principle, i.e., for each module, only the internal part can be omitted while the interactions with other modules must be preserved, such that another module cannot distinguish whether it is interacting with the original or a coarsened module. This indistinguishability ensures the correctness of compositional model checking.

Safe coarsening is done by following the rationale of interaction preservation in TLA⁺. In TLA⁺, we define the global states of a distributed system with variables and update the states with actions. We define dependency variables of an action as the variables in the enabling condition of the action; the dependency relation is transitive—if a dependency variable is calculated from another variable, that variable is also a dependency variable. The dependency variables of a module hence consist of dependency variables of all the actions in the module, where a module is a set of actions. We define interaction variables as dependency variables shared by two modules.¹¹1Our definition of interaction variables is conservative, because dependency variables in two modules may not convey any interaction. In practice, this case is rare so we make the definition concise and easy to use.

The coarsening preserves interaction if: (1) all dependency variables of the target module, as well as all interaction variables, remain unchanged after the coarsening; (2) all the updates of the dependency variables and interaction variables remain unchanged after the coarsening. These two constraints relate actions of a fine-grained module to those of the coarsened module through dependency variables and interaction variables along with their updates.

We denote a specification $S$ that consists of $n$ modules as $S=\bigcup_{1\leq i\leq n}M_{i}$, and $\widetilde{M_{i}}$ as a module obtained by coarsening $M_{i}$ following the above two constraints. $S_{i}$ is denoted as the specification by coarsening every other module except $M_{i}$, i.e., $S_{i}=(\bigcup_{j\neq i}\widetilde{M_{j}})\cup M_{i}$.

Let the traces allowed by $S$ and $S_{i}$ be $T_{S}$ and $T_{S_{i}}$ respectively. When we are only concerned with the states of the target module $M_{i}$, all traces in $T_{S}$ and $T_{S_{i}}$ are projected to $M_{i}$, which are denoted as $T_{S}|_{M_{i}}$ and $T_{S_{i}}|_{M_{i}}$. Then we can talk about the equivalence relation between traces with respect to a target module, which is defined as: $T_{S}\stackrel{{\scriptstyle M_{i}}}{{\sim}}T_{S_{i}}\xlongequal{def}T_{S}|_{M_{i}}=T_{S_{i}}|_{M_{i}}$.

The safety of the coarsening is captured by the equivalence between traces, as in the following theorem.

Appendix B provides the proof sketch of the theorem.

The key concept of the theorem is inspired by (gu2022compositional, ) but is used differently. In (gu2022compositional, ), interaction preservation is used to establish the abstraction-refinement relation between different levels of specifications. In this work, coarse-grained and fine-grained specifications do not have abstraction-refinement relations due to model-code gaps. Therefore, coarsening does not enforce abstraction relations. Both fine-grained and coarse-grained specifications are checked against implementation for conformance. The correctness of modules under verification is guaranteed by ensuring invariants of system design. (The correctness is not guaranteed by abstraction relations between fine-grained and the coarse-grained specifications as in (gu2022compositional, ).) The interaction preservation ensures that all possible behaviors of the target module under verification is systematically explored, without noticing that internal details of its interacting modules are omitted.

In our experience, identifying interaction variables in TLA⁺ specifications is straightforward, especially for systems designed with modularity and loose coupling. We can also potentially borrow ideas from implementation-level model checking (guo2011practical, ) to dynamically identify interaction.

Composing modules and their actions is naturally supported by TLA⁺. The common practice is to define a next-state action that nondeterministically chooses one action from one module to run in each step.

We define four modules corresponding to the four phases of the Zab protocol and compose their specifications into mixed-grained specifications. The four phases are Election, Discovery, Synchronization and Broadcast as shown in Figure 6. Figure 7 shows one example composition, including one (coarsened) action for both Election and Discovery, (fine-grained) actions for Synchronization, and actions for Broadcast. In addition, the next-state action also includes actions for modeling faults (e.g., a node crash). The entire specification is defined as the initial state (Init) and the state transition represented by the next-state action ([]Next). Note that [] is the temporal operator $\Box$ which, in this context, means that the next-state action keeps running forever.

This style of composition captures non-deterministic natures of distributed systems. In each step, Next chooses any possible action to run since it is defined as the disjunction (\/) of all actions. If the action involves certain leader or follower node, Next also chooses any possible node using the existential quantifier (\E). So, the definition of Next allows any (enabled) action with any server to happen at any point.

In this way, we compose different combinations of coarse- and fine-grained modules into different specifications for checking different phases of the Zab implementation.

We ensure that all the TLA⁺ specifications we wrote, despite their granularities, match the implementation through conformance checking. For a specification, the conformance checker explores model-level state space to generate traces, replays them in the implementation, and compares model- and code-level execution traces (§3.5.2).

If the conformance checking detects a discrepancy between the model and the implementation, we debug the discrepancy and revise the specification to match code behavior. The deterministic replay provided by our conformance checker makes it easy to debug at the code level. After the specification is updated, we run a new round of conformance checking until it passes. The continuous conformance checking helped us find several discrepancies, ranging from inconsistent message types between specification and implementation, to incorrect conditional branches and wrong variable assignments that lead to unrealistic state transitions.

We wrote Remix as a framework and tooling support for creating mixed-grained specifications by composing multi-grained specifications of different modules, interfacing the model checker (TLC), and providing conformance checking.

We present the mixed-grained specifications composed using Remix to verify the log replication of ZooKeeper. We focus on log replication because it is the main procedure for achieving consensus in ZooKeeper.²²2We also verify the other part of ZooKeeper and find bugs including ZK-2776, ZK-3336, ZK-3707, ZK-4040, ZK-4416 and ZK-4781. These bugs are known bugs but still exist in the checked versions. It involves both the Synchronization and Broadcast modules. The log replication implementation has been greatly optimized during the evolution of ZooKeeper; hence, the code, especially the Synchronization module, can be described by neither the protocol nor the system specifications effectively. As a result, severe bugs (e.g., those that lead to data loss or inconsistencies) were constantly reported, such as (ZK-2355, ; ZK-2845, ; ZK-3023, ; ZK-3104, ; ZK-3911, ).

Our first step is to ensure the system specifications match the ZooKeeper implementation. We run conformance checking and find a discrepancy where the model-level traces cannot be successfully replayed at the code level due to an unexpected exception. The root cause is that at the code level, if the follower receives the COMMIT message after the NEWLEADER message, it will throw an unexpected NullPointerException and terminate the Synchronization phase. The issue had been reported in ZK-4394 (ZK-4394, ) but was not resolved. So, we adjusted the specification by adding new conditions and a new commit assertion to check the NullPointerException and avoiding further exploration once ZK-4394 occurs.

Based on the conformed system specification, we create the following mixed-grained specifications (Table 1).

• •

  mSpec-1. Since we target log replication, we coarsen the Election and Discovery modules. Section 3.2 described the coarsening which coarsens the eight actions of the Election and Discovery modules in the system specification into one ElectionAndDiscovery action.

• •

  mSpec-2. We write a fine-grained specification of the Synchronization module to model the non-atomic updates of epoch and history when a follower receives the NEWLEADER message (§3.1). So, the model checker can explore intermediate states between the updates of epoch and history, which is induced by node crashes. As we focus on log replication, mSpec-2 uses the coarsened action for the Election and Discovery modules in mSpec-1.

• •

  mSpec-3. This specification further models multithreading concurrency in log replication. There are three main threads in the follower process, for handling incoming messages, logging transactions, and committing transactions, respectively. We distinguish the asynchronous actions executed by different threads, and specify them into separate actions, as demonstrated in §3.1. mSpec-3 also uses the coarsened ElectionAndDiscovery.

• •

  mSpec-4. As a reference, we create mSpec-4 by composing system specifications of the Election and Discovery modules and fine-grained log replication modules in mSpec-3.

We specify 14 invariants that are checked throughout the model checking, as shown in Table 2. Ten invariants are safety properties defined by the Zab protocol (junqueira2010dissecting, ), including both core properties and lemmata; these invariants must be satisfied by any Zab implementations. During model checking, these ten invariants are checked upon state transition. These invariants apply to specifications of any granularity.

We also define other four types of invariants (11 instances in total) based on the code-level implementation of ZooKeeper. We observe that developers add additional checks on specific behavior (e.g., by throwing exceptions and using assertions). Some of them are not reflected by the safety properties of the Zab protocol, and thus shall be included in fine-grained specifications. Some of them are caused by known, but still not resolved bugs in the code, such as ZK-4394 discussed in §4.1. Essentially, each of these invariants specify the execution is on an error path. These invariants are checked whenever the model checker reaches the corresponding execution path. Note that code-level invariants are specific to certain granularities that model the corresponding execution.

Table 3 shows the efforts of writing multi-grained specifications in Table 1. The efforts of writing and maintaining multi-grained specifications are manageable, especially when baseline specifications are available. Fine-grained modeling and coarsening can be done incrementally on top of the reference specification. For example, the differences between the specifications are less than 500 lines. Following composable formal methods, all invariants and most variables in the baseline specification are directly reused. In addition, we need to provide a mapping from the newly added (coarse- or fine-grained) model-level actions to the code-level actions so that Remix can instrument code for deterministic execution (§3.5.3) at different granularities. Overall, the effort is done within 40 person-hours, and is done by one person who is familiar with the ZooKeeper code and is proficient in TLA⁺. Further, as more specifications are written, the reusability of composable components grows higher, amortizing the cost.

We use a configuration of a three-node ZooKeeper cluster with up to four transactions, up to three node crashes, and up to three network partitions. This configuration is a common practice used in prior work (leesatapornwongsa2014samc, ; lukman2019flymc, ; wang2023mocket, ; tang2024sandtable, ). We use TLC to run model checking on TLA⁺ specifications on a single machine. We use TLC’s breath-first search (BFS) as the strategy for state-space exploration. With BFS, once an invariant is violated, we can obtain the buggy trace with minimal depth.

We start to systematically model check ZooKeeper using mixed-grained specifications since version v3.9.1. ZooKeeper did not pass the verification. The model checking exposes a total of six severe bugs, as shown in Table 4. All these bugs have serious consequences, including data loss, data inconsistencies, and data synchronization failures. All these bugs are deep bugs, as their manifestations take minimal depths of tens of actions and more than tens of thousands of states, which are hard for developers to reason about. For the same reason, they are also hard to fix (§5.3).

Those bugs are found when TLC reports violations of invariants in the traces during model checking. We then confirmed the bugs by deterministically replaying the traces at the code level using Remix (§3.5). To debug a violation, we analyze the model-level trace to understand the triggers and locate the root cause in the code. The debugging is eased by traces with minimal depth explored by the BFS strategy. In practice, we start the model checking from a small configuration (e.g., one node crash) to a large one (up to three node crashes), which helps us obtain a simple and concise trace.

Table 4 shows the most efficient specification that found each bug. All the bugs except one (ZK-4394) require mSpec-2 and mSpec-3. In other words, finding these bugs needs fine-grained modeling of non-atomic actions and local concurrency; these bugs cannot be found by the baseline system specification (§2.1.2). The results show the importance of closing the model-code gap with fine-grained specifications that reflect code-level behavior. We discussed ZK-4394 in §4.1. It can be found by mSpec-1 with conformance checking. The bug can be found with the system specification also; but, compared with the system specification, mSpec-1 significantly reduces the time to find the bug. In fact, all the bugs were found in less than two minutes with the benefits of specification coarsening. We discuss more about the efficiency of coarse-grained specifications in §5.2.

Three of the six bugs violate protocol-level invariants and the others violate code-level invariants. We find that code-level invariants are important, as developers often directly throw exceptions to abort the execution in certain critical cases, which is not defined in the protocol (§4.2).

Appendix A describes these bugs in more details.

Mixed-grained specifications effectively improve efficiency of verifying the target modules. We evaluate the verification efficiency of the five specifications in Table 1. We set the time budget to be 24 hours and the violation limit to be 10,000. We use each of these specifications to verify ZooKeeper v3.7.0. We run TLC (v1.7.0) with the BFS mode for exploration. All the experiments were run on an Ubuntu 22.04 server with two AMD EPYC 7642 processors at 3.3GHz; each processor has 48 cores and 96 hyperthreads. Each specification is checked by 16 workers (threads) with 32 GB memory.

Table 5 shows verification efficiency results in two modes: (5(a)) stopping at the first violation, and (5(b)) running to completion (till the limit). The mixed-grained specifications with fine-grained modeling (mSpec-2, -3, and -4) detect violations within the time limit. The baseline (system specification) and mSpec-1 find no violation because of not modeling fine-grained behavior. The baseline and mSpec-4 cannot finish in 24 hours, and we observe that TLC spends most of the time in the Election module without reaching other modules (the leader election algorithm is complex and takes many steps). mSpec-4 costs 2793$\times$ more time to detect the first violation compared to mSpec-3 as it does not coarsen the Election and Discovery modules. In comparison, mSpec-1, -2, and -3 coarsen the Election and Discovery modules, enabling TLC to more efficiently check the log replication modules. As a result, these three mixed-grained specifications can finish in tens of minutes, with mSpec-2 and -3 finding the first violation in minutes. So, mixed-grained specifications provide the flexibility to help the model checker focus on target modules with fine-grained models, which is critical to finding bugs and receiving prompt feedback.

Fixing bugs is challenging—it is easier to prevent specific symptoms, but harder to rule out root causes due to the complexity of reasoning about all interleavings of actions. We first fixed ZK-4712, but found the other bugs are much harder to fix. They are rooted in various performance optimizations since 2017, triggered by ZK-2678 (ZK-2678, ). These optimizations have introduced over ten data loss/inconsistency bugs (ZK-2845, ; ZK-3023, ; ZK-3642, ; ZK-3911, ; ZK-4394, ; ZK-4541, ; ZK-4643, ; ZK-4646, ; ZK-4685, ; ZK-4712, ; ZK-4785, ). Some of them were fixed, while others were not. Without sufficient verification at the time, some accepted fixes introduced new bugs (Figure 8).

Mixed-grained specifications enable us to verify bug fixes efficiently. We verify four Pull Requests (PRs) that attempted to fix the bugs in Table 4. All the PRs use multithreading with non-atomic updates of epoch and history in the Synchronization phase. Therefore, we use the specification of mSpec-3 as the base specification. With the fix of ZK-4712, we updated mSpec-3, referred to as mSpec-3⁺ (verified using TLC). For each PR, we map the code changes to mSpec-3⁺ and update mSpec-3⁺ accordingly. Table 6 shows the verification results with the mode of stopping at the first violation. All four fixes are detected with invariant violations within five minutes.

The challenge of fixing these bugs is that all of them involve the logic of handling the NEWLEADER message. So, it is hard to fix them in isolation—the fix of one bug can make it hard to fix the others. For example, PR-1993 targets ZK-4646 and ZK-4685, without considering other bugs like ZK-4394. To further fix ZK-4394 based on PR-1993 need heavy revision on the code logic of PR-1993. If we consider the full picture, PR-1993 is not a good step towards a complete solution. Some fixes lead to new bugs, e.g., the merged fix for ZK-3911 did not prevent the same violation, but opened new triggering paths of ZK-3023 and induced new bugs like ZK-4685.

The mixed-grained specifications help us understand the root causes and systematically verify whether a given code fix is completed by exhaustively exercising all the possible interleavings modeled by the fine-grained specification. With a holistic understanding, we developed a fix that resolves all the bugs in Figure 8, and verify it with extensive model checking. We discuss our resolution in §5.4. The verified fixes have been merged to the latest version of ZooKeeper.

One essential reason for the error-proneness of log replication is rooted in deviation of the implementation in ZooKeeper from the Zab protocol. For example, the atomicity of updating epoch and history is explicitly required by the protocol, but is not followed by the implementations.³³3The Zab protocol implemented in ZooKeeper (Zab-wiki, ) has evolved in several other ways different from original papers (junqueira2010dissecting, ; junqueira2011zab, ). Basically, the protocol no longer guides the implementation. To fix existing bugs and also make it easy to implement correctly, we remove the atomicity requirement of the two updates from the Zab protocol but require their order—the follower updates its history before updating its epoch.

We update the protocol specification (§2.1.1), which splits the action of handling a NEWLEADER message into two serialized actions of updating history and epoch. In this way, the model checker can explore traces when a follower crashes right after it updates history. We add a variable servingState to help express the enabling conditions of updating history and epoch. We run extensive model checking with TLC to verify the new protocol specification and it passes all the ten protocol-level invariants (Table 2).

We then update the implementation based on the new protocol, and address all the bugs in Figure 8 by enforcing the order of several critical events. We change asynchronous logging into synchronous logging and the overhead is acceptable as it only occurs when the follower receives COMMIT before handling the NEWLEADER message (synchronous logging is already used in v3.9.2). The new implementation conforms to the new protocol specification. We updated mSpecs in Table 1 which passed model checking.