Practical Rateless Set Reconciliation

Our main task is to design the algorithm that encodes any set $S$ into an infinite sequence of coded symbols, denoted as $s_{0},s_{1},s_{2},\dots$. It should provide the following properties:

• •

  Decodability. With high probability, the peeling decoder can recover all source symbols in a set $S$ using a prefix of $s_{0},s_{1},s_{2},\dots$ with length $O(|S|)$.

• •

  Linearity. For any sets $A$ and $B$, $a_{0}\oplus b_{0},a_{1}\oplus b_{1},a_{2}\oplus b_{2},\dots$ is the coded symbol sequence for $A\bigtriangleup B$.

• •

  Universality. The encoding algorithm does not need any extra information other than the set being encoded.

These properties allow us to build the following simple protocol for rateless reconciliation. To reconcile $A$ and $B$, Alice incrementally sends $a_{0},a_{1},a_{2},\dots$ to Bob. Bob computes $a_{0}\oplus b_{0},a_{1}\oplus b_{1},a_{2}\oplus b_{2},\dots$, and tries to decode these symbols using the peeling decoder. Bob notifies Alice to stop when he has recovered all source symbols in $A\bigtriangleup B$. As we will soon show, the first symbol $a_{0}\oplus b_{0}$ in Rateless IBLT is decoded only after all source symbols are recovered. This is the indicator for Bob to terminate.

Linearity guarantees that $a_{0}\oplus b_{0},a_{1}\oplus b_{1},a_{2}\oplus b_{2},\dots$ is the coded symbol sequence for $A\bigtriangleup B$. Decodability guarantees that Bob can decode after receiving $O(|A\bigtriangleup B|)$ coded symbols and recover all source symbols in $A\bigtriangleup B$. Universality guarantees that Alice and Bob do not need any prior context to run the protocol.

If Alice regularly reconciles with multiple peers, she may cache coded symbols for $A$ to avoid recomputing them every session. Universality implies that Alice can reuse the same cached symbols across different peers. Linearity implies that if she updates her set $A$, she can incrementally update the cached symbols by treating the updates $A\bigtriangleup A^{\prime}$ as a set and subtracting its coded symbols from the cached ones for $A$.

We now discuss how we design an encoding algorithm that satisfies the three properties we set to achieve.

We now design an efficient deterministic algorithm for mapping a source symbol $s$ to coded symbols that achieves the mapping probability rule identified in the previous section.

Recall that for a random source symbol $s$, we want to make the probability that $s$ is mapped to the $i$-th coded symbol to be $\rho(i)$ in Eq. 1. A simple strawman solution, for example, is to use a hash function that, given $s$, outputs a hash value uniformly distributed in $[0,1)$. We then compare the hash value to $\rho(i)$, and decide to map $s$ to the $i$-th coded symbol if the hash value is smaller. Given a random $s$, because its hash value distributes uniformly, the mapping happens with probability $\rho(i)$.

However, this approach has major issues. First, it requires comparing hash values and $\rho(i)$ for every pair of source symbol $s$ and index $i$. As mentioned in § 4.1.2, the density of the mapping is $O(\log m)$ for the first $m$ coded symbols. In contrast, generating the $m$ coded symbols using this algorithm would require $m$ comparisons for each source symbol, significantly inflating the computation cost. Another issue is that we cannot use the same hash function when mapping $s$ to different indices $i$ and $j$. Otherwise, the mappings to them would not be independent: if $\rho(i)<\rho(j)$ and $s$ is mapped to $i$, then it will always be mapped to $j$. Using different, independent hash functions when mapping the same source symbol to different indices means that we also need to hash the symbol $m$ times.

We design an algorithm that maps each source symbol to the first $m$ coded symbols using only $O(\log m)$ computation. The strawman solution is inefficient because we roll a dice (compare hash and $\rho(i)$) for every index $i$, even though we end up not mapping $s$ to the majority of them ($m-O(\log m)$ out of $m$), so reaching the next mapped index takes many dice rolls ($m/O(\log m)$ on average). Our key idea is to directly sample the distance (number of indices) to skip before reaching the next mapped index. We achieve it with constant cost per sample, so we can jump from one mapped index straight to the next in constant time.

We describe our algorithm recursively. Suppose that, according to our algorithm, a source symbol $s$ has been mapped to the $i$-th coded symbol. We now wish to compute, in constant time, the next index $j$ that $s$ is mapped to. Let $G$ be the random variable such that $j-i=G$ for a random $s$, and let $P_{g}$ ($g\geq 1$) be the probability that $G=g$. In other words, $P_{g}$ is the probability that a random $s$ is not mapped to any of $i+1,i+2,\dots,i+g-1$, but is mapped to $i+g$, which are all independent events. So,

Generating $j$ is then equivalent to sampling $g\leftarrow G$, whose distribution is described by $P_{g}$, and then computing $j=i+g$.

However, since there are $g$ (which can go to infinity) terms in $P_{g}$, it is still unclear how to sample $G$ in constant time. The key observation is that the cumulative mass function of $G$, denoted as $C(x)$, has a remarkably simple form. In particular,

We defer the step-by-step derivation to § B.

Let $C^{-1}(r)$ be the inverse of $C(x)$. The simple form of $C(x)$ allows us to compute $C^{-1}(r)$ easily, which we will soon explain. To sample $G$, we sample $r\leftarrow[0,1)$ uniformly, and compute $g=\lceil C^{-1}(r)\rceil$. To make the algorithm deterministic, $r$ may come from a pseudorandom number generator seeded with the source symbol $s$. The algorithm outputs $i+g$ as the next index to which $s$ is mapped, updates $i\leftarrow i+g$, and is ready to produce another index. Because every source symbol is mapped to the first coded symbol (recall that $\rho(0)=1$), we start the recursion with $i=0$.

Finally, we explain how to compute $C^{-1}(r)$. It is the most simple if we set the parameter $\alpha$ in $\rho(i)$ to $0.5$. Plugging $\alpha=0.5$ into Eq. 2, we get

Its inverse is

For a generic $\alpha$, we can use Stirling’s approximation (Robbins, 1955), and get

Consequently,

In our final design, we set $\alpha=0.5$. The main reason is that computing $C^{-1}(r)$ when $\alpha=0.5$ only requires computing square roots, while it otherwise involves raising $1-r$ to other non-integer powers. We observe that the latter is significantly slower on older CPUs. Meanwhile, as we will show in § 5, setting $\alpha=0.5$ results in negligible extra communication compared to the optimal setting.

In some applications, rogue users may inject items to Alice or Bob’s sets. For example, in a distributed social media application where servers exchange posts, users can craft any post they like. This setting may create an “adversarial workload,” where the hash of the symbol representing the user’s input does not distribute uniformly. If the user injects into Bob’s set a source symbol that hashes to the same value as another source symbol that Alice has, then Bob will never be able to reconcile its set with Alice. This is because Bob will XOR the malicious symbol into the coded symbol stream it receives from Alice, but it will only cancel out the hash of Alice’s colliding symbol from the checksum field, and will corrupt the sum field.

The literature on set reconciliation is aware of this issue, but typically does not specify the required properties from the hash function to mitigate it; most use hash functions with strong properties such as random oracles (Mitzenmacher and Pagh, 2018), which have long outputs (e.g., 256 bits). It is sufficient, however, to use a keyed hash function with uniform and shorter outputs (e.g., 64 bits). This allows Alice and Bob to coordinate a secret key and use it to choose a hash function from the family of keyed hashes. Although with short hashes, an attacker can computationally enumerate enough symbols to find a collision for an item that Alice has, the attacker does not know the key, i.e., the hash function that Alice and Bob use, so she cannot target a collision to one of Alice’s symbols. This allows Rateless IBLT to minimize the size of a coded symbol and save bandwidth, particularly in applications where symbols are short and checksums account for much of the overhead. In practice, we use the SipHash (Aumasson and Bernstein, 2012) keyed hash function. A trade-off we make is that Alice has to compute the checksums separately for each key she uses, which increases her computation load. We believe this is a worthwhile trade-off as SipHash is very efficient, and we find in experiments (§ 7.2) that computing the hashes has negligible cost compared to computing sums, which are still universal. Also, we expect using different keys only in applications where malicious workload is a concern.

Theorem 5.1 and Corollary 5.2 from the density evolution analysis state the behavior of Rateless IBLTs when the difference size $d$ goes to infinity. To understand the behavior when $d$ is finite, we run Monte Carlo simulations, and compare the results with the theorems.

Fig. 4 shows the main results. It compares the overhead predicted by Theorem 5.1 and that observed in simulations. First, notice that as the difference size increases, simulation results converge to the analysis for all $\alpha$. How fast the results converge depends on $\alpha$. For all $\alpha\leq 0.55$, convergence happens quickly, and the overhead observed in simulations stays within $10\%$ of the analysis even for the smallest difference size we test. On the other hand, for $\alpha=0.95$, simulation results are still $12\%$ higher than the analysis at the largest difference size we test. Second, the figure shows that setting $\alpha=0.5$ is close to optimal for the communication overhead. Setting $\alpha=0.5$ results in $\eta^{*}=1.35$, while the optimal setting is $\alpha=0.64$ which results in $\eta^{*}=1.31$, a difference of only $3\%$.

Next, we focus on $\alpha=0.5$, the parameter we choose for our final design. Fig. 5 shows the overhead as we vary the difference size $d$. It peaks at $1.72$ when $d=4$ and then converges to $1.35$ as predicted by Corollary 5.2. Convergence happens quickly: for all $d>128$, the overhead is less than $1.40$.

The density evolution analysis also predicts how decoding progresses as the decoder receives more coded symbols. The fixed points of $q$ in Eq. 3 represent the expected fraction of source symbols that the peeling decoder fails to recover before stalling, as $d$ goes to infinity. Fig. 6 compares this result with simulations (we plot $1-q$, the fraction that the decoder can recover) and they match closely. There is a sharp increase in the fraction of recovered source symbols towards the end, a behavior also seen in other codes that use the peeling decoder, such as LT codes (Luby, 2002).

We first measure the communication overhead, defined as the amount of data transmitted during reconciliation divided by the size of set difference accounted in bytes. We test with set differences of $1$–$400$ items. Beyond $400$, the overhead of all schemes stays stable. The set size is 1 million items (recall that it only affects Merkle trie’s communication cost). Each item is 32 bytes, the size of a SHA256 hash, commonly used as keys in open-permission distributed systems (Nakamoto, 2008; Trautwein et al., 2022). For Rateless IBLT and MET-IBLT, we generate coded symbols until decoding succeeds, repeat each experiment $100$ times, and then report the average overhead and the standard deviation. Regular IBLTs cannot be dynamically expanded, and tuning the number of coded symbols $m$ requires precise knowledge of the size of the set difference. Usually, this is achieved by sending an estimator before reconciliation (Eppstein et al., 2011), which incurs an extra communication cost of at least 15 KB according to the recommended setup (Lázaro and Matuz, 2023, § V-C). We report the overhead of regular IBLT with and without this extra cost. Also, unlike the other schemes, regular IBLTs may fail to decode probabilistically. We gradually increase the number of coded symbols $m$ until the decoding failure rate drops below $1/3\,000$.

Fig. 7 shows the overhead of all schemes except for Merkle trie, whose overhead is significantly higher than the rest at over $40$ across all difference sizes we test. Rateless IBLT consistently achieves lower overhead compared to regular IBLT and MET-IBLT, especially when the set difference is small. For example, the overhead is $2$–$4\times$ lower when the set difference is less than $50$. The improvement is more significant when considering the cost of the estimator for regular IBLTs. On the other hand, PinSketch consistently achieves an overhead of $1$, which is $37$–$60\%$ lower than Rateless IBLT. However, as we will soon show, Rateless IBLT incurs $2$–$2\,000\times$ less computation than PinSketch on both the encoder and the decoder. We believe that the extra communication cost is worthwhile in most applications for the significant reduction in computation cost.

There are two potential computation bottlenecks in set reconciliation: encoding the sets into coded symbols, and decoding the coded symbols to recover the symmetric difference. Encoding happens at Alice, and both encoding and decoding happen at Bob. In this experiment, we measure the encoding and decoding throughput for sets of various sizes and differences. We focus on comparing with PinSketch. We fix the item size to $8$ bytes, because this is the maximum size that the PinSketch implementation supports. We do not compare with regular IBLT or MET-IBLT as we cannot find high-quality open source implementations, and they have similar complexity as Rateless IBLT.⁵⁵5The complexity is linear to the average number of coded symbols each source symbol is mapped to.This is $O(\log(m))$ for Rateless IBLT and MET-IBLT (Lázaro and Matuz, 2023), and constant for regular IBLT, where $m$ is the number of coded symbols. However, the cost is amortized over the size of the set difference, which is $O(m)$. So, in all three IBLT-based schemes, the cost to encode for each set difference decreases quickly as $m$ increases. We will compare with Merkle trie in § 7.3. We run the benchmarks on a server with two Intel Xeon E5-2697 v4 CPUs. Both Rateless IBLT and PinSketch are single-threaded, and we pin the executions to one CPU core using cpuset(1).

As the difference size increases, the encoding throughput of Rateless IBLT increases almost linearly, enabling the encoder to scale to large differences. In Fig. 9, we plot in dashed lines the time it takes to finish encoding. As the difference size increases by $50\,000\times$, the encoding time of Rateless IBLT grows by less than $6\times$. Meanwhile, the encoding time of PinSketch grows by $5\,000\times$.

The set size $N$ affects encoding, but not decoding, because the decoder operates on coded symbols that represent the symmetric difference. The computation cost of encoding grows linearly with $N$, as each source symbol is mapped to the same number of coded symbols on average and thus adds the same amount of work. For example, in Fig. 9, the encoding time for $10^{3}$ differences is $2.9$ milliseconds when $N=10^{4}$, and $294$ milliseconds when $N=10^{6}$, a difference of $100\times$ that matches the change in $N$. Fig. 11 shows the encoding time measured in experiments with the same configuration for a wider range of $N$.

The difference size $d$ affects both encoding and decoding. Recall that Rateless IBLT uses about $1.35d$ coded symbols to reconcile $d$ differences (§ 5). As $d$ increases, the encoder needs to generate more coded symbols. However, unlike PinSketch where the cost is linear in $d$, the cost of Rateless IBLT grows logarithmically. For example, in Fig. 8(a), the encoding time grows by only $6\times$ as the set difference increases from $1$ to $10^{5}$. This is because the mapping from source to coded symbols is sparse: each source symbol is only mapped to an average of $O(\log d)$ coded symbols. The same result applies to decoding. For example, in Fig. 9, the decoding throughput drops by $2\times$ as the $d$ grows by $10^{4}\times$.

The item size $\ell$ affects both encoding and decoding because it decides the time it takes to compute the XOR of two symbols, which dominates the computation cost in Rateless IBLT. Fig. 11 shows the relative slowdown when as $\ell$ grows from $8$ bytes to $32$ KB. Initially, the slowdown is sublinear (e.g., less than $4\times$ when $\ell$ grows by $16\times$ from $8$ to $128$ bytes) because the other costs that are independent of $\ell$ (e.g., generating the mappings) are better amortized. However, after $2$ KB, the slowdown becomes linear. This implies that the data rate at which the encoder can process source symbols, measured in bytes per second, stays constant. For example, when encoding for $d=1000$, the encoder can process source symbols at $124.8$ MB/s. The same analysis applies to decoding. In comparison, the encoding complexity of PinSketch increases linearly with $\ell$, and the decoding complexity increases quadratically (Dodis et al., 2008; Wuille, 2018).

We now apply Rateless IBLT to a prominent application, the Ethereum blockchain. Whenever a blockchain replica comes online, it must synchronize with others to get the latest ledger state before it can validate new transactions or serve user queries. The ledger state is a key-value table, where the keys are 20-byte wallet addresses, and the values are 72-byte account states such as its balance. There are 230 million accounts as of January 4, 2024. Synchronizing the ledger state is equivalent to reconciling the set of all key-value pairs, a natural application of Rateless IBLT.

Ethereum (as well as most other blockchains) currently uses Merkle tries (§ 2) to synchronize ledger states between replicas. It applied a few optimizations: using a 16-ary trie instead of a binary one, and shortening sub-tries that have no branches. The protocol is called state heal and has been deployed in Geth (go-ethereum Authors, 2024a), the implementation which accounts for 84% of all Ethereum replicas (Sonic, 2024). Variants of Geth also power other major blockchains, such as Binance Smart Chain and Optimism.

State heal retains the issues with Merkle tries despite the optimizations. To discover a differing key-value pair (leaf), replicas must visit and compare every internal node on the branch from the root to the differing leaf. This amplifies the communication, computation, and storage I/O costs by as much as the depth of the trie, i.e., $O(\log(N))$ for a set of $N$ key-value pairs. In addition, replicas must descend the branch in lock steps, so the process takes $O(\log(N))$ round trips. As a result, some Ethereum replicas have reported spending weeks on state heal, e.g., (go-ethereum Authors, 2024b). In comparison, Rateless IBLT does not have these issues. Its communication and computation costs depend only on the size of the difference rather than the entire ledger state, and it requires no interactivity between replicas besides streaming coded symbols at line rate.

To obtain workload for experiments, we extract snapshots of Ethereum ledger states as of blocks 18908312–18938312, corresponding to a 100-hour time span between December 31, 2023 and January 4, 2024. Each snapshot represents the ledger state when a block was just produced in the live Ethereum blockchain.⁶⁶6Ethereum produces a block every 12 seconds. Each block is a batch of transactions that update the ledger state. For each experiment, we set up two replicas: Alice always loads the latest snapshot (block 18938312); Bob loads snapshots of different staleness and synchronizes with Alice. This simulates the scenario where Bob goes offline at some point in time (depending on the snapshot he loads), wakes up when block 18938312 was just produced, and synchronizes with Alice to get the latest ledger state. We run both replicas on a server with two Intel Xeon E5-2698 v4 CPUs running FreeBSD 14.0. We use Dummynet (Rizzo, 1997) to inject a 50 ms one-way propagation delay between the replicas and enforce different bandwidth caps of 10 to 100 Mbps.

In our experiments, state heal requires at least 11 rounds of interactivity, as Alice and Bob descend from the roots of their tries to the differing leaves in lock steps. Rateless IBLT, in comparison, only requires half of a round because Alice streams coded symbols without waiting for any feedback. This advantage is the most obvious when reconciling a small difference, where the system would be latency-bound. For example, Rateless IBLT is $8.2\times$ faster than state heal when Bob’s ledger state is only 1 block (12 seconds) stale.

We quickly highlight the impact of interactivity. Fig. 13 shows traces of bandwidth usage when synchronizing one block worth of state difference. For Rateless IBLT, the first coded symbol arrives at Bob in 1 round-trip time (RTT) after his TCP socket opens (0.5 RTT for TCP ACK to reach Alice, and another 0.5 RTT for the first symbol to arrive). Subsequent symbols arrive at line rate, as the peak at 1 RTT indicates. In comparison, for state heal, Alice and Bob reach the bottom of their tries after 11 RTTs; before that, they do not know the actual key-value pairs that differ, and the network link stays almost idle.

Finally, we demonstrate that Rateless IBLT consistently outperforms state heal across different network conditions. We fix Bob’s snapshot to be 10 hours stale and vary the bandwidth cap. Fig. 14 shows the results. Rateless IBLT is $4.8\times$ faster than state heal at 10 Mbps, and the gain increases to $16\times$ at 100 Mbps. Notice that the completion time of state heal stays constant after 20 Mbps; it cannot utilize any extra bandwidth. We observe that state heal becomes compute-bound: Bob cannot process the trie nodes he receives fast enough to saturate the network. The completion time does not change even if we remove the bandwidth cap. In contrast, Rateless IBLT is throughput-bound, as its completion time keeps decreasing with the increasing bandwidth. If we remove the bandwidth cap, Rateless IBLT takes 2.5 seconds to finish and can saturate a 170 Mbps link using one CPU core on each side.

Before ending, we quickly discuss a few other potential solutions and how Rateless IBLT compares with them. When Bob’s state is consistent with some particular block, he may request Alice to compute and send the state delta from his block to the latest block, which would be as efficient as an optimal set reconciliation scheme. However, this is often not the case when Bob needs synchronization, such as when he recovers from database corruption or he has downloaded inconsistent shards of state snapshots from multiple sources, routine when Geth replicas bootstrap (Taverna and Paterson, 2023). Rateless IBLT (and state heal) does not assume consistent states. Coded symbols in traditional set reconciliation schemes like regular IBLT are tailored to a fixed difference size (§ 3). Alice has to generate a new batch of coded symbols for each peer with a different state. This would add minutes to the latency for large sets like Ethereum, incur significant computation costs, and create denial-of-service vulnerabilities.⁷⁷7These issues also apply to the aforementioned state delta solution to a lesser degree, because Alice has to compute the state deltas on the fly. In contrast, Rateless IBLT allows Alice to prepare a single stream of coded symbols that is efficient for all peers. Because of linearity (§ 4), Alice can incrementally update the coded symbols as her ledger state changes. For an average Ethereum block, it takes 11 ms to update 50 million coded symbols (7 GB) using one CPU core to reflect the state changes.