Height-bounded Lempel-Ziv encodings

Dictionary compressors are a family of algorithms that produce compressed representations of input strings essentially as sequences of copy and paste operations. These representations are widely used in general compression tools such as gzip and LZ4, and have also received much recent attention since they are especially effective for highly repetitive data sets, such as versioned documents and pangenomes [23].

A desirable operation to support on compressed data is that of random access to arbitrary positions in the original data. Access should be supported without decompressing the data in its entirety, and ideally by decompressing little else than the sought positions. Recent years have seen intense research on the access problem in the context of dictionary compression, most notably for grammar compressors (SLPs) and Lempel-Ziv (LZ)-like schemes. The general approach is to impose structure on the output of the compressor and in doing so add small amounts of information to support fast queries. While LZ77 [34] is known to be theoretically and practically one of the smallest compressed representations that can be computed efficiently, a long-standing question is whether $O(\polylog(n))$ time access can be achieved with a data structure using $O(z)$ space [23], where $z$ is the size of the LZ77 parsing.

In an LZ-like compression scheme, the input string $T$ of length $n$ is parsed into $z^{\prime}\leq n$ phrases (substrings of $T$), where each phrase is of length $1$, or, is a phrase of length at least $2$ which has a previous occurrence. Each phrase can be encoded by a pair $(\ell,s)$, where $\ell\geq 1$ is the length of the phrase, and $s$ is the symbol representing the phrase if $\ell=1$, or otherwise, a position of a previous occurrence (source) of the phrase. A greedy left-to-right algorithm that takes the longest prefix of the remaining string that can be a phrase leads to the aforementioned LZ77 parsing. An LZ-like encoding of a string induces a referencing forest, where each position of the string is a node, phrases of length 1 are roots of a tree in the forest, and the parent of all other positions are induced by the previous occurrence of the phrase it is contained in. The principle hurdle to support fast access to an arbitrary symbol $T[i]$ from an LZ-like parsing is to trace the symbol to a root in the referencing forest. For LZ77, the height of the referencing forest can be $\Theta(n)$.

We first consider modifying the definition of the height of the referencing forest under some conditions, by implicitly rerouting edges. Consider the case of a unary string $T=\mathtt{a}^{n}$ and LZ-like encoding $(1,\mathtt{a}),(n-1,1)$. The straightforward definition above gives a height of $n-1$ for position $n$. This is a result of the second phrase being self-referencing, i.e., the phrase overlaps with its referenced occurrence, and each position references its preceding position. An important observation is that self-referencing phrases are periodic; in particular a self-referencing phrase starting at position $b_{i}$ with source $s_{i}$ has period $b_{i}-s_{i}$. Due to this periodicity, any position in the phrase can refer to an appropriate position in $T[s_{i}..b_{i})$.

More formally, let $\mathcal{E}=(\ell_{1},s_{1}),\ldots,(\ell_{z^{\prime}},s_{z^{\prime}})$ be an LZ-like encoding of $T$. For any position $i$, let $b_{j}$ be the starting position of the phrase $(\ell_{j},s_{j})$ such that $b_{j}\leq i<b_{j}+\ell_{j}$. Then, $T[i]=s_{j}$ if $\ell_{j}=0$, and $T[i]=T[s_{j}+(i-b_{j})]=T[s_{j}+((i-b_{j})\bmod(b_{j}-s_{j}))]$ otherwise.

We define the height $0pt_{\mathcal{E}}(i)$ of position $i$ by an encoding $\mathcal{E}$ as

The subscript will be omitted if the underlying encoding considered is clear. Since $j$ can be computed using a predecessor query on the set of phrase starting positions, $T[i]$ can be computed using $O(0pt(i)Q(z))$ time using a data structure of size $O(z)$, where $Q(z)$ is the time required for predecessor queries on $z$ elements in $[1,n]$ using $O(z)$ space. $Q(z)$ is $O(\log n)$ using a simple binary search, and faster using more sophisticated methods [21].

For example, for the encoding $(1,\mathtt{a}),(1,\mathtt{a}),(1,\mathtt{b}),(3,2),(1,\mathtt{c}),(4,3)$ of string $\mathtt{aababacbaba}$, the heights are: $0,0,0,1,1,1,0,1,2,2,2$. Notice that the phrase $(3,2)$ that starts at position $4$, representing $\mathtt{aba}$ is self-referencing, and the height of the last position in the phrase (position 6) is $1$, since it is defined to reference position $2=2+(6-4)\bmod(6-4)$. We note that even with this modified definition, the height of the optimal LZ-like encoding (LZ77) can be $\Theta(n)$. See Appendix A in Appendix A.

In order to bound the worstcase query time complexity for access operations, we consider LZ-like encodings with bounded height. An $h$-bounded LZ-like encoding is an LZ-like encoding where $\max\{0pt(i)\mid i\in[1,n]\}\leq h$.

There are many ways one could enforce such a height restriction. Unfortunately, finding the smallest such encoding was very recently shown to be NP-hard by Cicalese and Ugazio [8]. Therefore, we propose several greedy heuristics to compute $h$-bounded LZ-like encodings. Given an encoding for $T[1..b_{j})$ (which defines $0pt(i)$ for any $i\in[1,b_{j})$), the next phrase $(\ell_{j},s_{j})$ starting at position $b_{j}$ is defined as follows.

LZHB1

$\ell_{j}$ is the largest value such that $T[s_{j}..s_{j}+\ell_{j})$ satisfies the height constraint, where $s_{j}=\mathrm{lmocc}_{T}(b_{j},\mathrm{lpf}_{T}(b_{j}))$, i.e., the leftmost occurrence of the longest previously occurring factor at position $b_{j}$.

LZHB2

$\ell_{j}$ is the largest value such that for all $1\leq\ell^{\prime}\leq\ell_{j}$, the leftmost occurrence of $T[b_{j}..b_{j}+\ell^{\prime})$ satisfies the height constraint. $s_{j}$ is the leftmost occurrence of $T[b_{j}..b_{j}+\ell_{j})$.

LZHB3

$\ell_{j}$ is the largest value such that for all $1\leq\ell^{\prime}\leq\ell_{j}$, there exists some previous occurrence of $T[b_{j}..b_{j}+\ell^{\prime})$ that satisfies the height constraint. $s_{j}$ is the leftmost occurrence of $T[b_{j}..b_{j}+\ell_{j})$ that satisfies the height constraint.

Note that for all variations, $\ell_{j}=1$ if there is no occurrence of $T[b_{j}]$ that satisfies the corresponding conditions described above, in which case, $s_{j}=T[b_{j}]$.

We note that LZHB1 corresponds to the baseline algorithm BATLZ2 in [18]. LZHB3 essentially corresponds to greedy-BATLZ in [18] as well, but greedy-BATLZ does not require that the occurrence is leftmost. The difference between LZHB2 and LZHB3 lies in the priority of the choice of the leftmost occurrence and when to check the height constraint. LZHB2 greedily extends the prefix, checking each time whether its leftmost occurrence satisfies the height constraint. LZHB3 finds the leftmost occurrence out of the longest prefix that can satisfy the height constraint. Note that when the height is unbounded, the size of all three variants will be equivalent to the regular LZ77 parsing.

We show that LZHB1 and LZHB2 can be computed in $O(n)$ time and space for linearly-sortable alphabets, and LZHB3 can be computed online in $O(n\log\sigma)$ time and $O(n)$ space for general ordered alphabets.

We also propose a new encoding of LZ-like parsings that re-routes edges of the implicit referencing forest in an attempt to further reduce the heights, again using periodicity. In the case of self-referencing phrases, our definition of height in Equation (1) utilized the fact that self-referencing phrases implies a period $b_{j}-s_{j}$ in the phrase. Using this period, we could re-route the parents of all positions inside the phrase to the first period in the previous occurrence of the phrase, which is outside the phrase. Here, we make two further observations: 1) since the referenced substring is the first period of the phrase, we could extended the phrase for free while the period continues, possibly making the phrase longer, and 2) the referenced substring could have a previous occurrence further to the left, which should tend to have shorter heights. Therefore, if we were to store the period of the phrase explicitly, this could be applied to and benefit all periodic phrases, self-referencing or not.

Specifically then, under this scheme, a phrase with period $1$ can be encoded as the triple $(\ell,c,1)$, where $c$ is a symbol, and a phrase with period $p\geq 2$ can be encoded as the triple $(\ell,s,p)$, where $\ell\geq p$ is the length of the phrase with period $p\geq 2$, and $s$ is a previous occurrence of the length-$p$ prefix of the phrase. We will call such an encoding, a modified LZ-like encoding. For example, $(2,\mathtt{a},1),(1,\mathtt{b},1),(3,2,2),(1,\mathtt{c},1),(4,3,2)$ would be a modified LZ-like encoding for the string $\mathtt{aababacbaba}$.

The implicit referencing forest and heights of the modified LZ-like encoding can be defined analogously: a position $i$ in the $j$th phrase $(\ell_{j},s_{j},p_{j})$ that begins at position $b_{j}$, i.e., $b_{j}\leq i<b_{j}+\ell_{j}$, will reference position $s_{j}+((i-b_{j})\bmod p_{j})\bmod(b_{j}-s_{j})$, where the second $\bmod$ is to deal with the case where the occurrence of the prefix period is self-referencing. For the above example, the heights are: $0,0,0,1,1,1,0,1,2,1,2$.

A greedy left-to-right algorithm computes the $j$th phrase $(\ell_{j},s_{j},p_{j})$ starting at $b_{j}$ as:

LZHB4

$\ell_{j}$ is the largest value such that $T[b_{j}..b_{j}+\ell_{j})$ has period $1$, or, for all $1\leq\ell^{\prime}\leq\ell_{j}$, there exists some previous occurrence of $T[b_{j}..b_{j}+p^{\prime})$ that satisfies the height constraint, where $p^{\prime}$ is the minimum period of $T[b_{j}..b_{j}+\ell^{\prime})$. $s_{j}$ is the leftmost occurrence of $T[b_{j}..b_{j}+\ell_{j})$ that satisfies the height constraint, and $p_{j}$ is the minimum period of $T[b_{j}..b_{j}+\ell_{j})$.

Here, we consider the strength of height bounded encodings as repetitiveness measures [23]. Denote by $\hat{z}_{\mathit{HB}(h)}$, the optimal (smallest) LZHB encoding whose height is at most $h$. It easily follows that $\hat{z}_{\mathit{HB}(h)}$ is monotonically non-increasing in $h$, i.e., for any $h\leq h^{\prime}$, it holds that $\hat{z}_{\mathit{HB}(h)}\geq\hat{z}_{\mathit{HB}(h^{\prime})}$. We will also denote by $\hat{\tilde{z}}_{\mathit{HB}(h)}$, the optimal modified LZ-like encoding whose height is at most $h$.

A first, and obvious, observation is that if the height is allowed to grow to $n$, then the size of the height-bounded encoding and the LZ parsing are equivalent. {observation} $\hat{z}_{\mathit{HB}(n)}=z$.

It is also interesting to note height-bounded and run-length encodings can be related: {observation} $\hat{\tilde{z}}_{\mathit{HB}(0)}$ is equivalent to the run-length encoding.

The following relation between $\hat{z}_{\mathit{HB}(h)}$ and the smallest size $\hat{g}_{\mathrm{rl}}$ of RLSLPs (SLPs with run-length rules) and $\hat{g}_{it(d)}$ of ISLPs [26] can be shown.

When there is no height constraint, LZHB3 is equivalent to LZ77 and thus gives an optimal LZ-like parsing. For modified LZ-like encodings with no height contraints, LZHB4 can be worse than optimal. For example, for the string $\mathtt{abaxabcdababca}$, greedy gives $\mathtt{a}|\mathtt{b}|\mathtt{a}|\mathtt{x}|\mathtt{ab}|\mathtt{c}|\mathtt{d}|\mathtt{abab}|\mathtt{c}|\mathtt{a}$ while $\mathtt{a}|\mathtt{b}|\mathtt{a}|\mathtt{x}|\mathtt{ab}|\mathtt{c}|\mathtt{d}|\mathtt{ab}|\mathtt{abca}$ is a slightly smaller parsing. However, we can show that the greedy algorithm has an approximation ratio of at most $2$.

We have developed prototype implementations of our algorithms described above, in C++. Since Lipták et al. [18] have already demonstrated the effectiveness and superiority of greedy-BATLZ and greedier-BATLZ (which correspond to LZHB3 and the greedier LZHB3) compared to the baselines and that the height can be reduced without increasing the size of the parse too much, our experiments here focus on the performance of our algorithms LZHB3 and LZHB4 in comparison with greedy-BATLZ and greedier-BATLZ. For the BAT-LZ variants, we use the code provided by Lipt’ak et al.³³3Available at: https://github.com/fmasillo/BAT-LZ. with slight modifications to fit our test environment. Our implementations for LZHB3 and LZHB4 are available at https://github.com/dscalgo/lzhb/.

Experiments were conducted on a system with dual AMD EPYC 9654 2.4GHz processors and 768GB RAM running RedHat Linux 8. We use the Real subset of the repetitive corpus⁴⁴4https://pizzachili.dcc.uchile.cl/repcorpus.html.

Figure 2 shows the running times for various height constraints of the BAT-LZ variants as well as LZHB3, LZHB3SA, LZHB4, and LZHB4SA, and their greedier versions, where the suffix SA denotes the suffix array implementation. Although greedy-BATLZ is fast when there is no height constraint, its performance seems to diminish rapidly when a height constraint is set. Interestingly, greedier-BATLZ seemed to run faster than greedy-BATLZ. We observe that all versions of our LZHB3 and LZHB4 implementation outperform greedy-BATLZ and greedier-BATLZ by a large margin. For the greedier variants, the running times increase as the height constraint decreases. This is because stricter height constraints generally lead to shorter phrase lengths and therefore increases $z^{\prime}$ and $\mathit{occ}$. This becomes more apparent in the suffix array versions, perhaps because the $n$ in the $O(\log n)$ factor for finding the suffix array range is actually the size of the range considered, which becomes larger for shorter pharses.

Figure 3 shows the memory usage of the above mentioned implementations. The memory usage for LZHB3 and LZHB4 (the suffix tree versions) are smallest for small heights, because smaller height constraints imply that many paths in the suffix tree become truncated. On the other hand, the memory usage is the highest for larger heights. Memory usage for the other versions is unaffected by the height constraint. LZHB3SA and LZHB4SA are always superior to greedy-BATLZ, and the same can be said for the greedier versions and their counterparts.

In summary, our suffix array implementations are always faster and use less memory than their BATLZ counterparts, with the exception of greedy-BATLZ with no height constraints.

Figure 4 shows the sizes of the phrases of LZHB3, LZHB4 and their greedier versions⁵⁵5We also ran experiments for LZ End parsing, but do not include results in the figures because they distort the plots. We include statistics for LZ End in Table 1 in the Appendix.. Although a phrase of the modified LZ-like encoding is slightly larger than the traditional LZ-like encoding since it includes the period, we can see that a much smaller encoding with the same height, which can more than compensate for this increase, can be obtained in some cases. This seemed to be prominent for cere, para, and Escherichia_Coli, which are all DNA.

We have introduced new variants of LZ-like encodings that directly focus on supporting random access to the underlying data. We also proposed a modified LZ-like encoding, which naturally connects the run-length encoding ($h=0$), and LZ77 ($h=n$). We described linear time algorithms for several greedy variants of such encodings, as well as how to adapt them to support the greedier heuristic. We showed that our algorithms are faster both theoretically and practically, compared to contemporaneously proposed algorithms by Lipták et al. [18]. We also showed some relations between height-bounded encodings and existing repetitiveness measures. In particular, we have shown that the optimal LZ-like encoding with a height constraint of $O(\log n)$, is one of the asymptotically smallest repetitiveness measures achieving fast (i.e., $O(\polylog(n))$-time) access.

There are numerous avenues future work could take. An obvious next step is to engineer actual random-access data structures from height-bounded encodings, which our experiments presented here indicate should be competitive with current state-of-the-art methods. Also, other heuristics for reducing the height while retaining the small size should be explored.

From a theoretical perspective, a natural open problem is whether we can remove the $\log\sigma$ factor from the running times of LZHB3 and LZHB4. The biggest open problem we leave is whether better bounds on the size of height-bounded encodings can be established. Cicalese and Ugazio show that $\hat{z}_{\mathit{HB}(h)}=O(z)$ [8] cannot be achieved for costant $h$. We note that since there exist linear sized data structures that can answer predecessor queries for a set of values in the range $[1,n]$ in $O(\log\log n)$ time [33], even better lower bounds on the height for which $\hat{z}_{\mathit{HB}(h)}=O(z)$ can be shown by using access time lower bound results of Verbin and Yu [31]. However, these do not rule out $O(\polylog(n))$ height. Can we balance the height (achieve $O(\polylog(n))$ height) of an LZ-like encoding or a modified LZ-like encoding, by only increasing the size by a constant factor?