Sublinear Algorithms for Approximating String Compressibility

For RLE, we present sublinear algorithms for all three approximation notions defined above, providing a trade-off between the quality of approximation and the running time. The algorithms that allow an additive approximation run in time independent of the input size. Specifically, an $\epsilon n$-additive estimate can be obtained in time³³3The notation $\tilde{O}(g(k))$ for a function $g$ of a parameter $k$ means $O(g(k)\cdot{\rm polylog}(g(k))$ where ${\rm polylog}(g(k))=\log^{c}(g(k))$ for some constant $c$. $\tilde{O}(1/\epsilon^{3})$, and a combined estimate, with a multiplicative error of $3$ and an additive error of $\epsilon n$, can be obtained in time $\tilde{O}(1/\epsilon)$. As for a strict multiplicative approximation, we give a simple 4-multiplicative approximation algorithm that runs in expected time $\tilde{O}(\frac{n}{C_{\rm rle}(w)})$ where $C_{\rm rle}(w)$ denotes the compression cost of the string $w$. For any $\gamma>0$, the multiplicative error can be improved to $1+\gamma$ at the cost of multiplying the running time by ${\rm poly}(1/\gamma)$. Observe that the algorithm is more efficient when the string is less compressible, and less efficient when the string is more compressible. One of our lower bounds justifies such a behavior and, in particular, shows that a constant factor approximation requires linear time for strings that are very compressible. We also give a lower bound of $\Omega(1/\epsilon^{2})$ for $\epsilon n$-additive approximation.

We prove that approximating compressibility with respect to LZ is closely related to the following problem, which we call Colors: Given access to a string $\tau$ of length $n$ over alphabet $\Psi$, approximate the number of distinct symbols (“colors”) in $\tau$. This is essentially equivalent to estimating the support size of a distribution [17]. Variants of this problem have been considered under various guises in the literature: in databases it is referred to as approximating distinct values (Charikar et al.  [7]), in statistics as estimating the number of species in a population (see the over 800 references maintained by Bunge [6]), and in streaming as approximating the frequency moment $F_{0}$ (Alon et al.  [1], Bar-Yossef et al.  [2]). Most of these works, however, consider models different from ours. For our model, there is an $A$-multiplicative approximation algorithm of [7], that runs in time $O\left(\frac{n}{A^{2}}\right)$, matching the lower bound in [7, 2]. There is also an almost linear lower bound for approximating Colors with additive error [17].

We give a reduction from LZ compressibility to Colors and vice versa. These reductions allow us to employ the known results on Colors to give algorithms and lower bounds for this problem. Our approximation algorithm for LZ compressibility combines a multiplicative and additive error. The running time of the algorithm is $\tilde{O}\left(\frac{n}{A^{3}\epsilon}\right)$ where $A$ is the multiplicative error and $\epsilon n$ is the additive error. In particular, this implies that for any $\alpha>0$, we can distinguish, in sublinear time $\tilde{O}(n^{1-\alpha})$, strings compressible to $O(n^{1-\alpha})$ symbols from strings only compressible to $\Omega(n)$ symbols.⁴⁴4To see this, set $A=o(n^{\alpha/2})$ and $\epsilon=o(n^{-\alpha/2})$.

The main tool in the algorithm consists of two combinatorial structural lemmas that relate compressibility of the string to the number of distinct short substrings contained in it. Roughly, they say that a string is well compressible with respect to LZ if and only if it contains few distinct substrings of length $\ell$ for all small $\ell$ (when considering all $n-\ell+1$ possible overlapping substrings). The simpler of the two lemmas was inspired by a structural lemma for grammars by Lehman and Shelat [12]. The combinatorial lemmas allow us to establish a reduction from LZ compressibility to Colors and employ a (simple) algorithm for approximating Colors in our algorithm for LZ.

Interestingly, we can show that there is also a reduction in the opposite direction: namely, approximating Colors reduces to approximating LZ compressibility. The lower bound of [17], combined with the reduction from Colors to LZ, implies that our algorithm for LZ cannot be improved significantly. In particular, our lower bound implies that for any $B=n^{o(1)}$, distinguishing strings compressible by LZ to $\tilde{O}(n/B)$ symbols from strings compressible to $\tilde{\Omega}(n)$ symbols requires $n^{1-o(1)}$ queries.

It would be interesting to extend our results for estimating the compressibility under LZ77 to other variants of LZ, such as dictionary-based LZ78 [19]. Compressibility under LZ78 can be drastically different from compressibility under LZ77: e.g., for $0^{n}$ they differ roughly by a factor of $\sqrt{n}$. Another open question is approximating compressibility for schemes other than RLE and LZ. In particular, it would be interesting to design approximation algorithms for lossy compression schemes such as JPEG, MPEG and MP3. One lossy compression scheme to which our results extend directly is Lossy RLE, where some characters, e.g., the ones that represent similar colors, are treated as the same character.

We start with some definitions in Section 2. Section 3 contains our results for RLE. Section 4 deals with the LZ scheme. All missing details (descriptions of algorithms and proofs of claims) can be found in [16].

Our first algorithm for approximating the cost of RLE is very simple: it samples a few positions in the input string uniformly at random and bounds the lengths of the runs to which they belong by looking at the positions to the left and to the right of each sample. If the corresponding run is short, its length is established exactly; if it is long, we argue that it does not contribute much to the encoding cost. For each index $t\in[n]$, let $\ell(t)$ be the length of the run to which $w_{t}$ belongs. The cost contribution of index $t$ is defined as

By definition, $\displaystyle\frac{C_{\rm rle}(w)}{n}=\operatorname*{{\mathsf{E}}}_{t\in[n]}[c(t)]$, where $\operatorname*{{\mathsf{E}}}_{t\in[n]}$ denotes expectation over a uniformly random choice of $t$. The algorithm, presented below, estimates the encoding cost by the average of the cost contributions of the sampled short runs, multiplied by $n$.

Algorithm I: An $\epsilon n$-additive Approximation for $C_{\rm rle}(w)$ 1. Select $q=\Theta\left(\frac{1}{\epsilon^{2}}\right)$ indices $t_{1},\dots,t_{q}$ uniformly and independently at random. 2. For each $i\in[q]:$ (a) Query $t_{i}$ and up to $\ell_{0}=\frac{8\log(4|\Sigma|/\epsilon)}{\epsilon}$ positions in its vicinity to bound $\ell(t_{i})$. (b) Set $\hat{c}(t_{i})=c(t_{i})$ if $\ell(t_{i})<\ell_{0}$ and $\hat{c}(t_{i})=0$ otherwise. 3. Output $\displaystyle\widehat{C}_{\rm rle}=n\cdot\operatorname*{{\mathsf{E}}}_{i\in[q]}[\hat{c}(t_{i})]$.

After stating Theorem 1 that summarizes our positive results, we briefly discuss some of the ideas used in the algorithms omitted from this version of the paper.

Section 3.1 gives a complete proof of Item 1. The algorithm in Item 2 partitions the positions in the string into buckets according to the length of the runs they belong to. It estimates the sizes of different buckets with different precision, depending on the size of the bucket and the length of the runs it contains. The main idea in Item 3 is to search for $C_{\rm rle}(w)$, using the algorithm from Item 2 repeatedly (with different parameters) to establish successively better estimates.

We give two lower bounds, for multiplicative and additive approximation, respectively, which establish that the running times in Items 1 and 3 of Theorem 1 are essentially tight.

Our algorithm for approximating the compressibility of an input string with respect to LZ77 uses an approximation algorithm for Colors (defined in the introduction) as a subroutine. The main tool in the reduction from LZ77 to Colors is the relation between $C_{\text{LZ}}(w)$ and the number of distinct substrings in $w$, formalized in the two structural lemmas. In what follows, $d_{\ell}(w)$ denotes the number of distinct substrings of length $\ell$ in $w$. Unlike compressed segments in $w$, which are disjoint, these substrings may overlap.

For the general case, fix $\ell>1$. Recall that $w_{t}\dots w_{t+k-1}$ of $w$ is a compressed segment if it is represented by one symbol $(p,k)$ in ${LZ}(w)$. Any substring of lenth $\ell$ that occurs within a compressed segment must have occurred previously in the string. Such substrings can be ignored for our purposes: the number of distinct length-$\ell$ substrings is bounded above by the number of length-$\ell$ substrings that start inside one compressed segment and end in another. Each segment (except the last) contributes $(\ell-1)$ such substrings. Therefore, $d_{\ell}(w)\leq(C_{\text{LZ}}(w)-1)(\ell-1)<C_{\text{LZ}}(w)\cdot\ell$ for every $\ell>1$.     

It remains to prove Equation (4). We do so below by induction on $\ell$, using the following claim.

We call a substring new if no instance of it started in the previous portion of $w$. Namely, $w_{t}\dots w_{t+\ell-1}$ is new if there is no $p<t$ such that $w_{t}\dots w_{t+\ell-1}=w_{p}\dots w_{p+\ell-1}$. Consider a compressed substring $w_{t}\dots w_{t+k-1}$ of length $k\leq\ell$. The substrings of length greater than $k$ that start at $w_{t}$ must be new, since LZ77 finds the longest substring that appeared before. Furthermore, every substring that contains such a new substring is also new. That is, every substring $w_{t^{\prime}}\dots w_{t+k^{\prime}}$ where $t^{\prime}\leq t$ and $k^{\prime}\geq k+(t^{\prime}-t)$, is new.

Map each position $j\in\{\ell,\dots,n-\ell\}$ in the compressed substring $w_{t}\dots w_{t+k-1}$ to the length-$2\ell$ substring that ends at $w_{j+\ell}$. Then each position in $\{\ell,\dots,n-\ell\}$ that appears in a compressed substring of length at most $\ell$ is mapped to a distinct length-$2\ell$ substring, as desired.      (Claim 6)

This subsection describes an algorithm for approximating the compressibility of an input string with respect to LZ77, which uses an approximation algorithm for Colors as a subroutine. The main tool in the reduction from LZ77 to Colors consists of structural lemmas 4 and 5, summarized in the following corollary.

The corollary allows us to approximate $C_{\text{LZ}}$ from estimates for $d_{\ell}$ for all $\ell\in[\ell_{0}]$. To obtain these estimates, we use the algorithm of [7] for Colors as a subroutine (in the full version [16] we also describe a simpler Colors algorithm with the same provable guarantees). Recall that an algorithm for Colors approximates the number of distinct colors in an input string, where the $i$th character represents the $i$th color. We denote the number of colors in an input string $\tau$ by ${C_{\rm COL}}(\tau)$. To approximate $d_{\ell}$, the number of distinct length-$\ell$ substrings in $w$, using an algorithm for Colors, view each length-$\ell$ substring as a separate color. Each query of the algorithm for Colors can be implemented by $\ell$ queries to $w$.

Let $\mbox{\sc Estimate}(\ell,B,\delta)$ be a procedure that, given access to $w$, an index $\ell\in[n]$, an approximation parameter $B=B(n,\ell)>1$ and a confidence parameter $\delta\in[0,1]$, computes a $B$-estimate for $d_{\ell}$ with probability at least $1-\delta$. It can be implemented using an algorithm for Colors, as described above, and employing standard amplification techniques to boost success probability from $\frac{2}{3}$ to $1-\delta$: running the basic algorithm $\Theta(\log\delta^{-1})$ times and outputting the median. Since the algorithm of [7] requires $O(n/B^{2})$ queries, the query complexity of $\mbox{\sc Estimate}(\ell,B,\delta)$ is $O\left(\frac{n}{B^{2}}\;\ell\log\delta^{-1}\right)$. Using $\mbox{\sc Estimate}(\ell,B,\delta)$ as a subroutine, we get the following approximation algorithm for the cost of LZ77.

Algorithm II: An $(A,\epsilon)$-approximation for $C_{\text{LZ}}(w)$ 1. Set $\ell_{0}=\left\lceil{\frac{2}{A\epsilon}}\right\rceil$ and $B=\frac{A}{2\sqrt{\log(2/(A\epsilon))}}$. 2. For all $\ell$ in $[\ell_{0}]$, let $\hat{d}_{\ell}=\mbox{\sc Estimate}(\ell,B,\frac{1}{3\ell_{0}})$. 3. Combine the estimates to get an approximation of $m$ from Corollary 7:   set $\hat{m}=\displaystyle\max_{\ell}\frac{\hat{d}_{\ell}}{\ell}$. 4. Output $\widehat{C}_{\text{LZ}}=\hat{m}\cdot{\frac{A}{B}}+\epsilon n.$

As explained before the algorithm statement, each call to $\mbox{\sc Estimate}(\ell,B,\frac{1}{3\ell_{0}})$ costs $O\left(\frac{n}{B^{2}}\;\ell\log\ell_{0}\right)$ queries. Since the subroutine is called for all $\ell\in[\ell_{0}]$, the straightforward implementation of the algorithm would result in $O\left(\frac{n}{B^{2}}\ell^{2}_{0}\log\ell_{0}\right)$ queries. Our analysis of the algorithm, however, does not rely on independence of queries used in different calls to the subroutine, since we employ the Union Bound to calculate the error probability. It will still apply if we first run Estimate to approximate $d_{\ell_{0}}$ and then reuse its queries for the remaining calls to the subroutine, as though it requested to query only the length-$\ell$ prefixes of the length-$\ell_{0}$ substrings queried in the first call. With this implementation, the query complexity is $O\left(\frac{n}{B^{2}}\ell_{0}\log\ell_{0}\right)=O\left(\frac{n}{A^{3}\epsilon}\log^{2}\frac{1}{A\epsilon}\right).$ To get the same running time, one can maintain counters for all $\ell\in[\ell_{0}]$ for the number of distinct length-$\ell$ substrings seen so far and use a trie to keep the information about the queried substrings. Every time a new node at some depth $\ell$ is added to the trie, the $\ell$th counter is incremented.     

We have demonstrated that estimating the LZ77 compressibility of a string reduces to Colors. As shown in [17], Colors is quite hard, and it is not possible to improve much on the simple approximation algorithm in [7] , on which we base the LZ77 approximation algorithm in the previous subsection. A natural question is whether there is a better algorithm for the LZ77 estimation problem. That is, is the LZ77 estimation strictly easier than Colors? As we shall see, it is not much easier in general.

Two notes are in place regarding the reduction. The first is that the gap between the parameters $\alpha^{\prime}$ and $\beta^{\prime}$ that is required by the Colors algorithm obtained in Lemma 9, is larger than the gap between the parameters $\alpha$ and $\beta$ for which the LZ-compressibility algorithm works, by a factor of $4\cdot\max\left\{1,\frac{4\log n^{\prime}}{\log|\Sigma|}\right\}$. In particular, for binary strings $\frac{\beta^{\prime}}{\alpha^{\prime}}=O\left(\log n^{\prime}\cdot\frac{\beta}{\alpha}\right)$, while if the alphabet is large, say, of size at least $n^{\prime}$, then $\frac{\beta^{\prime}}{\alpha^{\prime}}=O\left(\frac{\beta}{\alpha}\right)$. In general, the gap increases by at most $O(\log n^{\prime})$. The second note is that the number of queries, $q$, is a function of the parameters of the LZ-compressibility problem and, in particular, of the length of the input strings, $n$. Hence, when writing $q$ as a function of the parameters of Colors and, in particular, as a function of $n^{\prime}=\Theta(\alpha n)$, the complexity may be somewhat larger. It is an open question whether a reduction without such increase is possible.

Prior to proving the lemma , we discuss its implications.  [17] give a strong lower bound on the sample complexity of approximation algorithms for Colors. An interesting special case is that a subpolynomial-factor approximation for Colors requires many queries even with a promise that the strings are only slightly compressible: for any $B=n^{o(1)}$, distinguishing inputs with $n/11$ colors from those with $n/B$ colors requires $n^{1-o(1)}$ queries. Lemma 9 extends that bound to estimating LZ compressibility: For any $B=n^{o(1)}$, and any alphabet $\Sigma$, distinguishing strings with LZ compression cost $\tilde{\Omega}(n)$ from strings with cost $\tilde{O}(n/B)$ requires $n^{1-o(1)}$ queries.

The lower bound for Colors in [17] applies to a broad range of parameters, and yields the following general statement when combined with Lemma 9:

Given a Colors instance $\tau$ of length $n^{\prime}$, we transform it into a string of length $n=n^{\prime}\cdot k$ over $\Sigma$, where $k=\lceil\frac{1}{\alpha}\rceil$. We then run ${\cal A}_{\text{LZ}}$ on $w$ to obtain information about $\tau$. We begin by replacing each color in $\tau$ with a uniformly selected substring in $\Sigma^{k}$. The string $w$ is the concatenation of the corresponding substrings (which we call blocks). We show that:

1. 1.

   If $\tau$ has at most $\alpha^{\prime}n^{\prime}$ colors, then $C_{\text{LZ}}(w)\leq 2\alpha^{\prime}n$;

2. 2.

   If $\tau$ has at least $\beta^{\prime}n^{\prime}$ colors, then ${\rm Pr}_{w}[C_{\text{LZ}}(w)\geq\frac{1}{2}\cdot\min\left\{1,\frac{\log|\Sigma|}{4\log n^{\prime}}\right\}\cdot\beta^{\prime}n]\geq\frac{7}{8}.$

That is, in the first case we get an input $w$ for Colors such that $C_{\text{LZ}}(w)\leq\alpha n$ for $\alpha=2\alpha^{\prime}$, and in the second case, with probability at least $7/8$, $C_{\text{LZ}}(w)\geq\beta n$ for $\beta=\frac{1}{2}\cdot\min\left\{1,\frac{\log|\Sigma|}{4\log n^{\prime}}\right\}\cdot\beta^{\prime}$. Recall that the gap between $\alpha^{\prime}$ and $\beta^{\prime}$ is assumed to be sufficiently large so that $\alpha<\beta$. To distinguish the case that ${C_{\rm COL}}(\tau)\leq\alpha^{\prime}n^{\prime}$ from the case that ${C_{\rm COL}}(\tau)>\beta^{\prime}n^{\prime}$, we can run ${\cal A}_{\text{LZ}}$ on $w$ and output its answer. Taking into account the failure probability of ${\cal A}_{\text{LZ}}$ and the failure probability in Item 2 above, the Lemma follows.

We prove these two claims momentarily, but first observe that in order to run the algorithm ${\cal A}_{\text{LZ}}$, there is no need to generate the whole string $w$. Rather, upon each query of ${\cal A}_{\text{LZ}}$ to $w$, if the index of the query belongs to a block that has already been generated, the answer to ${\cal A}_{\text{LZ}}$ is determined. Otherwise, we query the element (color) in $\tau$ that corresponds to the block. If this color was not yet observed, then we set the block to a uniformly selected substring in $\Sigma^{k}$. If this color was already observed in $\tau$, then we set the block according to the substring that was already selected for the color. In either case, the query to $w$ can now be answered. Thus, each query to $w$ is answered by performing at most one query to $\tau$.

It remains to prove the two items concerning the relation between the number of colors in $\tau$ and $C_{\text{LZ}}(w)$. If $\tau$ has at most $\alpha^{\prime}n^{\prime}$ colors then $w$ contains at most $\alpha^{\prime}n^{\prime}$ distinct blocks. Since each block is of length $k$, at most $k$ compressed segments start in each new block. By definition of LZ77, at most one compressed segment starts in each repeated block. Hence,

If $\tau$ contains $\beta^{\prime}n^{\prime}$ or more colors, $w$ is generated using at least $\beta^{\prime}n^{\prime}\cdot\log(|\Sigma|^{k})=\beta^{\prime}n\log|\Sigma|$ random bits. Hence, with high probability (e.g., at least $7/8$) over the choice of these random bits, any lossless compression algorithm (and in particular LZ77) must use at least $\beta^{\prime}n\log|\Sigma|-3$ bits to compress $w$. Each symbol of the compressed version of $w$ can be represented by $\max\{\lceil\log|\Sigma|\rceil,2\lceil\log n\rceil\}+1$ bits, since it is either an alphabet symbol or a pointer-length pair. Since $n=n^{\prime}\lceil 1/\alpha^{\prime}\rceil$, and $\alpha^{\prime}>1/n^{\prime}$, each symbol takes at most $\max\{4\log n^{\prime},\log|\Sigma|\}+2$ bits to represent. This means the number of symbols in the compressed version of $w$ is

where we have used the fact that $\beta^{\prime}n^{\prime}$, and hence $\beta^{\prime}n$, is at least some sufficiently large constant.