Lossy Compression in Near-Linear Time
via Efficient Random Codebooks and Databases

One of the last major outstanding classical problems of information theory is the development of general-purpose, practical, efficiently implementable lossy compression algorithms. The corresponding problem for lossless data compression was essentially settled in the late 1970s by the advance of the Lempel-Ziv (LZ) family of algorithms [58][56][59] and arithmetic coding [42][39][26]; see also the texts [20][5]. Similarly, from the early- to mid-1990s on, efficient channel coding strategies emerged that perform close to capacity, primarily using sparse graph codes, turbo codes, and local message-passing decoding algorithms; see, e.g., [47][30][44][31], the texts [15][29][40], and the references therein.

For lossy data compression, although there is a rich and varied literature on both theoretical results and practical compression schemes, near-optimal, efficiently implementable algorithms are yet to be discovered. From rate-distortion theory [6][43] we know that it is possible to achieve a sometimes dramatic improvement in compression performance by allowing for a certain amount of distortion in the reconstructed data. But the majority of existing algorithms are either compression-suboptimal or they involve exhaustive searches of exponential complexity at the encoder, making them unsuitable for realistic practical implementation.

Until the late 1990s, most of the research effort was devoted to addressing the issue of universality, see [23] and the references therein, as well as [55][37][57][38][36][53][54][49]; algorithms emphasizing more practical aspects have been proposed in [51]. In addition to many application-specific families of compression standards (e.g., JPEG for images and MPEG for video), there is a general theory of algorithm design based on vector quantization; see [16][27][7][17] and the references therein. Yet another line of research, closer in spirit to the present work, is on lossy extensions of the celebrated Lempel-Ziv schemes, based on approximate pattern matching; see [35][45][52][28][2][50][3][13][24][1].

More recently, there has been renewed interest in the compression-complexity trade-off, and in the development of low-complexity compressors that give near-optimal performance, at least for simple sources with known statistics. The lossy LZ algorithm of [24] is rate-distortion optimal and of polynomial complexity, although, in part due the penalty paid for universality, its convergence is slow. For the uniform Bernoulli source, [33][34][32] present codes based on sparse graphs, and, although their performance is promising, like earlier approaches they rely on exponential searches at the encoder. In related work, [46][8] present sparse-graph compression schemes with much more attractive complexity characteristics, but suboptimal compression performance. Rissanen and Tabus [41] describe a different method which, unlike most of the earlier approaches, is not based on a random (or otherwise exponentially large) codebook. It has linear complexity in the encoder and decoder and, although it appears to be rate-distortion suboptimal, it is an effective practical scheme for Bernoulli sources. Sparse-graph codes that are compression-optimal and of subexponential complexity are constructed in [18]. A simulation-based iterative algorithm is presented in [22] and it is shown to be compression-optimal, although its complexity is hard to evaluate precisely as it depends on the convergence of a Markov chain Monte Carlo sampler. The more recent work [21] on the lossy compression of discrete Markov sources also contains promising results; it is based on the combination of a Viterbi-like optimization algorithm at the encoder followed by universal lossless compression.

The present work is partly motivated by the results reported in [19] by Gupta-Verdú-Weissman (GVW). Their compression schemes are based on the “divide-and-conquer” approach, namely the idea that instead of encoding a long message $x_{1}^{n}=(x_{1},x_{2},\ldots,x_{n})$ using a classical random codebook of blocklength $n$, it is preferable to break up $x_{1}^{n}$ into shorter sub-blocks of shorter length $\ell$, say, and encode the sub-blocks separately. The main results in [19] state that, with an appropriately chosen sub-block length $\ell$, it is possible to achieve asymptotically optimal rate-distortion performance with near-linear implementation complexity (in a sense made precise in Section 3 below).

Our starting point is the observation that there is a closely related, in a sense dual, point of view. On a conceptual as well as mathematical level, the divide-and-conquer approach is very closely related to a pattern-matching scheme with a restricted database. In the divide-and-conquer setting, given a target distortion level $D$ and an $\ell\geq 1$, each sub-block of length $\ell$ in the original message $x_{1}^{n}$ is encoded using a random codebook consisting of $\approx 2^{\ell R(D)}$ codewords, where $R(D)$ is the rate-distortion function of the source being compressed (see the following section for more details and rigorous definitions). To encode each sub-block, the encoder searches all $2^{\ell R(D)}$ entries of the codebook, in order to find the one which has the smallest distortion with respect to that sub-block.

Now suppose that, instead of a random codebook, the encoder and decoder share a random database with length $M\approx 2^{\ell R(D)}$, generated from the same distribution as the Shannon-optimal codebook. As in [24], the encoder searches for the longest prefix $x_{1}^{L}=(x_{1},x_{2},\ldots,x_{L})$ of the message $x_{1}^{n}$ that matches somewhere in the database with distortion $D$ or less. Then the prefix $x_{1}^{L}$ is described to the decoder by describing the position and length of the match in the database, and the same process is repeated inductively starting at $x_{L+1}$. Although the match-length $L$ is random, we know [13][24] that, asymptotically, it behaves like,

Therefore, because the length $M$ of the database was chosen to be $\approx 2^{\ell R(D)}$, in effect both schemes will individually encode sub-blocks of approximately the same length $\ell$, and will also have comparable implementation complexity at the encoder.¹¹1It is well-known that the main difficulty in designing effective lossy compressors is in the implementation complexity of the encoder. Therefore, in all subsequent results dealing with complexity issues we focus on the case of the encoder. Moreover, it is easy to see that the decoding complexity of all the schemes considered here is linear in the message length.

Thus motivated, after reviewing the GVW scheme in Section 2 we introduce a (non-universal) version of the lossy LZ scheme in [24], termed LLZ, and we compare its performance to that of GVW. Theorem 1 shows that LLZ is asymptotically optimal in the rate-distortion sense for compressing data from a known discrete memoryless source with respect to a single-letter distortion criterion. Simulation results are also presented, comparing the performance of LLZ and GVW on a simple Bernoulli source. These results indicate that for blocklengths around 1000 bits, GVW offers better compression than LLZ at a given distortion level, but it requires significantly more memory for its execution. [The same findings are also confirmed in the other simulation examples presented in Section 4.]

In order to combine the different advantages of the two schemes, in Section 3 we introduce a hybrid algorithm (HYB), which utilizes both the divide-and-conquer idea of GVW and the single-database structure of LLZ. In Theorems 2 and 3 we prove that HYB shares with GVW the exact same rate-distortion performance and implementation complexity, in that it operates in near-linear time at the encoder and linear time at the decoder. Moreover, like LLZ, the HYB scheme requires much less memory, by an unbounded factor, depending on the choice of parameters in the design of the two algorithms. Experimental results are presented in Section 4, comparing the performance of GVW and HYB. These confirm the theoretical findings, and indicate that HYB outperforms existing schemes for the compression of some simple discrete sources with respect to the Hamming distortion criterion. The earlier theoretical results stating that HYB’s rate-distortion performance is the same as GVW’s are confirmed empirically, and it is also shown that, again for blocklengths of approximately 1000 symbols, the HYB scheme requires much less memory, by a factor ranging between 15 and 240.

After a brief discussion on potential extensions of the present results, some conclusions are collected in Section 5. The appendix contains the proofs of the theorems in Sections 2 and 3.

After describing the basic setting within which all later results will be developed, in Section 2.2 we recall the divide-and-conquer idea of the GVW scheme, and in Section 2.3 we present a new, non-universal lossy LZ algorithm and examine its properties.

In order to combine the rate-distortion advantage of GVW with the memory advantage of LLZ, in this section we introduce a hybrid algorithm and examine its performance.

The new algorithm, termed HYB, uses the divide-and-conquer approach of GVW, but based on a random database like the LLZ instead of a random codebook. It is a fixed-rate, variable-distortion code of blocklength $n$ and target distortion $D\in(0,\mbox{$D_{\rm max}$})$, and it is described in terms of two parameters; a “small” $\gamma>0$, and an integer $\ell$ so that $n=k\ell$.

Like with the GVW, given a target distortion level $D$, let $R=R(D)+\gamma$ and take $\overline{D}$ as in (2). Now, like for the LLZ algorithm, let $m=m(\ell)=\lfloor 2^{\ell R}\rfloor+\ell-1$ as in (3), and generate a random database $Y_{1}^{m}=(Y_{1},Y_{2},\ldots,Y_{m})$, where the $Y_{i}$ are drawn i.i.d. from the optimal reproduction distribution at level $\overline{D}$. The database is made available to both the encoder and the decoder, and the message $x_{1}^{n}$ to be compressed is parsed into $k=n/\ell$ non-overlapping blocks, $x_{1}^{n}=x_{1}^{\ell}*x_{\ell+1}^{2\ell}*\cdots*x_{(k-1)\ell+1}^{k\ell}$.

The first sub-block $x_{1}^{\ell}$ is matched to its $\rho_{\ell}$-nearest neighbor in the database, where we consider each possible $Y_{i}^{i+\ell-l}$, $i=1,2,\ldots,\lfloor 2^{\ell R}\rfloor$ as a potential reproduction word. Then $x_{1}^{\ell}$ is described to the decoder by describing the position of its matching reproduction block in the database using $\approx\ell R$ bits, and the same process is repeated on each of the $k$ sub-blocks, to produce $k$ reconstruction strings. The description of $x_{1}^{n}$ is the concatenation of the descriptions of the individual sub-blocks, and the reconstruction string itself is the concatenation of the corresponding reproduction blocks. The overall description length of this code is $k\lceil\log\lfloor 2^{\ell R}\rfloor\rceil\leq k\ell R=nR$ bits.

The following result shows that the HYB algorithm shares the exact same rate-distortion performance, as well as the same implementation complexity characteristics, as the GVW. Let:

– Encoding time per source symbol is proportional to $(\lambda_{1}/\epsilon)^{\lambda_{2}(D)/\gamma^{2}}$,

– Decoding time per symbol is independent of $\gamma$ and $\epsilon$,

1. Theorem 2 is an exact analog of Theorem 1 proved for GVW in [19], the only difference being that we consider average distortion instead of the probability-of-excess distortion criterion. The reason is that, instead of presenting an existence proof for an algorithm with certain desired properties, here we examine the performance of the HYB algorithm itself. Indeed, the proof of Theorem 2 can easily be modified to prove the stronger claim that there exists some instance of the random database $Y_{1}^{m}$ such that, using that particular database, the HYB algorithm also has the additional property that the probability of excess distortion vanishes as $n\to\infty$. The same comments apply to Theorem 3 below.

2. In [19] a similar result is proved with the roles and $\epsilon$ and $\gamma$ interchanged. In fact, it should be pointed out that the scheme we call “the” GVW algorithm here corresponds to the scheme used in the proof of [19, Theorem 1]. A slight variant (having to do with the choice of parameter values and not with the mechanics of the algorithm itself) is used to prove [19, Theorem 2]. Having gone over the proofs, it would be obvious to the reader that, once the corresponding changes are made for HYB, an analogous result can be proved for HYB. The straightforward but tedious details are omitted.

3. In terms of memory, the GVW scheme requires $\ell\lfloor 2^{\ell R}\rfloor$ reproduction symbols for storing the codebook, while using the same memory parameters the HYB algorithm needs $m(\ell)=\lfloor 2^{\ell R}\rfloor+\ell-1$ symbols. The ratio between the two is,

so that the GVW needs $\approx\ell$ times more memory than HYB. Moreover, the closer we require the algorithm to come to achieving an optimal $(D,R(D))$ point, the smaller the values of $\epsilon$ and $\alpha$ need to be taken in Theorem 2, and the larger the corresponding value of $\ell$; cf. equation (12). Therefore, not only the difference, but even the ratio of the memory required by GVW compared to HYB, is unbounded.

The next result shows that, choosing the parameters $\ell$ and $\gamma$ in HYB appropriately, optimal compression performance can be achieved with linear decoding complexity and near-linear encoding complexity. It is a parallel result to [19, Theorem 3].

The actual empirical performance of HYB on simulated data is compared to that of GVW and LLZ in the following section.

Here the empirical performance of the HYB scheme is compared with that of GVW and LLZ, on three simulated data sets from simple memoryless sources.⁵⁵5We do not present comparison results with earlier schemes apart from the GVW, since extensive such studies already exist in the literature; in particular, the GVW is compared in [19] with the algorithms proposed in [46], [18] and [41]. The following parameter values were used in all of the experiments. For the GVW and HYB algorithms, $\ell$ was chosen as in [19] to be $\ell=\lceil\frac{22}{R(D)+\gamma}\rceil$, where $R(D)$ is the rate-distortion function of the source, and $\gamma$ was taken equal to $0.002$. Similarly, for LLZ we took $\ell=\lceil 22/R(D)\rceil$, $\gamma=0.03$ and $\alpha=0.1$. Note that, with this choice of parameters, the complexity of all three algorithms is approximately linear in the message length $n$. All experiments were performed on a Sony Vaio laptop running Ubuntu Linux, under identical conditions.⁶⁶6Although there is a wealth of efficient algorithms for the problem of approximate string matching (see, e.g., [11][2][4][10] and the references therein), since HYB clearly outperforms LLZ, our version of the LLZ scheme was implemented using the naive, greedy scheme consistent with the definition of algorithm.

First we revisit the example of Section 2; $n=1050$ bits generated by a Bernoulli source with parameter $p=0.4$, are compressed by all three algorithms at various different distortion levels with respect to Hamming distortion. Figure 2 shows the rate-distortion pairs achieved.

The second example is again on a Bernoulli source with respect to Hamming distortion, this time with source parameter $p=0.2$. The corresponding simulation results are displayed in Figure 3 and Table 3.

Finally, in the third example $\{X_{n}\}$ is taken as a memoryless source uniformly distributed on $\{0,1,2,3\}$, to be compressed with respect to Hamming distortion. The empirical results are shown in Figure 4 and Table 4.

In both these cases, the same qualitative conclusions are drawn. The rate-distortion performance of the GVW and HYB algorithms is essentially indistinguishable, while the compression achieved by LLZ is generally somewhat worse, though in several instances not significantly so. In the second example note that the memory required by HYB is smaller than that of GVW by a factor that ranges between 44 and 242, while in the third example the corresponding factors are between 16 and 218. And again, although the theoretical implementation complexity of GVW and HYB is identical, because of their different memory requirements the encoding time of HYB is smaller than that of GVW by a factor ranging between approximately 3 and 9 in the second example, and between 1.25 and 1.5 in the third example.

The starting point for this work was the observation that there is a certain duality relationship between the divide-and-conquer compression schemes of Gupta-Verdú-Weissman (GVW) in [19], and certain lossy Lempel-Ziv schemes based on a fixed-database as in [24]. To explore this duality, LLZ, a new (non-universal) lossy LZ algorithm was introduced, and it was shown to be asymptotically rate-distortion optimal. To combine the low-complexity advantage of GVW with the low-memory requirement of LLZ, a hybrid algorithm, called HYB, was then proposed, and its properties were explored both theoretically and empirically.

The main contribution of this short paper is the introduction of memory considerations in the usual compression-complexity trade-off. Building on the success of the GVW algorithm, it was shown that the HYB scheme simultaneously achieves three goals: 1. Its rate-distortion performance can be made arbitrarily close to the fundamental rate-distortion limit; 2. The encoding complexity can be tuned in a rigorous manner so as to balance the trade-off of encoding complexity vs. compression redundancy; and 3. The memory required for the execution of the algorithm is much smaller than that required by GVW, a difference which may be made arbitrarily large depending on the choice of parameters.

Moreover, empirically, for blocklengths of the order of thousands, the HYB scheme appears to outperform existing schemes for the compression of simple memoryless sources with respect to Hamming distortion.

Lastly, we briefly mention that the results presented in this paper can be extended in several directions. First we note that the finite-alphabet assumption was made exclusively for the sake of simplicity of exposition and to avoid cumbersome technicalities. While keeping the structure of all three algorithms exactly the same, this assumption can easily be relaxed, at the price of longer, more technical proofs, along the lines of arguments, e.g., in [49][24][25][19]. For example, Theorem 4 of [19] which gives precise performance and complexity bounds for the GVW used with general source and reproduction alphabets and with respect to an unbounded distortion measure, can easily be generalized to HYB. Similarly, Theorem 5 of [19] which describes the performance of a universal version of GVW can also be generalized to the corresponding statement for a universal version of HYB (with obvious modifications), although, as noted in [19], the utility of that result is purely of theoretical interest.

We thank Sergio Verdú for sharing with us preprints of [18] and [19].