Compressibility and probabilistic proofs

We repeat the same simple proof three times, in three different languages.

(Probabilistic language) Let us choose matrix elements using independent tosses of a fair coin. For a given $k$ colums and $k$ rows, the probability of getting a monochromatic minor at their intersection is $2^{-k^{2}+1}$. (Both zero-minor and one-minor have probability $2^{-k^{2}}$.) There are at most $n^{k}$ choices for columns and the same number for rows, so by the union bound the probability of getting at least one monochromatic minor is bounded by

and the last expression is less then $1$ if, say, $k=3\log n$ and $n$ is suffuciently large.

(Combinatorial language) Let us count the number of bad matrices. For a given choice of columns and rows we have $2$ possibilities for the minor and $2^{n^{2}-k^{2}}$ possibilities for the rest, and there is at most $n^{k}$ choices for raws and columns, so the total number of matrices with monochromatic minor is

and this is less than $2^{n^{2}}$, the total number of Boolean $(n\times n)$-matrices.

(Compression language) To specify the matrix that has a monochromatic minor, it is enough to specify $2k$ numbers between $1$ and $n$ (rows and column numbers), the color of the monochromatic minor ($0$ or $1$) and the remaining $n^{2}-k^{2}$ bits in the matrix (their positions are already known). So we save $k^{2}$ bits (compared to the straightforward list of all $n^{2}$ bits) using $2k\log n+1$ bits instead (each number in the range $1\ldots n$ requires $\log n$ bits; to be exact, we may use $\lceil\log n\rceil$), so we can compress the matrix with a monochromatic minor if $2k\log n+1\ll k^{2}$, and not all matrices are compressible. ∎

Assume that some rotation (cyclic shift by $k$ positions) transforms $x$ into a string $y$ that coincides almost everywhere with $x$. We may assume that $k\leq n/2$: the cyclic shift by $k$ positions changes as many bits as the cyclic shift by $n-k$ (the inverse one). Imagine that we dictate the string $x$ from left to right. First $k$ bits we dictate normally. But then the bits start to repeat (mostly) the previous ones ($k$ positions before), so we can just say “the same” or “not the same”, and if $\varepsilon$ is small, we know that most of the time we say “the same”. Technically, we have $\varepsilon n$ different bits, and at least $n-k\geq n/2$ bits to dictate after the first $k$, so the fraction of “not the same” signals is at most $2\varepsilon$. It is well known that strings of symbols where some symbols appear more often than others can be encoded efficiently. Shannon tells us that a string with two symbols with frequencies $p$ and $q$ (so $p+q=1$) can be encoded using

bits per symbol and that $H(p,q)=1$ only when $p=q=1/2$. In our case, for small $\varepsilon$, one of the frequencies is close to $0$ (at most $2\varepsilon$), and the other one is close to $1$, so $H(p,q)$ is significantly less than $1$. So we get a significant compression for every string that is bad for the theorem, therefore most string are good (so good string do exist).

More precisely, every string $x_{0}\ldots x_{n-1}$ that does not satisfy the requirements, can be described by

• •

  $k$  [$\log n$ bits]

• •

  $x_{0},\ldots,x_{k-1}$  [$k$ bits]

• •

  $x_{k}\oplus x_{0},x_{k+1}\oplus x_{1},\ldots,x_{n-1}\oplus x_{n-k-1}$   [$n-k$ bits where the fraction of $1$s is at most $2\varepsilon$, compressed to $(n-k)H(2\varepsilon,1-2\varepsilon)$ bits]

For $\varepsilon<1/4$ and for large enough $n$ the economy in the third part (compared to $n-k$) is more important than $\log n$ in the first part. ∎

It is convenient to present a proof using the compression language, as suggested by Lance Fortnow. Consider the following procedure $\textsc{Fix}(C)$ whose argument is a clause (from our CNF).

{ $C$ is false }

$\textsc{Fix}(C)$:

$\textsc{Resample}(C)$

for all $C^{\prime}$ that are neighbors of $C$:

if $C^{\prime}$ is false then $\textsc{Fix}(C^{\prime})$

{ $C$ is true; other clauses that were true remain true }

Here $\textsc{Resample}(C)$ is the procedure that assigns fresh random values to all variables in $C$. The pre-condition (the first line) says that the procedure is called only in the situation where $C$ is false. The post-condition (the last line) says that if the procedure terminates, then $C$ is true after termination, and, moreover, all other clauses of our CNF that were true before the call remain true. (The ones that were false may be true or false.)

Note that up to now we do not say anything about the termination: note that the procedure is randomized and it may happen that it does not terminate (for example, if all Resample calls are unlucky to choose the same old bad values).

Simple observation: if we have such a procedure, we may apply it to all clauses one by one and after all calls (assuming they terminate and the procedure works according to the specification) we get a satisfying assignment.

Another simple observation: it is easy to prove the “conditional correctness” of the procedure $\textsc{Fix}(C)$. In other words, it achieves its goal assuming that (1) it terminates; (2) all the recursive calls $\textsc{Fix}(C^{\prime})$ achieve their goals. It is almost obvious: the $\textsc{Resample}(C)$ call may destroy (=make false) only clauses that are neighbors to $C$, and all these clauses are Fix-ed after that. Note that $C$ is its own neighbor, so the for-loop includes also a recursive call $\textsc{Fix}(C)$, so after all these calls (that terminate and satisfy the post-condition by assumption) the clause $C$ and all its neighbors are true and no other clause is damaged.

Note that the last argument remains valid even if we delete the only line that really changes something, i.e., the line $\textsc{Resample}(C)$. In this case the procedure never changes anything but still is conditionally correct; it just does not terminate if one of the clauses is false.

It remains to prove that the call $\textsc{Fix}(C)$ terminates with high probability. In fact, it terminates with probability $1$ if there are no time limits and with probability exponentially close to $1$ in polynomial time. To prove this, one may use a compression argument: we show that if the procedure works for a long time without terminating, then the sequence of random bits used for resampling is compressible. We assume that each call of $\textsc{Resample}()$ uses $n$ fresh bits from the sequence. Finally, we note that this compressibility may happen only with exponentially small probability.

Imagine that $\textsc{Fix}(C)$ is called and during its recursive execution performs many calls

(in this order) but does not terminate (yet). We stop it at some moment and examine the values of all the variables.

Now we need to show that the sequence of clauses $C_{1},\ldots,C_{N}$ used for resampling can be described by less bits than $nN$ (the number of random bits used). Here we use the assumption saying each clause has at most $2^{n-3}$ neighbors and that the clauses $C^{\prime}$ for which $\textsc{Fix}(C^{\prime})$ is called from $\textsc{Fix}(C)$, are neighbors of $C$.

One could try to say that since $C_{i+1}$ is a neighbor of $C_{i}$, we need only $n-3$ bits to specify it (there are at most $2^{n-3}$ neighbors by assumption), so we save $3$ bits per clause (compared to $n$ random bits used by resampling). But this argument is wrong: $C_{i+1}$ is not always the neighbor of $C_{i}$, since we may return from a recursive call that causes resampling of $C_{i}$ and then make a new recursive call that resamples $C_{i+1}$.

To get a correct argument, we should look more closely at the tree of recursive calls generated by one call $\textsc{Fix}(C)$ (Fig. 2). In this tree the sons of each vertex correspond to neighbor clauses of the father-clause.

The sequence of calls is determined by a walk in this tree, but we go up and down, not only up (as we assumed in the wrong argument). How many bits we need to encode this walk (and therefore the sequence of calls)? We use one bit to distinguish between steps up and down. If we are going down, no other information is needed. If we are going up (and resample a new clause), we need one bit to say that we are going up, and $n-3$ bits for the number of neighbor we are going to. For accounting purposes we combine these bits with a bit needed to encode the step back (this may happen later or not happen at all), and we see that in total we need at most $(n-3)+1+1=n-1$ bits per each resampling. This is still less than $n$, so we save one bit for each resampling. If $N$ is much bigger than the number of variables, we indeed compress the sequence of random bits used for resampling, and this happens with exponentially small probability.

This argument finishes the proof. ∎

Here is the idea. We start with an empty string and add randomly chosen letters to its right end. If some forbidden string appears as a suffix, it is immediately deleted. So forbidden strings may appear only as suffixes, and only for a short time. After this “backtracking” we continue adding new letters. (This resembles the famous “tetris game” when blocks fall down and then disappear under some conditions.)

We want to show that if this process is unsuccessful in the sense that after many steps we still have a short string, then the sequence of added random letters is compressible, so this cannot happen always, and therefore a long string without forbidden factors exists. Let us consider a “record” (log file) for this process that is a sequence of symbols “$+$” and “$+\langle\text{deleted string}\rangle$” (for each forbidden string we have a symbol, plus one more symbol without a string). If a letter was added and no forbidden string appears, we just add ‘$+$’ to the record. If we have to delete some forbidden string $s$ after a letter was added, we write this string in brackets after the $+$ sign. Note that we do not record the added letters, only the deleted substrings. (It may happen that several forbidden suffixes appear; in this case we may choose any of them.)

If after many (say, $T$) steps we still have a short current string, then the sequence of random letters can be described by the record (due to the Lemma; we ignore the current string part since it is short). As we will see, the record can be encoded with less bits than it should have been (i.e., less than $T\log m$ bits). Let us describe this encoding and show that it is efficient (assuming the inequality $\sum a_{j}x^{j}<mx-1$).

We use arithmetic encoding for the lengths. Arithmetic encoding for $M$ symbols starts by choosing positive reals $q_{1},\ldots,q_{M}$ such that $q_{1}+\ldots+q_{M}=1$. Then we split the interval $[0,1]$ into parts of length $q_{1},\ldots,q_{M}$ that correspond to these $M$ symbols. Adding a new symbol corresponds to splitting the current interval in the same proportion and choosing the right subinterval. For example, the sequence $(a,b)$ corresponds to $b$th subinterval of $a$th interval; this interval has length $q_{a}q_{b}$. The sequence $(a,b,\ldots,c)$ corresponds to interval of length $q_{a}q_{b}\ldots q_{c}$ and can be reconstructed given any point of this interval (assuming $q_{1},\ldots,q_{M}$ are fixed); to specify some binary fraction in this interval we need at most $-\log(q_{a}q_{b}\ldots q_{c})+O(1)$ bits, i.e., $-\log q_{a}-\log q_{b}-\ldots-\log q_{c}+O(1)$ bits.

Now let us apply this technique to our situation. For $+$ without brackets we use $\log(1/p_{0})$ bits, and for $+\langle s\rangle$ where $s$ is of length $j$, we use $\log(1/p_{j})+\log a_{j}$ bits. Here $p_{j}$ are some positive reals to be chosen later; we need $p_{0}+\sum p_{j}=1$. Indeed, we may split $p_{j}$ into $a_{j}$ equal parts (of size $p_{j}/a_{j}$) and use these parts as $q_{s}$ in the description of arithmetical coding above; splitting adds $\log a_{j}$ to the code length for strings of length $j$.

To bound the total number of bits used for encoding the record, we perform amortised accounting and show that the average number of bits per letter is less than $\log m$. Note that the number of letters is equal to the number of $+$ signs in the record. Each $+$ without brackets increases the length of the string by one letter, and we want to use less that $\log m-c$ bits for its encoding, where $c>0$ is some constant saying how much is saved as a reserve for amortized analysis. And $+\langle s\rangle$ for a string $s$ of length $j$ decreases the length by $j-1$, so we want to use less than $\log m+c(j-1)$ bits (using the reserve).

So we need:

together with

Technically is it easier to use non-strict inequalities in the first two cases and a strict one in the last case (and then increase $p_{i}$ a bit):

Then for a given $c$ we take minimal possible $p_{i}$:

and it remains to show that the sum is less than $1$ for a suitable choice of $c$. Let $x=2^{-c}$, then the inequality can be rewritten as

or

and this is our assumption.

Now we see the role of this mystical $x$ in the condition: it is just a parameter that determines the constant used for the amortised analysis. ∎

Assume that the series in the denominator does not have a root, but the inverse series has all positive coefficients. In fact, non-negative coefficients are enough to get a contradiction. For a series with all non-negative coefficients, or with finitely many negative coefficients, the radius of convergence is determined by behavior of the sum on the real line: when the argument approaches the convergence radius, the sum of the series goes to infinity. Now we have the product of two series

that is equal to $1$. One of these series should have finite convergence radius, otherwise both are everywhere defined and the product is everywhere $1$, but both are large for large $x$. Look at the minimal convergence radius (of two); one of the series goes to infinity near the corresponding point on the real line, so the other one converges to zero, so it has bigger convergence radius and reaches zero at the real line. Finally, note that only the first factor (the denominator) may have a zero, since the other one has all non-negative coefficients.

Now assume that the denominator has a zero; we have to prove that the inverse series has only positive coefficients. In general, the following result is true (D. Piontkovsky): if the series

has $a_{0}>0$, and $a_{2},a_{3},\ldots\geq 0$, and for some positive $x$ this series converges to $0$, then the inverse series has all positive coefficients. To prove this statement, let $\alpha$ be the root, so $A(\alpha)=0$. Recall the long division process that computes the inverse series. It produce the sequence of remainders: the first $R_{0}$ is $1$; then we subtract from the $k$th remainder

the product $(R^{(k)}/a_{0})Ax^{k}$ to cancel the first term, and get the next remainder $R^{(k+1)}$. By induction we prove that for each remainder $R^{(k)}$:

• •

  $R^{(k)}(\alpha)=1$;

• •

  all the coefficients $R^{(k)}_{k+1}$, $R^{(k)}_{k+2},\ldots$, except the first one, are negative or zeros;

• •

  the first coefficient $R^{(k)}_{k}$ is positive.

The first claim is true, since it was true for $R^{(k-1)}$ by induction assumption and we subtract a series that equals zero at $\alpha$.

The second claim: by induction assumption the first coefficient in $R^{(k-1)}$ was positive, so we subtract the series $(R^{(k-1)}/a_{0})Ax^{k-1}$ with positive first and non-negative third, fourth, etc. coefficients. The first term cancels the first term in $R^{(k-1)}$, the second term does not matter now, but all the subsequent coefficients are negative or zeros, since we subtract non-negative coefficients from non-positive ones.

Finally, the third claim is the consequence of the first two: if the sum is positive (equal to $1$) and all the terms except one are non-positive, then the remaining term is positive.

Therefore, all coefficients in the inverse series are positive. ∎