Computing in the Limit

Computing in the Limit started with the pioneering work in Machine Learning of Ray Solomonoff[5][6] and Mark Gold[3]. They developed some initial results on a type of Machine Learning called Identification in the Limit. Identification in the Limit is a learning task where the learner is presented with a sequence of elements from a recursively enumerable set and is given the task to determine an index $i$ for the function $\phi_{i}$ whose domain is the set. Besides its applications in learning, Gold[2] considered the problem of identification of sets in the limit (Limiting Recursive Sets), and Kolmogorov based his complexity theory on work by Ray Solomonoff.

One basic result from Kolmogorov Complexity is the function $K(x)=y$, where Turing machine $TM_{y}$ is the machine with the shortest index $y$ that, starting with an empty tape, halts with $x$ on its tape. The Incompressible Numbers are numbers where, essentially, $K(x)=|x|+c$. This happens when there is no shorter way of generating these numbers except by storing a copy of the number $x$ in the state table of the Turing machine, which is just the size of $x$ plus a constant $c$. These values are known to be immune - no infinite recursively enumerable set is a subset of the Incompressible Numbers.

In this paper, we will consider a more general problem. We cosider the class of functions$f_{i}(x)$ based on all two-input recursive functions $\phi_{i}(x,y)$ where $f_{i}(x)=z$ if $\phi_{i}(x,y)$ converges to $z$ for increasing values of $y$, and try to discover some properties of of the functions in this class. We will show that the class of Computable in the Limit functions has a nice normal form theorem to characterize the classs, similar to the Kleene Normal Form Theorem.

The authors Gold[2], Stephan and Zeugmann[7] and Terwijn[8] have studied this class of functions. Gold shows that functions in this class that are are limiting recursive ($P_{S}(x)=1$ if $x\in S$, otherwise $P_{S}(x)=0$) is 2-recursive and limiting recursively enumerable functions ($P_{S}(x)$ converges iff $x\in S$) is 2-r.e. The term 2-recursive means that the functions are in EA and AE in the arithmetical hierarchy, and 2-r.e. is EA. Stephan and Zeugmann showed that although problems like the Halting Problem and other function classes are learnable in the limit, some ae not reliably learned. Terwijn extends these results by considering learning algorithms where zero, one or two mind changes are allowed and compares them to a function with access to an oracle for the set to be learned.

Just as in the class of computatble functions, there are functions with propertiers that are similar that fall into this class. We will explore some properties that are Computable in the Limit and some that are not.

Using the notion of Rogers[4] we define the Tau function as

Where

and $\pi_{i}^{k}(x)$ is the inverse function for the $i^{th}$ element of $\tau^{k}$.

Without loss of generality, an enumeration of the partial recursive functions is given as $\phi_{i}$ and the primitive recursive functions is given as $F_{i}$. A function with two arguments $\phi_{i}(x,i)$ is simply $\phi_{i}(\tau(x,y))$ and similarly for $F_{i}(x,y)$.

We begin with the definition of Computing in the Limit. This is similar to Gold’s definition of a Limiting Recursive Set.

For example, the set $<x,y>$ of all shortest algorithmic descriptions of Kolmogorov complexity is expressed by such a function $f_{K}$. $f_{K}$ takes the input $x$ and generates $y$ if $y$ is the smallest integer such that $\phi_{y}(0)=x$. This requires a dovetail function $\phi_{K}i(x,t)$ that runs all $s\leq t$ where $\phi_{s}(0)=x$ in time less than $t$ and for all $r<s$ either $\phi_{r}(0)=z\neq x$ or $\phi_{r}(0)$ does not converge in time $t$. If no such $\phi_{s}(0)=x$ is found, return the value $t$.

The notation $f_{i}(x)$ for properties Computable in the Limit looks similar to the standard definition of a partial recursive function $\phi_{i}(s)$, except that partial recursive functions are defined using the primitive recursive functions $\Phi_{i}(x,t)$. Actually we shall see that the only difference is the terminating condition. This allows us to construct a Normal Form Theorem that is similar to Kleene’s Normal Form theorem except that instead of using the mu function $\mu(x)$ to find the first element where a Boolean predicate is true, we use a new function $\lambda(x)$ to find the last of a finite number of guesses.

Kleene’s Normal Form Theorem is given as follows.

The Boolean function $T(e,x,y)$ encodes in the variable $y$ the computation history of Turing machine $TM_{e}$ on input $x$. The predicate $T(e,x,y)$ returns true if $y$ is a halting sequence. The function $U(z)$ recovers the output from the computation history $y$. We shall assume that if $T(e,x,y)$ is true then for all $z>y$ where $z$ is $y$ with one or more copies of the last tape configuration of $y$ appended to $y$, then $T(e,x,z)$ is true also.

Gold shows that limiting primitive recursive, limiting total recursive and limiting partial recursive functions are equally as powerful. This normal form theorem shows why this is so. The use of the lambda function with the primitive recursive functions are powerful enough to compute anything in this class. Since partial recursive functions can be computed with primitive recursive functions and the application of the mu function, the substituion of the mu function by the lambda function gives the same expressive power to a primitive recursive function as to a total or partial recursive function.

Here are a number of properties that are Computable in the Limit but not effectively computable. These are all pretty obvious, so I shall just state them.

Note that Computable in the Limit properties can be l-partial functions, where for $P_{p}(x)$ there are either an infinite number of guesses or no guess at all. Some of the examples here are l-total $P_{p}(x)$ functions. In all of these cases, the class of sequences is $A_{Z}$:

• •

  The Kolmorogorov set $K(x)=P_{K}(x)=y$. The description was given in the Definitions as an example. $P_{K}$ is l-total.

• •

  Kolmogorov incompressible numbers: $P_{I}(n)=x$, where $x$ is incompressible. Use $P_{K}(x)$ and discard any values $y$ that are compressible. When that happens, shift the guesses for $P_{I}(n)$ down. At each value $n$ there comes a time when all of the previous compressible values are found, and $P_{I}(n)$ is never shifted. $P_{I}$ is l-total.

• •

  The set of all partial recursive functions $TM_{x}$ where $P_{P}(x)=\langle x,y\rangle$ only if $TM_{x}$ is partial and $y$ is the smallest value where $TM_{x}(y)$ diverges. The value $P_{P}(x,y)$ is computed as follows: for all $z\leq y$ run $TM_{x}(z)$ for $y$ steps at most. If there is a value $z$ where $TM_{x}(z)$ has not converged in $y$ steps or less, output $\langle x,z\rangle$. If they all converge, output $\langle x,y\rangle$. If $TM_{x}$ is total, then if $\Phi_{x}(y)<y$ for all but a finite $y$, then $P_{P}(x,y)=\langle x,y\rangle$ for all but a finite number of cases and does not l-converge. Otherwise, the output of $P_{P}(x,y)$ changes infinitely often as the smaller inputs converge. In this case, $P_{P}$ is l-partial.

• •

  The set of all partial recursive functions $TM_{x}$ where for every $n$ there is a unique $x$ such that $P_{Q}(n)=x$. Use the same trick as above for the l-total property of Kolmogorov incompressible numbers. In this case, we must keep track of both the function $TM_{a}$ and the value $P_{P}(a)=\langle a,b\rangle$ at which it diverges. Ordering the $\langle a,b\rangle$ pairs, $P_{Q}(n)$ outputs the value $c$, where $\langle c,d\rangle$ is the $n^{th}$ smallest pair. This takes l-partial $P_{P}$ and turns it into l-total $P_{Q}$.

• •

  The set of all partial functions with finite domain. Instead of keeping track of $\langle a,b\rangle$ where $TM_{a}(b)$ diverges, keep track of $\langle a,\langle b_{0},...,b_{n}\rangle\rangle$ where $TM_{a}(b_{i})$ converges, for each $i\leq n$.

• •

  The minimum index for finite sets - similar to the Kolmorogorov set.

• •

  The set of all functions $TM_{i}$ with a single element in its domain.

• •

  Given an enumeration of indexes of Polynomial functions $A$ and enumeration of indexes of NP functions $B$, the Boolean property over pairs $\langle i,j\rangle$ where $P_{eq}(\langle i,j\rangle)=1$ iff $i\in A$, $j\in B$ and $\phi_{i}=\phi_{j}$. Otherwise $P_{eq}(\langle i,j\rangle)=0$. If $i\notin A$ or $j\notin B$, then $P_{eq}(\langle i,j\rangle)$ diverges.

• •

  The property $P_{exp}(i)=\langle i,e\rangle$ where $\phi_{i}$ has polynomial complexity with exponent $e$. If $\phi_{i}$ is not polynomial, then $P_{exp}(i)$ l-diverges.

The Polynomial - NP property can be generalized to any pairs of recursively enumerable classes of total functions. We will present two types of generalizations. Let the class of total functions $A$ be enumerated by a total function $\phi_{a}$ whose range is indexes of total functions in the set $A$, and similarly for $B$ and $\phi_{b}$. A theorem by Blum and Blum on the extrapolation of total recursive functions uses the concept of an h-easy function. They show that each recursively enumerable class of functions is bounded, up to a finite number of exceptions, by the computational complexity of a total function $h$.[1]

This definition differs slightly from Blum and Blum’s in that $\phi_{i}$ can diverge in a finite number of cases.

So the Boolean test of equality for h-easy classes of total recursive functions is Computable in the Limit.

Note, though, that neither the class of Polynomial functions nor the class of NP functions can be defined in this way, because the h-easy function will grow to include all polynomial exponents. This allows for functions whose computation time is not within any exponent. We can, though, enumerate indexes of the polynomial functions of the NP functions, and use them instead of $g$ and $h$.

Note that these two theorems do not work for sets of partial recursive functions, because we can get stuck on a case where $\phi_{i}(x)$ converges and $\phi_{j}(x)$ diverges. The procedure does not identify this case, so it will return equality even though one function diverged.

It is also possible to Compute in the Limit the ratio of equal to unequal values in the two classes, if the ratio is a rational number bounded in the limit.

Note that, although this theorem does not apply to the reals in general, any real number that can be computed as the output of a Turing machine starting with a blank tape (such as $e$ or $\pi$) can be added to this function by using the index of the associated Turing Machine as a possible output.

Given a Computing in the Limit Problem $P_{p}$ where the output is a class of total functions $\phi_{i}$, we can generate a Canonical listing, where every function appears only once.

A Complexity Bound Enumeration finds all of the algorithms that are faster than the one in the Canonical Enumeration.

We end with a couple of examples of properties that are not Computable in the Limit.

We can strengthen this result by showing that no Complete Enumeration of even a single total recursive function is Computable in the Limit by any $P_{p}$, even if it is l-partial.

The class of properties that are Computable in the Limit are an interesting extension of the effectively computable functions. Although they are not computable in a finite time, they model our everyday notion of what learning and generalization are. The restricted subset of problems termed “Identification in the Limit” have been extensively covered in the Machine Learning and Computational Learning Theory fields. But the formal properties of the class itself, outside of its use as a model for learning, is itself interesting. This paper serves as a start for the exploration of this class.

It is obvious that the substitution of $\mu(x)$ for $\lambda(x)$ in the Normal Form Theorem can be further extended to classes where the computing function changes its mind an infinite number of times, and there is some predicate that defines a sort of limiting condition of that sequence. This would make it possible to extend Theorem 4.6 to the reals. This extension would of course be another superset, with characteristics all its own.

I would like to acknowledge Patrick Boyle and David Buhanan for helpful discussions.