Statistical distributions and entropy considerations in gene codes

In last years there appeared a possibility to provide some knowledge of the genome sequence data in many organisms. The genome was studied intensively by number of different methods Holste et al. (2003); Li and Kaneko (1992); Mantegna et al. (1994, 1995); Messer et al. (2005); Peng et al. (1992, 1994); de Sousa Vieira (1999); Stanley et al. (1999); Voss (1992). The statistical analysis of DNA is complicated because of its complex structure; it consists of coding and non coding regions, repetitive elements, etc., which have an influence on the local sequence decomposition. Long range correlations in sequence compositions of DNA are much more complex than simple power law; moreover effective exponents of scale regions are varying considerably between different species Holste et al. (2003), Messer et al. (2005). In papers Mantegna et al. (1994, 1995) the Zipf approach to analyzing linguistic texts has been extended to statistical study of DNA base pair sequences.

In our paper we take into account some linguistic features of genomes to study the statistics of gene codes. One of the fundamental observations based on the information theory says that if the system is described by special distribution of hyperbolic type it implies a possibility of entropy loss. Distributions of hyperbolic type describe also property of stability and flexibility which explain that the languages can develop without changing its basic structure (see Harremoës and Topsøe (2001)). In the experimental part of our research we show that a similar situation can occur in a genome sequence data which carries the genetic information.

To demonstrate analogies between coding of genes and representation of words in human languages we generated histograms of the sizes of groups of words of the same length in selected different European languages as well as histograms of the sizes of groups of genes of the same length in a genome. Three genomes were selected for evaluation, that is, two bacteria: Borrelia burgdorferi and Escherischia coli, and a yeast: Saccharomyces cerevisiae S288c. The gene data were obtained from NCBI GeneBank Gen . Each of the histograms was successfully approximated by Asymmetric Normal Inverse Gaussian Distribution.

Then, for each of the three genomes histograms of the sizes of groups of identical genes were generated. We showed that each of them can be modelled by distributions of hyperbolic type defined in the paper.

Additionally, the histograms for sizes of groups of identical fragments of genes of different sizes were calculated. In this case any two genes are regarded as similar when there exist a gene fragment, that is, an ordered sequence of symbols of size $n$ which can be found in both genes. Obtained results confirmed our hypothesis that the basic structure of genome remain stabile and entropy loss is possible.

The paper is organized as follows. In Sect. II we begin with a brief presentation of some common features of linguistics and genetics. Then, in Sect. III we describe Code Length Game, Maximal Entropy Principle and the theorem about hyperbolic type distributions and entropy loss. Sect. IV contains the experimental research concerned analysis of statistical properties of gene strings in three selected genomes and consists of three parts. Histograms of gene lengths are presented in the first part. Evaluation of the frequency of genes in a genome is the subject of the second part, that is, the results of searching for replicated genes in three genomes are presented. In the third, last part we observe frequency of appearance of gene fragments in a genome for different lengths of the fragments. Sect. V concludes the paper.

A language is characterized by an alphabet with letters, like for the case of latin languages: a, b, c, etc. The words are denoted as sequences of $n$ letters. In quantum physics the analogy to letters can be attached to pure states, and texts correspond to mixed general states. Languages can have very different alphabets, for example: computers — 0,1 (two bits), English language — 27 letters with space, or DNA — four nitric bases: G(guanine), A(adenine), C(cytosine), T(thymine). The collection of letters can be ordered or disordered. To quantify the disorder of different collections of the letters we use an entropy

where $p_{i}$ denotes probability of occurrence of the $i$-th letter. If we take the base of logarithm $2$ this will lead to the entropy measured in bits. When all letters have the same probability in all states obviously the entropy has maximum value $H_{max}$.

A real entropy has lower value $H_{eff}$, because in a real languages the letters have not the same probability of appearance. Redundancy $R$ of language is defined Shannon (1948) as follows:

The quantity $H_{max}$  - $H_{eff}$ is called an information. Information depends on difference between the maximum entropy and the actual entropy. The bigger actual entropy means the smaller redundancy. Redundancy can be measured by values of the frequencies with which different letters occur in one or more texts. Redundancy $R$ denotes the number that if we remove the part $R$ of the letters determined by redundancy, the meaning of the text will be still understood.

In English some letters occur more frequently than other and similarly in DNA of vertebrates the frequency of nitric bases C and G pairs is usually less than the frequency of A and T pairs. The low value of redundancy allows in easier way to fight transcription errors in a gene code. In the papers Mantegna et al. (1994, 1995) it is demonstrated that non coding regions of eukaryotes display a smaller entropy and larger redundancy than coding regions.

In this section we provide some mathematics from the information theory which will be helpful in the quantitative formulation of our approach, for details see Harremoës and Topsøe (2001).

Let $A$ be the $alphabet$ which is a discrete finite or countable infinite set. Let $M_{+}^{1}$ and ${}^{\sim}M_{+}^{1}(A)$ are respectively, the set of probability measures on $A$ and the set of non-negative measures $P$, such that $P(A)\leq 1$. The elements in $A$ can be thought as letters. By $K(A)$ we denote the set of mappings, compact codes, $k\,:A\,\rightarrow[0,\infty]$, which satisfy Kraft’s equality Cover and Thomas (1991):

By ${}^{\sim}K(A)$ we denote the set of all mappings, general codes, $k\,:\,A\rightarrow[0,\infty]$, which satisfy Kraft’s inequality Cover and Thomas (1991):

For $k\in^{\sim}K(A)$ and $i\in A$, $k_{i}$ is the code length, for example the length of the word. For $k\in$ ${}^{\sim}K(A)$ and $P\in M_{+}^{1}(A)$ the average code length is defined as

There is bijective correspondence between $p_{i}$ and $k_{i}$

For $P\in M_{+}^{1}(A)$ we also introduce the entropy

The entropy can be represented as minimal average code length, (see Harremoës and Topsøe (2001)):

Let ${\cal P}\subseteq{\cal M}_{+}^{1}(A)$ than

The formula (7) presents the Nash optimal strategies. ${\cal R}_{min}$ denotes $minimum$ $risk$, $k$ denotes the Nash equilibrium code, $P$ denotes probability. For example, in a linguistics the listener is a minimizer, speaker is a maximizer. We have words with distributions $p_{i}$ and their codes $k_{i}$, $i=1,2,...$. The listener chooses codes $k_{i}$,the speaker chooses probability distributions $p_{i}$.
We can notice that Zipf argued Zipf (1949) that in the development of a language vocabulary balance is reached as a result of two opposing forces: $unification$ which tends to reduce the vocabulary and corresponds to a principle of least effort, seen from point of view of speaker and diversification connected with the listeners wish to know meaning of speech.

The principle (7) is so basic as Maximum Entropy Principle has a sense that search for one type of optimal strategy called as Code Length Game translates directly into a search for distributions with maximum entropy. It is a given a code $k\in\,^{\sim}K(A)$ and distribution $P\in M^{1}_{+}(A)$. Optimal strategy according $H(P)=inf_{k\in\,^{\sim}K}<k,P>$ is represented by entropy $H(P)$, where actual strategy is represented by $<k,P>$.

Zipf’s law is an empirical observation which relates rank and frequency of words in natural language Zipf (1949). This law suggests modeling by distributions of hyperbolic type Harremoës and Topsøe (2001), because no distributions over $N$ have probabilities proportional to $1/i$, due to the lack of normalization condition.
We consider a class of distributions $P=(p_{1},p_{2},...)$ over $N$. If $p_{1}\geq,p_{2}\geq...$, $P$ is said to be hyperbolic if for any given $\alpha>1$, $p_{i}\geq i^{-\alpha}$ for infinitely many indexes $i$. As an example we can choose $p_{i}\sim i^{-1}(\log(i))^{-c}$ for some constant $c>2$.

The Code Length Game for model ${\cal P}\in M_{+}^{1}(N)$ with codes $k:A\rightarrow[0,\infty]$ for which $\sum\limits_{i\in A}\exp(-k_{i})=1$, is in equilibrium if and only if $H_{max}(co({\cal P})=H_{max}({\cal P})$. In such a case a distribution $P^{*}$ is the $H_{max}$ attractor such that $P_{n}\rightarrow P^{*}$, if for every sequence $(P_{n})_{n>1}\subseteq\cal P$ for which $H(P_{n})\rightarrow H_{max}(\cal P)$. One expects that $H(P^{*})=H_{max}(\cal P)$ but in the case with entropy loss we have $H(P^{*})<H_{max}(\cal P)$. Such possibility appears when distributions are hyperbolic. It follows from theorem in Harremoës and Topsøe (2001).

Theorem. Assume that $P^{*}\in M_{+}^{1}(N)$ is of finite entropy and it has ordered point probabilities. The necessary and sufficient condition that $P^{*}$ can occur as $H_{max}$–attractor in a model with entropy loss, is that $P^{*}$ is hyperbolic. If this condition is fulfilled than for every $h$ with $H(P^{*})<h<\infty$, there exists a model ${\cal P}={\cal P}_{h}$ with $P^{*}$ as $H_{max}$–attractor and $H_{max}({\cal P}_{h})=h$. In fact, ${\cal P}_{h}=(P|<k^{*},P>\leq h)$ is a largest model. $k^{*}$ denotes the code adopted to $P^{*}$, that is, $k^{*}=-\ln(p^{*})$, $i>1$.

As an example we can consider ”an ideal” language where the frequencies of words are described by hyperbolic distribution $P^{*}$ with a finite entropy. At a certain stage of life one is able to communicate at reasonably height rate about $H(P^{*})$ and improve language by introduction of specialized words, which occur seldom in a language as a whole. This process can be continued during the rest of life. One is able to express complicated ideas develop language without changing a basic structure of the language. We can expect similar situation in gene strings, which also carry an information.

In this paper we follow the statistical description of genes in three selected genomes aimed at finding arguments supporting thesis that the entropy loss in gene ”language” is possible and the basic structure of genomes remain stabile. We show that histograms of gene length and word length are similar and both can be modeled by Asymmetric Inverse Gaussian distributions. Additionally, tails of distributions for gene lengths appear to be of the power type. We test selected genomes and describe the gene code statistics dealing with the number of repetitions and replicated gene fragments in genomes. Considering that distributions of hyperbolic type describe property of stability and flexibility, we show that histograms of identical genes interpreted as probabilities of gene appearance in the genome are of hyperbolic type in the sense of definition $p_{i}\geq i^{-\alpha}$, and show example values of $\alpha$ for each of the genomes. We show also regularity and similarity of histograms with numbers of replicated genes and replicated gene fragments. All the obtained results confirm our thesis.