The transmission sense of information

Inspired by Dretske Dretske (1983), several authors Griffiths and Gray (1994); Sterelny and Griffiths (1999); Griffiths (2001); Godfrey-Smith (2008) have explored Shannon theory as a grounding for information language in biology. They derive roughly the following picture: The key currency in information theory is entropy $H(X)$ and the key statistic is mutual information $I(X;Y)$, where $X$ and $Y$ are random variables. Information is conveyed in Grice’s sense of natural meaning Grice (1957): whenever $Y$ is correlated with $X$, we can say that $Y$ carries information about $X$. There is no deep notion of meaning or coding here. “[W]hen a biologist introduces information in this sense to a description of gene action or other processes, she is not introducing some new and special kind of relation or property”, Godfrey Smith writes, “She is just adopting a particular quantitative framework for describing ordinary correlations.” Godfrey-Smith (2008). In this causal sense of information, genes carry information about phenotypes just as smoke carries information about fire, nothing more. If this is all that biologists are doing when they talk about how the genotype carries information about phenotype, this is a shallow use of the information concept!

Not only is this sense shallow, it fails to capture the directional flow of information from genotype to phenotype that is the central dogma of molecular biology Crick (1970). If by “G has information about P” we mean only that $I(G;P)>0$, then we are not acknowleding the direction of information from genotype to phenotype (Griffiths, 2001; Godfrey-Smith, 2000b, 2008). The reason is that the mutual information $I$ is by definition a symmetric quantity; $I(G;P)=I(P;G)$. Mathematically, the amount of information that knowing the genotype $G$ provides about the phenotype $P$ is always exactly equal to the amount of information that knowing the phenotype $P$ provides about the genotype $G$ (see Fig. 1).

Worse still, this notion of mutual information fails to capture the sense of privilege that biologists tend to ascribe to the informational molecule DNA over other contributors to phenotype. So far as causal covariance is concerned, both genes $G$ and environment $E$ influence phenotype $P$ — and in principle we can equally well compute either $I(G;P)$ and $I(E;P)$. So it seems that Shannon theory has no way of singling out DNA as an information-bearing entity.

This criticism is formalized as the parity thesis, and is crafted around an important result in information theory that the roles of source and channel conditions are exchangeable (Griffiths and Gray, 1994). Typically when one sits in front of the television, the football broadcast is the signal. The weather, a crow landing on the television antenna, interference from a neighbor’s microwave — these are sources of noise, the channel conditions for our transmission. But a television repairman has an opposite view. He doesn’t want to watch the game, he wants to learn about what is altering the transmission from the station. So he tunes your set to a station broadcasting a test pattern. For the repairman, this test pattern provides channel conditions to read off the signal of how the transmission is being altered. As Sterelny and Griffiths point out (1999), “The sender/channel distinction is a fact about our interests, not a fact about the physical world.” The parity thesis applies this logic to genes and environment. In the parity view, whether it is genome or environment that carries information must be a fact about our interests, not a fact about the world.

The problem with these arguments is that they adopt a few tools from Shannon theory, but neglect its raison d’être: the underlying decision problem of how to package information for transport. Before delving deeper into Shannon theory, we will take a brief detour to summarize the semantic sense of information in biology.

In addition to the limitations enumerated above, the causal view fails to highlight the intentional, representational nature of genes and other biological objects Griffiths and Gray (1994); Sterelny and Griffiths (1999); Shea (2007); Godfrey-Smith (2008). When biologists talk about genes as informational molecules, this argument goes, it is not because they are correlated with other things (e.g. amino acid sequence or phenotype), but rather because they represent other things. This semantic sense of information in which “genes semantically specify their normal products” Godfrey-Smith (2007) cannot be captured using Shannon theory, which is by design silent on semantic matters.

But what is it that genes are supposed to represent? Much of the conventional language of molecular biology suggests phenotypes as an obvious candidate, and this is the approach that Maynard Smith takes in a target article that triggered much of the recent debate over the information concept in biology Maynard Smith (2000). At first glance, this view has several things to recommend it. A semantic notion of genetic information captures the directionality discussed above: genes represent phenotypes but phenotypes do not represent genes. One could also try to argue that the semantic view privileges genes in that we can say that genes have a representational message about phenotype but environment does not. Finally, it allows for misrepresentation or false representation, whereas causal information does not Griffiths (2001)¹¹1We think that Stegmann handles the misrepresentation issue even more cleanly with a shallow semantic notion of genes as conveying instructional, as opposed to representational, content Stegmann (2004).

Does this mean that the problem is solved? No — Griffiths (2001) and Godfrey-Smith (2008) argue that semantic information remains vulnerable to the parity thesis. Moreover, Godfrey-Smith (1999,2008) and Griffiths (2001) note that the reach of the semantic information concept within genetics is very shallow: legitimate talk of semantic representation can go no further than genes coding for amino acids. Beyond this point, the mapping from genotype forward is context-dependent and hopelessly entangled in a mass of causal interactions. Thus, these authors conclude, the relation from genes to phenotype cannot be a representational one. Accordingly, it seems as if the semantic view of information has been pushed as far as it will go, and yet we are left without a fully satisfactory account of the information concept in biology. Let us therefore return to Shannon’s information theory, but move beyond the causal sense.

By viewing Shannon information as a result of a decision process instead of as a correlation measure, we can resolve the concern that Godfrey-Smith raises about the bidirectional flow of information in Shannon theory. Godfrey-Smith dismisses the causal sense of information because “information in the Shannon sense ‘flows’ in both directions, as it involves no more than learning about the state of one variable by attending to another” Godfrey-Smith (2008). Here Godfrey-Smith is referring to the symmetry of the mutual information measure: $I(X;Y)=I(Y;X)$ (Figure 1).

What is the mutual information actually measuring when we apply this equality in a communication context? An example helps. Peter and Paul are concerned about the state of the world $W$. Suppose that Peter observes a correlate $X$ of the random variable $W$. We want him to communicate his observation to Paul, and he does so using a signal $Y$. The mutual information $I(X;Y)$ tells us how effectively he conveys what he sees, on average, given the statistical distribution of possible $X$ values, the properties of the channel across which the signal is sent, etc. Specifically, $I(X;Y)=H(X)-H(X|Y)$ measures how much Paul learns by knowing $Y$ about what Peter saw, $X$, again on average. Because $I(X;Y)=H(X)-H(X|Y)=H(X)+H(Y)-H(X,Y)=H(Y)-H(Y|X)=I(Y;X)$, Peter knows exactly as much about what Paul learns as Paul learns about what Peter saw. But $I(Y;X)$ is usually irrelevant when we think about the decision problem of communicating. In this context we want Peter to get a message about the world to Paul, and we rarely care how much Peter knows afterwards about what Paul has learned.

This directionality is manifested within Shannon theory by the data processing inequality (Figure 3) Cover and Thomas (2006); Yeung (2002). This theorem states that the act of processing data, deterministically or stochastically, can never generate additional information. A corollary pertains to communication: along a communication chain, information about the original source can be lost but never gained. In the scenario described above, both the observation step $W\rightarrow X$ and the communication step $X\rightarrow Y$ are steps in a Markov chain. For any Markov chain $W\rightarrow X\rightarrow Y$, the data processing inequality states that $I(W;X)\geq I(W;Y)$. In our example, the data processing inequality reveals that communication between Peter and Paul is not symmetric. Paul may know as much about what Peter sent as Peter knows about what Paul received, but Paul does not in general know as much about what matters — the state of the world — as Peter does. Shannon theory is not symmetric with respect to the direction of communication.

In the introduction to this paper, we described the parity thesis. While there is a good case to be made for parity between genes and environment when we restrict our view to the horizontal development of phenotype from genotype, that parity is shattered when we look along the vertical axis of intergenerational transmission.

Look at Figure 2, which corresponds to a neo-Darwinian view of evolution. In this model of the biological world, the transmission concept cleanly separates genes from environments. The former are transmitted across generations, the latter are not. Moreover, taking a teleofunctional view as Sterelny et al. (1996) do for replicators in general, the hypothesis that genes are for transmission across generations is richly supported by the physical structure of the DNA. Genes are made out of DNA, a molecule that is exquisitely fashioned so as to (1) encode lots of sequence information in small space, (2) be incredibly easy to replicate, (3) be arbitrarily and infinitely extensible in what it can say, and (4) be structurally very stable and inert Lewontin (1992). In fact, DNA is perhaps the most impressive known substance with respect to (1) and (2). No machine can look at a protein and run off a copy; DNA is exquisitely adapted so that a relatively simple machine, the DNA polymerase, does this at great speed and high fidelity. Think about how amazing it is that PCR works. It is as if you could throw a hard drive in a water bath with a few enzymes and a few raw materials, run the temperature through a few cycles, and pull out millions of identical hard drives. DNA practically screams “I am for storage and transmission!”

One might object that Figure 2 conveys an over-simplified view of the world. This is true. A more sophisticated view of the evolutionary process allows for additional channels of intergenerational transmission and information flow: environments can be constructed and inherited Odling-Smee et al. (2003). Non-genetic biological structures such as membranes and centrosomes are inherited Griffiths and Knight (1998) and can even be argued to carry some information Griesemer (2005). Methylation provides an extensive layer of markup on top of nucleic acid sequence. Developmental switches actively transduce environmental information into epigenetically heritable forms Griffiths (2001).

But such an objection misses our point. Our aim with the transmission sense of information is not to single out uniquely the genes as having some special property which we deny to all other biological structures, but rather to identify those components of biological systems that have information storage, transmission, and retrieval as their primary function. Methylation tags are obvious members of this information-bearing class: they carry information across generations in the transmission sense and this appears to be their primary function. Extrinsic features of the environment such as ambient temperature are obviously not members of this class: they are not transmitted across generations, they carry information only in the causal sense and information transmission is not their role under any reasonable teleofunctional explanation. Biological structures such as membranes and centrosomes may appear as some sort of middle ground, but we note that (1) their primary function is not an informational one and (2) their bandwidth is extremely restricted compared to that of DNA sequence. Birds’ nests Sterelny et al. (1996) could be seen as an environmental analogue to these intracellular structures, whereas libraries start to push toward genes and methylation tags in their informational capacity. Developmental switches Griffiths (2001) are another interesting case; these transduce environmental information but in addition to their bandwidth limitations they appear to have a more limited intergenerational transmission role. Genes may not be unique in their ability to convey information across generations — but at the same time a transmission view makes it clear that not all components of the developmental matrix Griffiths and Gray (1994); Griffiths (2001) enjoy parity in their informational capacities.

The parity thesis typically is linked to Shannon’s information theory via the claim that “The source/channel distinction is imposed on a natural causal system by the observer.” (Griffiths 2001, p.398) What is signal, and what is noise — Sterelny and Griffiths (1999) take this to be merely a reflection of our interests. So must we impose our own notions of what makes an appropriate reference frame in order to single out certain components of the developmental matrix as signal and others as noise? No. We are not the ones who pick the reference frame, natural selection does. Because natural selection operates on heritable variation, it acts upon some components of the developmental matrix differently than others. For a biologist trying to understand life, the source-channel distinction is imposed not by the observer but rather by the process of natural selection from which life arose and diversified.

To better understand the role of natural selection it helps to expand Figure 2 somewhat. (For simplicity we retain our focus on the genes as transmitted elements, but one could extend this to include other heritable structures). In Figure 2, we highlight the fact that it is the genes, and not the environment, which is transmitted from generation to generation. In Figure 4 we highlight the fact that not all genes are transmitted to the next generation. It is by the means of variation in the genes and selection on the phenotypes with which they are correlated that information can built up in the genome over time Felsenstein (1971).

One motivation for replacing causal sense views of information with semantic-sense views is that the causal sense of information appears to cast too broad of a net. Any physical system with correlations among its components carries causal information — but in their use of the information concept, biologists appear to mean something stronger than the notion of natural meaning that has smoke in the sky carrying information about a fire below Godfrey-Smith (2008). If we substitute a transmission view of information for a semantic view, will we be driven back to this overly-broad notion of information? Not at all. Like naturalized views of semantics, the transmission notion of information rests upon function: to say that X carries information, we require that the function of X be to reduce uncertainty on the part of a receiver.

The failure to consider function when talking about information sometimes generates confusion among practicing biologists. After all, there are correlations everywhere in biological systems, measuring them is what we do as biologists, and we often talk about these correlations as information. This language is understandable; indeed these correlations provide us with information about biological systems. But this is merely causal sense information Godfrey-Smith (1999). As Godfrey-Smith explains, when a systematist uses gene sequences to make inferences about population history, “there is no more to this kind of information than correlation or ‘natural meaning’; the genes are not trying to tell us about their past.” Godfrey-Smith (1999). In other words, these correlations do not convey teleosemantic information. Nor can these correlations be considered information in the transmission sense.

To expand upon this distinction, an example from population genetics is helpful. Voight and colleagues (Voight et al., 2006) developed a method for inferring positive selection at polymorphic loci in the human genome. Their key insight is that with enough sequence data from sufficiently many members of the population, we can pick out regions of the genome that have unusually long haplotypes of low diversity. Such extended haplotype blocks tend to surround an allele that has recently risen in frequency due to strong selection, because there has not been enough time for recombination to break down the association between the favored allele and the genetic background in which it arose. Using this method, Voight and colleagues find strong evidence for recent selection among Europeans in the lactase gene LCT, which is important for metabolism of lactose beyond early childhood. The favored allele results from a single nucleotide change 14 kb upstream of the lactase gene on the nearly 1 Mb haplotype. Positive selection has presumably occurred because the ability to process lactase throughout life became advantageous with the invention of animal agriculture approximately 10,000 years ago.

What does this have to do with information? There is information about the history of selection in the population-level correlations. Voight et al. found an extended haplotype length surrounding the LCT+ allele relative to that around the LCT- allele. But notice that we have to observe the genotypes of multiple individuals in order to determine that one allele at the LCT locus is surrounded by longer haplotype blocks than is the other. Once we have made observations of multiple genomes, we as external observers can conclude something about the history of selection on the population. But this information is not available at the level of a single individual. A single individual cannot look at its own genome and notice a longer (or shorter) haplotype block around any given focal locus — these haplotype blocks are defined with respect to the genotypes of others in the population. A single individual can only look at its own genome and see a sequence of base pairs. This sequence of base pairs is what is transmitted, it is what has the function of reducing uncertainty on the part of the agent who observes it⁴⁴4Although causal-sense information is transmitted from the population at time t to the population at time t+1 in the population frequencies of haplotypes, this is not transmission-sense information because the function of these population-level haplotype assemblages is not to reduce uncertainty on the part of future populations.. These individual gene sequences are the entities that have an informational function in biology (though bioinformaticians have not always recognized this distinction Adami (2002)). The population-level statistics that geneticists use to infer history are informative but they are not information in the transmission sense. They are merely the smoke that is cast off by the fire of natural selection.

The transmission sense of information allows us to separate claims about how information is transmitted from claims about what information means⁵⁵5The source-channel separation theorem (Cover and Thomas 2006, Chapter 7, p.218) proves that in any physical communication system for error-free transmission over a noisy channel, one can entirely decouple the process of tuning the code to the nature of the specific channel from not only the semantic reference of the signal but indeed from all statistics of the message source. This follows because the theorem states that one can achieve channel capacity with separate source and channel coders — and in this setup the source coder can always be configured so as to return output that maximizes the entropy given the symbol set.. Indeed, we can study how information is transmitted without having any knowledge of the “codebook” for how to interpret the message, or even what the information represents.

In many biological studies, we are in exactly this position. Again an example — this time drawn from neurobiology — is helpful. In a study of the fly visual system, de Ruyter van Steveninck and colleagues presented flies with a moving grating as a visual stimulus, and made single-cell recordings of the spike train from the H1 visual neuron de Ruyter van Steveninck et al. (1997). This neuron is sensitive to movement, but it is not known how movement information is encoded into the spike train, nor even what aspects of movement are being represented. Nonetheless de Ruyter van Steveninck and colleagues were able to determine how much information this neuron is able to encode. The investigators exposed a fly to the stimulus, and measured the (average) entropy of the spike train. This is the so-called total entropy for the neuron’s output. They then looked at what happens if you play the same stimulus back repeatedly: how much does the resultant spike train vary from previous trials? This is the so-called noise entropy. The information that the spike train carries about the stimulus, i.e., the mutual information between spike train and stimulus, is simply the difference of these two quantities. Using this approach, the authors were able to show that this single insect neuron conveys approximately 80 bits of information per second about a dynamic stimulus. Thus an individual visual neuron achieves a bit-rate that is roughly 7 times the bit rate of a skilled touch typist⁶⁶6Using Shannon’s 1950 upper bound on bits per letter and his estimate of letters per word in the English language Shannon (1950), we can estimate the bit rate of a touch typist as $\frac{120\text{ words}}{\text{ minute}}\frac{1\text{ minute}}{\text{60 seconds}}\frac{4.5\text{ letters}}{\text{ word}}\frac{1.3\text{ bits}}{\text{letter}}=11.7\text{ bits/second}$. ! More importantly, the researchers were able to compare the response of this neuron to static stimuli with the response of the neuron to natural patterns of motion. They found that for natural patterns the neuron is able to attain the high bandwidth that it does by “establishing precise temporal relations between individual action potentials and events in the sensory stimulus”. By doing so, the neuron’s response to the dynamic stimulus greatly surpasses the bit rate that could be obtained if the stimulus was encoded by a simple matching of spike rate to stimulus intensity. Subsequent investigators have used related methods to show that evolved sensory systems are tuned to natural stimuli, to study the properties of neural adaptation and history dependence, and to examine temporal sensitivity — all without knowing the way in which the signals that they study are actually encoded.

de Ruyter van Steveninck et al. sum up the power of being able to study information without appeal to semantics: “This characterization of … information transmission is independent of any assumptions about [or knowledge of!] which features of the stimulus are being encoded or about which features of the spike train are most important in the code” de Ruyter van Steveninck et al. (1997)

This brings us back to Shannon theory. When information theorists think about coding, they are not thinking about semantic properties. All of the semantic properties are stuffed into the codebook, the interface between source structure and channel structure, which to information theorists is as interesting as a phonebook is to sociologists. When an information theorist says “Tell me how data stream A codes for message set B”, she is not asking you to read her the codebook. She is asking you to tell her about compression, channel capacity, distortion structure, redundancy, and so forth. That is, she wants to know how the structure of the code reflects the statistical properties of the data source and the channel, with respect to the decision problem of effectively packaging information for transport.

With these things in focus, we can now look at the concept of arbitrariness, what it means, and why this concept is critically important in biological coding.