466 Information Theory and Biology /25 : 2 



penny which generates the symbols "heads" and "tails." The fre- 

 quency of occurrence of heads for a small number of tosses is random 

 but approaches 1/2 as the number of tosses is increased. 



B. Markhoff Process 



This is a process in which intersymbol influences exist, so that the prob- 

 ability of i following j, p i} -, can be defined. In general, the pi/s are all 

 different. The coin tossing is not a Markhoff process because each result 

 is independent of the last. In contrast, the probability of one English 

 letter following another is measurable. A process generating English 

 letters in words, for example, writing, is a Markhoff process. 



C. Ergodic Sequence 



This is a sequence of symbols in which the intersymbol influence falls 

 off exponentially or disappears after a finite number of symbols. In 

 English, the probability of a given letter following the first one is not 

 random. Nor is the probability of a second or third following letter 

 determined at random. Definite intersymbol influences can be found 

 out to eight letters. Thereafter, the probability is essentially random. 

 Thus, English letters form ergodic sequences. 



(The word "ergodic" comes from the Greek; literally, it means 

 energy pathway. Its relationship to the foregoing is not trivially 

 obvious. Students of statistical mechanics will recognize that the fore- 

 going definition in terms of intersymbol influences is synonymous with 

 the definition in terms of energy pathways as used in Gibbs' statistics. 

 Information theory borrowed this word from statistical mechanics.) 



D. Redundancy 



If intersymbol influences exist, not all the symbols are necessary. A 

 different coding could reduce the number of symbols. Redundancy is 

 desirable in that it tends to increase the signal to noise ratio. 



One example of redundancy is the written English language. The 

 average information per letter H 1 has been computed, including various 

 intersymbol influences. These are given in Table IV, which shows that 

 a redundancy of about 1 bit, that is, twofold, exists. 



TABLE IV 



Average Information of English Letters 



Letters Hi 



Random 4.7 bits 



English frequency 4.15 bits 

 Intersymbol influences 



for 2 letters 3.57 bits 

 Intersymbol influences 



for 8 letters 3.25 bits 



