EFFICIENT CODING 731 



tastically large and complete statistical encoding becomes an economic 

 impossibility if not a technical one. 



Let B'I be a particular combination (the /") of k symbols in the i)ast 

 of the message. Each of these combinations at least partially determines 

 the state of the system. TIence we can write an approximation to (7): 



F,= -T. pili'i) Z pAj) log pu){j) (8) 



i J 



Fk -^ JJ, as k -^ 00 . If only m symbols in the past influence the present 

 state, then k need only be as great as //(, in order that F„» = //. In any 

 case the setiuence F\ , Fo , • • • I'\ is monotone decreasing. Naturally 

 one should always pick the A' symbols in the past which exert the great- 

 est effect upon the present state, i.e. which cause Pb\{J) to be as highly 

 peaked as possible, on the average. In English these would be the im- 

 mediately previous letters; in television, the picture elements in the im- 

 mediate space-time vicinity of the present element. 



Suppose we break the message up into blocks of length k. Each of these 

 l)locks may be considered to be a character in a new (and huge) alpha- 

 l)ct. If we ignore any influences from previous blocks, i.e. if we consider 

 the blocks to be independent, then the information per block will be 

 simply 



-ZyXi^i)logp(B-). (9) 



i 



Since there are A- symbols per block, the information per symbol, Gk is 



Gk= - iZpiB'i) log p{B\). (10) 



/v i 



As A; — > 00 , Ga — > //, since the amount of statistical influence ignored 

 (between blocks) becomes negligible compared with that taken into 

 account. 



If d is the number of binary digits refiuired to specify a message n 

 symbols long, then as n — ^ ^, d/nH — > 1. For large n there are thus 

 2"" messages which are at all likely out of 2""° = T possible sequences 

 (in an f letter alphabet). The probability that a purely random source 

 will produce a message (i.e., a sequence with all the proper statistics) is 

 therefore 



p ^ 2""^"°""^ (11) 



for large ?i. Even ii Ho — H is small, p -^ rapidly for large n. This is 

 why white noise never produces anything resembling a picture on a 

 television screen, for instance. For in television signals. Ho — H > 1 



