9/17 AN INTRODUCTION TO CYBERNETICS 



is producing letters A, B, C, D with frequencies in the ratio of 4, 2, 

 1, 1 respectively, the successive symbols being independent. A 

 typical portion of the sequence would be ...BAABDAAAA 

 BCABAADA.... At equilibrium the relative frequencies of 

 A, B, C, D would be |, I, |, | respectively, and the entropy is \l 

 bits per step (i.e. per letter). 



Now a channel that could produce, at each step, any one of four 

 states without constraint would have a capacity of 2 bits per step. 

 Shannon's theorem says that there must exist a coding that will 

 enable the latter channel (of capacity 2 bits per step) to transmit 

 such a sequence (with entropy If bits per step) so that any long 

 message requires fewer steps in the ratio of 2 to 1|, i.e. of 8 to 7. 

 The coding, devised by Shannon, that achieves this is as follows. 

 First code the message by 



A B C D ' 



''' 10 110 111 



e.g. the message above, 



.B. AAB. D . . AAAAB. C . . AB . AAD. . A 

 MOOOlOlllOOOOlOllOO ^10001110 



Now divide the lower line into pairs and re-code into a new set of 

 letters by 



00 01 10 11 

 ^ E F G H 



These codes convert any message in "A to D" into the letters "£ to 

 //", and conversely, without ambiguity. What is remarkable is 

 that if we take a typical set of eight of the original letters (each 

 represented with its typical frequency) we find that they can be 

 transmitted as seven of the new: 



AAAAB. 

 j 1 

 E . E . G 



thus demonstrating the possibihty of the compression, a compression 

 that was predicted quantitatively by the entropy of the original 

 message! 



Ex. 1 : Show that the coding gives a one-one correspondence between message 

 sent and message received (except for a possible ambiguity in the first letter). 



184 



