A Primer on Information Theory 45 



concerning language constraints which can be ehcited from a person familiar 

 with printed English by a carefully planned interrogation. The subject is given 

 a text which is truncated at some point; he is asked to guess the next letter. 

 If he is successful, then he is told to go on; if not, he is told to try again. Records 

 are taken of the number of times a letter is correctly identified at the first, 

 second, third, . . . statement. In this setup, the experimenter acts as source 

 of auxiliary infoimation, emitting sequences of the type 'wrong . . . wrong 

 right', with an 'alphabet' of twenty-six different sequences (if repetitions are 

 excluded, the letter must be identified after no more than twenty-five wrong 

 guesses). The informational output of the auxiliary source depends on the 

 relative probabilities of the various sequences. These probabilities are very 

 unequally distributed. In a large percentage of the cases, the first statement 

 is correct; the most frequent message from the auxiliary source is 'right'. 

 The next highest probability is for the sequence 'wrong-right'. Messages 

 with up to three 'wrongs' make up the vast majority of cases; the remaining 

 categories, with from 4 to 25 'wrongs', have low probabilities. As was pointed 

 out before, they contribute little to the estimated value of H. This means that 

 we arrive at an estimate of the information furnished by the auxiliary source 

 essentially as a function of two to four probabilities. 



The amount of information per single letter is known to be about 4.1 bits 

 (on the basis of relative frequency of letters in English texts). This is the amount 

 of information per letter which the subject needs to reconstruct the whole 

 text. Of this amount of information, a certain measurable fraction is furnished 

 by the auxiliary source. The remainder must come out of the subject's head, 

 and is based on his knowledge of language constraints. The amount of infor- 

 mation so elicited will not be quite as high as the information content of 

 language constraints, but it is a closely related quantity. By the ingenious 

 trick of effectively reducing the size of the alphabet, this quantity has been 

 made easily measurable. 



APPENDIX II 

 ANSWERS TO EXERCISES 



1 . One light — peace and quiet 



two lights, vertically — enemy approaches by land 

 two lights, horizontally — enemy approaches by sea 

 two lights, diagonally — enemy approaches by land and sea 

 (This is not the only possible solution) 



2. (a) 0, 1, 10,11,100,101,110,111, 10000, 1001,1010,1100,10000, 11110100011 



(b) 9, 11, 147,32 



(c) .125, .6703125 



3. (a) 10110100010000 

 (b) EDCBA 



4. 'Construct a confusion-free code using five binary digits for each letter and compare 

 the performance of this code with that of the above by encoding and decoding a message like 

 this one'. 



Use part of the 32 code words made up of 5 binary digits, such as: 1 1 1 1 1 , 1 1 1 10, 1 1 101 , 

 11100, etc. The message will be, on average, 21 per cent longer than with the most efficient 

 code (5 is 121 per cent of 4.14), but it is much easier to decode. Some of the unused code 

 words can be used for punctuation, etc. The teletype works on this principle. 



