9/14 AN INTRODUCTION TO CYBERNETICS 



uncertainty", and both measure it by tiie amount of uncertainty it 

 removes. Both further are concerned basically with the gain or 

 increase in information that occurs when a message arrives— the 

 absolute quantities present before or after being of minor interest. 



Now it is clear that when the probabilities are well spread, as in A 

 of Fig. 9/14/1, the uncertainty is greater than when they are compact, 

 as in B. 



ElvenXs Eve.uts> 



Fig. 9/14/1 



So the receipt of a message that makes the recipient revise his 

 estimate, of what will happen, from distribution A to distribution B, 

 contains a positive amount of information. Now ^p log p (where 

 S means "the sum of"), if applied to A, will give a more negative 

 number than if applied to B; both will be negative but A"s will be the 

 larger in absolute value. Thus A might give —20 for the sum and 

 B might give —3. If we use S/» log p multiplied by plus 1 as 

 amount of information to be associated with each distribution, i.e. 

 with each set of probabiUties, then as, in general, 



Gain (of anything) = Final quantity minus initial quantity 



so the gain of information will be 



(-3) -(-20) 



which is + 17, a positive quantity, which is what we want. Thus, 

 looked at from this point of view, which is Wiener's, Y>p\ogp 

 should be multiplied by plus 1, i.e. left unchanged; then we calculate 

 the gain. 



Shannon, however, is concerned throughout his book with the 

 special case in which the received message is known with certainty. 

 So the probabilities are all zero except for a single 1 . Over such a set 

 ^p log/; is just zero; so the final quantity is zero, and the gain of 

 information is 



— (initial quantity). 



In other words, the information in the message, which equals the 

 gain in information, is S/? log/7 calculated over the initial distribu- 

 tion, multiplied by minus 1, which gives Shannon's measure. 



178 



