MATHEMATICAL THEORY OF COMMUNICATION 393 



function of ;?. With equally likely events there is more choice, or un- 

 certainty, when there are more possible events. 

 3. If a choice be broken down into two successive choices, the original 

 H should be the weighted sum of the individual values of H. The 

 meaning of this is illustrated in Fig. 6. At the left we have three 

 possibilities pi = h p2 = i, Ps = i- ^^ the right we first choose be- 

 tween two possibilities each with probability J, and if the second occurs 

 make another choice with probabilities §, J. The final results have 

 the same probabilities as before. We require, in this special case, 

 that 



Hih h I) = H{h h) + hH{h \) 

 The coefficient J is because this second choice only occurs half the time. 



.1/6 

 Fig. 6 — Decomposition of a choice from three possibilities. 



In Appendix II, the following result is established: 

 Theorem 2: The only H satisfying the three above assumptions is of the 

 form : 



n 



H = -Kj2pilogPi 



1=1 



where X is a positive constant. 



This theorem, and the assumptions required for its proof, are in no way 

 necessary for the present theory. It is given chiefly to lend a certain plausi- 

 bility to som.e of our later definitions. The real justification of these defi- 

 nitions, however, will reside in their implications. 



Quantities of the form H = —'^^ pi log pi (the constant A' merely amounts 

 to a choice of a unit of measure) play a central role in information theory as 

 measures of information, choice and uncertainty. The form of H will be 

 recognized as that of entropy as defined in certain formulations of statistical 

 mechanics'^ where pi is the probability of a system being in cell i of its phase 

 space. H is then, for example, the H in Boltzmann's famous H theorem. 

 We shall call H = — "Z pi log pi the entropy of the set of probabilities 



^ See, for example, R. C. Tolman, "Principles of Statistical Mechanics," Oxford, 

 Clarendon, 1938. 



