MATHEMATICAL THEORY OF COMMUNICATION 395 



while 



H{x) = - Z p{i, j) log Z Pii, j) 



H(y) = - Z /'(^ i) log Z ^(^, i). 



I.; i 



It is easily shown that 



H(x, y) < H{x) + H(y) 



with equality only if the events are independent (i.e., p{i, j) = p{i) p(j)). 

 The uncertainty of a joint event is less than or equal to the sum of the 

 individual uncertainties. 



4. Any change toward equalization of the probabilities pi , p2 , • • • , pn 

 increases H. Thus if pi < p2 and we increase pi , decreasing p2 an equal 

 amount so that pi and p2 are more nearly equal, then H increases. More 

 generally, if we perform any "averaging" operation on the pi of the form 



p'i = Z (J^ijpj 

 i 



where Z <^ii = Z (^a = 1? and all dij > 0, then H increases (except in the 



i j 



special case where this transformation amounts to no more than a permuta- 

 tion of the pj with H of course remaining the same). 



5. Suppose there are two chance events x and 3; as in 3, not necessarily 

 independent. For any particular value i that x can assume there is a con- 

 ditional probability piij) that y has the value j. This is given by 



^'^'^ = i:p(ij)- 



i 



We define the conditional entropy of y, Hx{y) as the average of the entropy 

 of y for each value of x, weighted according to the probability of getting 

 that particular x. That is 



H:c{y) = -Z/>(^',i)log./>i(i). 



This quantity measures how uncertain we are of y on the average when we 

 know X. Substituting the value of pi{j) we obtain 



H.{y) = -Z ^0', i) log p{h j) + Z pi.h j) log Z pa, j) 



= H{x, y) - H{x) 



or 



H{x, y) = H{x) + H,{y) 



