396 BELL SYSTEM TECHNICAL JOURNAL 



Th6 uncertainty (or entropy) of the joint event x, y is the uncertainty of x 

 plus the uncertainty of y when x is known. 

 6. From 3 and 5 we have 



H(x) + H(y) > H(x, y) - H(x) + H .{y) 



Hence 



H(y) > HS) 



The uncertainty of y is never increased by knowledge of x. It will be de- 

 creased unless X and y are independent events, in which case it is not changed. 



7. The Entropy of an Information Source 



Consider a discrete source of the finite state type considered above. 

 For each possible state i there will be a set of probabilities pi{j) of pro- 

 ducing the various possible symbols j. Thus there is an entropy Hi for 

 each state. The entropy of the source will be defined as the average of 

 these Hi weighted in accordance with the probability of occurrence of the 

 states in question: 



H = J^PiHi 



= -llPiPi{j)\0opXj) 



i.j 



This is the entropy of the source per symbol of text. If the Markoff proc- 

 ess is proceeding at a definite time rate there is also an entropy per second 



H'=T.fiHi 



i 



where /t is the average frequency (occurrences per second) of state i. Clearly 



H' = mH 



where m is the average number of symbols produced per second. H or H' 

 measures the amount of information generated by the source per symbol 

 or per second. If the logarithmic base is 2, they will represent bits per 

 symbol or per second. 



If successive symbols are independent then H is simply —^ pi log pi 

 where pi is the probability of symbol i. Suppose in this case we consider a 

 long message of N symbols. It will contain with high probability about 

 piN occurrences of the first symbol, P2N occurrences of the second, etc. 

 Hence the probability of this particular message will be roughly 



p = pp'^fp^'-'-prr'' 



or 



