408 BELL SYSTEM TECHNICAL JOURNAL 



In the example considered above, if a is received the a postericri prob- 

 abiHty that a was transmitted is .99, and that a 1 was transmitted is 

 .01. These figures are reversed if a 1 is received. Hence 



; Hy{x) = - f.99 log .99 + 0.01 log 0.01] 



= .081 bits/symbol 



or 81 bits per second. We may say that the system is transmitting at a rate 

 1000 — 81 = 919 bits per second. In the extreme case where a is equally 

 likely to be received as a or 1 and similarly for 1, the a posteriori proba- 

 bilities are J, J and 



Hy{x) = - [^ log i + I log i] 



= 1 bit per symbol 



or 1000 bits per second. The rate of transmission is then as it should 

 be. 



The following theorem gives a direct intuitive interpretation of the 

 equivocation and also serves to justify it as the unique appropriate measure. 

 We consider a communication system and an observer (or auxiliary device) 

 who can see both what is sent and what is recovered (with erro'.s 

 due to noise) . This observer notes the errors in the recovered message and 

 transmits data to the receiving point over a ''correction channel" to enable 

 the receiver to correct the errors. The situation is indicated schematically 

 in Fig. 8. 



Theorem 10: If the correction channel has a capacity equal to Hy{x) it is 

 possible to so encode the correction data as to send it over this channel 

 and correct all but an arbitrarily small fraction e of the errors. This is not 

 possible if the channel capacity is less than Hy{x). 



Roughly then, Hy{x) is the amount of additional information that must be 

 supplied per second at the receiving point to correct the received message. 



To prove the first part, consider long sequences of received message M' 

 and corresponding original message M. There will be logarithmically 

 THy(x) of the M's which could reasonably have produced each M'. Thus 

 we have THy{x) binary digits to send each T seconds. This can be done 

 with € frequency of errors on a channel of capacity Hy{x). 



The second part can be proved by noting, first, that for any discrete chance 

 variables x^y^z 



Hy(x, z) > Hyix) 



The left-hand side can be expanded to give 



Hy,{x) > Hy{x) - Hy{z) > Hyix) - //(:) 



