What is Iiijormatwn Theory? 15 



500 bits per second while actually no information was being 

 transmitted at all. Equally good transmission would be obtained 

 by dispensing with the channel entirely and flipping a coin at 

 the receiving end. 



The proper correction to apply to the amount of information 

 transmitted is the uncertainty of what was actually sent after we 

 have received a signal. This reduction in received information 

 is the conditional entropy of the message and is called the equivo- 

 cation. It measures the average ambiguity of the received signal 

 or, in other words, the average uncertainty in the message when 

 the signal is known. For definiteness, let us calculate the equivo- 

 cation of the first example. In this example, noise caused an error 

 in about one out of each 1 00 symbols, so that if a zero was received, 

 the a posteriori probability that a zero was transmitted is .99 and 

 that a 1 was transmitted is .01. The equivocation, or the uncer- 

 tainty associated with each symbol, is exactly the entropy associated 

 with these concHtional probabilities. Thus, 



Equivocation per symbol = —[.99 log .99 + .01 log .01] 



= .081 bits. 



Since the source is producing information at a rate of 1,000 

 bits per second, the equivocation rate is 1,000 X .081 =81 bits 

 per second. Therefore, we may say that the system is transmitting 

 at a rate of 1,000 — 81 = 919 bits per second. Again, in the 

 extreme case where a is equally likely to be received as a or 1 

 and a 1 as a 1 or 0, the a posteriori probabilities are ,1/2 and 1/2, 



Equivocation = — U^ log ~y -{- 7, log -^ 



= 1 bit per symbol, 



or 1,000 bits per second. The rate of transmission is then zero as 

 it should be. 



These examples have demonstrated that noise causes a reduction 

 in received information and have shown precisely how this loss in 

 information is measured. Before leaving this subject, I would like 

 to quote a theorem (due to Shannon) which emphasizes why this 

 quantitative measure, called the equivocation, is so important. 



Shannon has shown that (in a noisy communication system) 

 if a correction channel is added which has a capacity equal to 



