414 BELL SYSTEM TECHNICAL JOURNAL 



An approximation to the ideal would have the property that if the signal 

 is altered in a reasonable way by the noise, the original can still be recovered. 

 In other words the alteration will not in general bring it closer to another 

 reasonable signal than the original. This is accomplished at the cost of a 

 certain amount of redundancy in the coding. The redundancy must be 

 introduced in the proper way to combat the particular noise structure 

 involved. However, any redundancy in the source will usually help if it is 

 utilized at the receiving point. In particular, if the source already has a 

 certain redundancy and no attempt is made to eliminate it in matching to the 

 channel, this redundancy will help combat noise. For example, in a noiseless 

 telegraph channel one could save about 50% in time by proper encoding of 

 the messages. This is not done and most of the redundnacy of English 

 remains in the channel symbols. This has the advantage, however, of 

 allowing considerable noise in the channel. A sizable fraction of the letters 

 can be received incorrectly and still reconstructed by the context. In 

 fact this is probably not a bad approximation to the ideal in many cases, 

 since the statistical structure of English is rather involved and the reasonable 

 English sequences are not too far (in the sense required for theorem) from a 

 random selection. 



As in the noiseless case a delay is generally required to approach the ideal 

 encoding. It now has the additional function of allowing a large sample of 

 noise to affect the signal before any judgment is made at the receiving point 

 as to the original message. Increasing the sample size always sharpens the 

 possible statistical assertions. 



The content of theorem 1 1 and its proof can be formulated in a somewhat 

 different way which exhibits the connection with the noiseless case more 

 clearly. Consider the possible signals of duration T and suppose a subset 

 of them is selected to be used. Let those in the subset all be used with equal 

 probabiUty, and suppose the receiver is constructed to select, as the original 

 signal, the most probable cause from the subset, when a perturbed signal 

 is received. We define N{T, q) to be the maximum number of signals we 

 can choose for the subset such that the probability of an incorrect inter- 

 pretation is less than or equal to q. 



Theorem 12: Lim — ^J;— ^ = C, where C is the channel capacity, pro- 



T-*ao 1 



vided that q does not equal or 1 . 



In other words, no matter how we set our limits of reliability, we can 

 distinguish reliably in time T enough messages to correspond to about CT 

 bits, when T is sufficiently large. Theorem 12 can be compared with the 

 definition of the capacity of a noiseless channel given in section 1 . 



