MATHEMATICAL THEORY OF COMMUNICATION 413 



The probability that none of the points in the fan is a message (apart from 

 the actual originating message) is 



Now R < H{x) - Hy{x) so R - H{x) = -Hy{x) - t] with 7; positive- 

 Consequently 



P = [1 — 2~^^«'^^^~^''l2^^l'^^^ 



approaches (as T ^ oc ) 



Hence the probability of an error approaches zero and the first part of the 

 theorem is proved. 



The second part of the theorem is easily shown by noting that we could 

 merely send C bits per second from the source, completely neglecting the 

 remainder of the information generated. At the receiver the neglected part 

 gives an equivocation H{x) — C and the part transmitted need only add e. 

 This limit can also be attained in many other ways, as will be shown when we 

 consider the continuous case. 



The last statement of the theorem is a simple consequence of our definition 

 of C. Suppose we can encode a source with R = C -\- am. such a way as to 

 obtain an equivocation Hy{x) = a — e with e positive. Then R = H{x) = 

 C -\- a and 



H{x) - Hy{x) = C + € 



with € positive. This contradicts the definition of C as the maximum of 

 H{x) - Hy{x). 



Actually more has been proved than was stated in the theorem. If the 

 average of a set of numbers is within e of their maximum, a fraction of at 

 most v^can be more than \/e below the maximum. Since e is arbitrarily 

 small we can say that almost all the systems are arbitrarily close to the ideal. 



14. Discussion 



The demonstration of theorem 11, while not a pure existence proof, has 

 some of the deficiencies of such proofs. An attempt to obtain a good 

 approximation to ideal coding by following the method of the proof is gen- 

 erally impractical. In fact, apart from some rather trivial cases and 

 certain limiting situations, no explicit description of a series of approxima- 

 tion to the ideal has been found. Probably this is no accident but is related 

 to the difficulty of giving an explicit construction for a good approximation 

 to a random sequence. 



