MATHEMATICAL THEORY OF COMMUNICATION 401 



By proper assignment of the transition probabilities the entropy of sym- 

 bols on a channel can be maximized at the channel capacity. 



9. The Fundamental Theorem for a Noiseless Channel 



We will now justify our interpretation of H as the rate of generating 

 information by proving that H determines the channel capacity required 

 with most efficient coding. 



Theorem 9: Let a source have entropy H (bits per symbol) and a channel 

 have a capacity C (bits per second). Then it is possible to encode the output 



C 



of the source in such a way as to transmit at the average rate -fj~ ^ symbols 



per second over the channel where t is arbitrarily small. It is not possible 



C 



to transmit at an average rate greater than — . 



C 

 The converse part of the theorem, that — cannot be exceeded, may be 



H 



proved by noting that the entropy of the channel input per second is equal 



to that of the source, since the transmitter must be non-singular, and also 



this entropy cannot exceed the channel capacity. Hence H' < C and the 



number of symbols per second = H' /H < C/H. 



The first part of the theorem will be proved in two different ways. The 



first method is to consider the set of all sequences of N symbols produced by 



the source. For N large we can divide these into two groups, one containing 



less than 2^''^''^'^ members and the second containing less than 2^"^ members 



(where R is the logarithm of the number of different symbols) and having a 



total probabiUty less than /x. As N increases rj and ju approach zero. The 



number of signals of duration T in the channel is greater than 2^^^^^ with 



6 small when T is large. If we choose 



=(?-) 



then there will be a sufficient number of sequences of channel symbols for 

 the high probability group when .V and T are sufficiently large (however 

 small X) and also some additional ones. The high probability group is 

 coded in an arbitrary one to one way into this set. The remaining sequences 

 are represented by larger sequences, starting and ending with one of the 

 sequences not used for the high probability group. This special sequence 

 acts as a start and stop signal for a different code. In between a sufficient 

 time is allowed to give enough different sequences for all the low probability 

 messages. This will require 



(?+')» 



