MATHEMATICAL THEORY OF COMMUNICATION 405 



^0 

 B 10 



C 110 



D 111 



The average number of binary digits used in encoding a sequence of N sym- 

 bols will be 



A^(| X 1 + i X 2 + f X 3) = iiV 



It is easily seen that the binary digits 0, 1 have probabilities J, J so the ^ for 

 the coded sequences is one bit per symbol. Since, on the average, we have i 

 binary symbols per original letter, the entropies on a time basis are the 

 same. The maximum possible entropy for the original set is log 4 = 2, 

 occurring when A, B,C, D have probabilities f, j, J, f . Hence the relative 

 entropy is J. We can translate the binary sequences into the original set of 

 symbols on a two- to-one basis by the following table: 



00 A' 



01 B' 



10 C 



11 D' 



This double process then encodes the original message into the same symbols 

 but with an average compression ratio I . 



As a second example consider a source which produces a sequence of .4's 

 and 5's with probability p for A and qior B. lip < < q we have 



H= -log p\\ - py-' 

 = -p\ogp{\-pr-'"' 



p 



In such a case one can construct a fairly good coding of the message on a 

 0, 1 channel by sending a special sequence, say 0000, for the infrequent 

 symbol .4 and then a sequence indicating the number of 5's following it. 

 This could be indicated by the binary representation with all numbers con- 

 taining the special sequence deleted. All numbers up to 16 are represented 

 as usual; 16 is represented by the next binary number after 16 which does 

 not contain four zeros, namely 17 = 10001, etc. 



It can be shown that as /? -^ the coding approaches ideal provided the 

 length of the special sequence is properly adjusted. 



