402 BELL SYSTEM - TECHNICAL JOURNAL 



where (p is small. The mean rate of transmission in message s>Tnbols per 

 second will then be greater than 



c 



As N increases 6, X and <p approach zero and the rate approaches — . 



H 



Another method of performing this coding and proving the theorem can 

 be described as follows: Arrange the messages of length N in order of decreas- 

 ing probability and suppose their probabilities sltq pi > p2 '> p3 . ■ - ^ pn - 



s-l 



Let P. = ^ pi ; that is P« is the cimiulative probability up to, but not 



mclading, pg . We first encode into a binary system. The binary code for 

 message s is obtained by expanding P« as a binary number. The expansion 

 is carried out to m, places, where nis is the integer satisfying: 



log2 -- < nis < 1 -{■ log2 — 

 p, ps 



Thus the messages of high probability are represented by short codes and 

 those of low probability by long codes. From these inequalities we have 



' <P,< ' 



2^9 — ^' 2"**"-^ 



The code for Ps will differ from all succeeding ones in one or more of its 



nig places, since all the remaining Pi are at least -^ larger and their binary 



expansions therefore differ in the first w« places. Consequently all the codes 

 are different and it is possible to recover the message from its code. If the 

 channel sequences are not already sequences of binary digits, they can be 

 ascribed binary numbers in an arbitrary fashion and the binary code thus 

 translated into signals suitable for the channel. 



The average number H' of binary digits used per symbol of original mes- 

 sage is easily estimated. We have 



H' =^i:m.pg 



But, 



i2(log,l),,<^2..^<i.s(l + log,l)^ 



and therefore, 



