382 BELL SYSTEM TECHNICAL JOURNAL 



discrete symbols. A typical case is telegraphy where the message is a 

 sequence of letters and the signal a sequence of dots, dashes and spaces. 

 A continuous system is one in which the message and signal are both treated 

 as continuous functions, e.g. radio or television. A mixed system is one in 

 which both discrete and continuous variables appear, e.g., PCM transmis- 

 sion of speech. 



We first consider the discrete case. This case has applications not only 

 in communication theory, but also in the theory of computing machines, 

 the design of telephone exchanges and other fields. In addition the discrete 

 case forms a foundation for the continuous and mixed cases which will be 

 treated in the second half of the paper. 



PART I: DISCRETE NOISELESS SYSTEMS 



1. The Discrete Noiseless Channel 



Teletype and telegraphy are two simple examples of a discrete channel 

 for transmitting information. Generally, a discrete channel will mean a 

 system whereby a sequence of choices from a finite set of elementary sym- 

 bols ^i • • • Sn can be transmitted from one point to another. Each of the 

 symbols Si is assumed to have a certain duration in time ti seconds (not 

 necessarily the same for different Si , for example the dots and dashes in 

 telegraphy). It is not required that all possible sequences of the Si be cap- 

 able of transmission on the system; certain sequences only may be allowed. 

 These will be possible signals for the channel. Thus in telegraphy suppose 

 the symbols are: (1) A dot, consisting of line closure for a unit of time and 

 then line open for a unit of time; (2) A dash, consisting of three time units 

 of closure and one unit open; (3) A letter space consisting of, say, three units 

 of line open; (4) A word space of six units of line open. We might place 

 the restriction on allowable sequences that no spaces follow each other (for 

 if two letter spaces are adjacent, it is identical with a word space). The 

 question we now consider is how one can measure the capacity of such a 

 channel to transmit information. 



In the teletype case where all symbols are of the same duration, and any 

 sequence of the 32 symbols is allowed the answer is easy. Each symbol 

 represents five bits of information. If the system transmits n symbols 

 per second it is natural to say that the channel has a capacity of Sn bits per 

 second. This does not mean that the teletype channel will always be trans- 

 mitting information at this rate — this is the maximum possible rate and 

 whether or not the actual rate reaches this maximum depends on the source 

 of information which feeds the channel, as will appear later. 



