A Primer on Information Theory 



35 



particular output. If the association is not unequivocal, then the channel is 

 said to be noisy. 'Noise' is defined as a random perturbation of the input- 

 output link. Those are nice, clean concepts, not to be confused with realities. 

 The 'channel' exists on paper only, and is not the same as the mechanism which 

 links two parts of a system. The infonnational relation between heights of 

 fathers and daughters does not reveal the nature of the mechanisms involved; 

 whether fathers affect their daughters' heights by means of their genes, or of 

 the food they provide, or of the mother they select for them, cannot be decided 

 on grounds of informational relations. Indeed, I believe that Buddhist tradition 

 would explain the correlation on the grounds that daughters select their fathers; 

 as far as information theory is concerned, this is perfectly acceptable. 



The scheme shown in Fig. 5 is a somewhat closer approximation to reality: 



NOISE 



SOURCE 



MESSAGE. 



• ENCODER 



TRANSMITTER 



SIGNALS 



CHANNEL 



SIGNALS 



DESTINATION [ J^^SSAGE ^ qe-qqqer l.^ RECEIVER (— ' 



Fig. 5. A diagrammatic representation of a communication system 



It is customary to treat all links but the channel as noise-free. If need be, one 

 can introduce noise into the other links of the model by some straight-forward 

 adaptations. 



If signals and channels are physical entities, then it is relevant to investigate 

 their physical capacity of carrying information. Suppose the nature of a unit 

 of action and the physical constraints are such that the channel can assume any 

 one of m states during one unit of action; then, these states can be made to 

 represent log., m bits of information. It is the function of the encoder-trans- 

 mitter system to match the diversity of messages generated by the source to 

 the diversity of states which can be assumed by the channel; those, in turn, are 

 matched to the diversity of messages intelligible at the destination by the 

 receiver-decoder system. 



As long as the demands on the channel are light, the matching process is 

 not much of a problem. However, it may become very difficult if the channel 

 is to be driven at capacity, and if the various states of the channel are not of 

 equal value; some may be more subject to noise effects than others, some may 

 need more time than others, some may necessitate more effort than others. 

 In general, one will tend to favor the safest, shortest, and easiest states. However, 

 this must not go too far; if one goes to the extreme of using the very 'best' 

 state, then the channel does not transmit any information at all. To find 

 optimum compromises between informational needs of source and destination 

 and physical capacities of the channel, between amount of information used to 

 carry messages and amount of information needed for noise reduction, is one 

 of the fundamental problems of the theory of information and communication. 



