A Primer on Information Theory 37 



Let H(x) be the amount of infonnation needed to know what state x is in. 

 If y is known, some of this information becomes unnecessary, or redundant. 

 This amount, T{x;y), is an index of the degree of coherence, constraint, integra- 

 tion, or organization which prevails in the system. 



Consider the pair of words 'green valley'. These two words form a small 

 system — a whole made up of interrelated parts. The whole has a meaning 

 which neither part alone has. The price for this feature is elimination of many 

 other possible connotations of 'green' and 'valley'. As a result, the information 

 content of the word combination is smaller than the combined information 

 contents of the two words. The difference must show up as redundant informa- 

 tion. The presence of redundancy implies that each word contains some 

 information about the other. This is best demonstrated by successful error 

 checking. The errors 'preen 'for 'green', and 'volley' for 'valley' would not be 

 found in isolated words, but can be spotted in the pair. 



System Analysis — There seem to be three general viewpoints under which 

 relations within a system are assessed: (a) the amount of information trans- 

 mitted — on the technical, semantic and pragmatic level ; (b) the degree of control 

 or cause-effect relations, dominance; and (c) the utility, or value, of the relation 

 to one or both of the related parts. Information theory deals only with the 

 first viewpoint. It does not concern cause-effect relations, or what causes the 

 information to flow, and it is not concerned either with the utility of the flow of 

 information. 



Informational analysis of a system will be of interest if and only if the 

 informational challenge is serious, that is, when a system has to process informa- 

 tion at a rate which crowds its capabilities. The informational challenge is 

 the result of: 



(1) The diversity which is characteristic of the tasks; this can be expressed 

 as ///task. A system which is faced with the same task all the time or most of 

 the time may be working very hard but the difficulty is not an informational 

 one. 



(2) The precision which is required ; this can be expressed as the ratio TIN. 

 That is, the diversity of tasks is informationally challenging only insofar as 

 it is expressed in a diversity of responses. A system with a small response 

 repertoire may be working very hard, but not in the informational domain. 



(3) The time which is allotted for the fulfillment of each task. A system 

 with very modest informational equipment can solve many tasks if given ample 

 time. For instance, the extremely simple logical machine devised by Turing (13) 

 will solve any solvable problem if given very much time. 



The time rate of informational challenge of the system is the product 



H T tasks _, . . 



X 7> X — TT — = Tlumt time. 



task H unit time 



The infoiTnational output of the system will be measured in //-measures 

 but the effective output, or informational performance, in terms of T-measures, 

 as T per task or T per unit time. The limits of the informational performance 

 of a system can be found by systematically varying the informational challenge 

 and observing the resulting performance. In such studies it is important to 



