A PRIMER ON INFORMATION THEORY* 



Henry Quastler 



Biology Department, Brookhaven National Laboratory, Upton, New York 



SYNOPSIS 



I. Introduction: Historic development of information theory; reason for its present 

 popularity. System theories in general; specific role of information theory. 



II. The representation of information: Paul Revere's code; essential features of representa- 

 tion of intelligence. Possibilities of representing information; variety of means; data and 

 operations; conscious and non-conscious acts of representation; generalized meaning of 

 'information'. 'Real' and 'symbolic' events; abstraction preceding representation. Symbol, 

 alphabet, 'words': units of representation. Binary representation, or standard method of 

 symbolization ; (a) simplest case: number of 'real' categories an integral power of two, 

 words of equal length; (b) any number of categories: words of unequal length, Fano's 

 'confusion-proof code, minimum-bulk code; (c) groups of events represented by single 

 words; (d) unequal probabilities: general rule to obtain a minimum-bulk code; (e) any 

 probabilities: general formula of minimum bulk in standard representation; (f) representation 

 theorem. Exercises. 



III. The measure of information or uncertainty : Information acquired and uncertainty 

 abolished. The amount of uncertainty a function of probabilities of events, not their nature, 

 causes and consequences; amount of information and representability; the //-function; the 

 'bit'. Some properties of the Shannon-Wiener information function: independence, con- 

 tinuity, additivity, naturalness of scale; values for probabilities zero and one; effects of 

 averaging and of pooling. Exercises. 



IV. Information measurements pertaining to two related variables: Generalized meaning 

 of 'communication'. An example of two related variables: heights of father and daughters; 

 joint uncertainty, internal constraints: the T-function; effects of scale on H and T. Two- 

 part systems in general: the six information functions for two variables. Communication 

 systems: nomenclature. Noise: the height correlation as example of a noisy channel; 

 channel capacity; manipulation of information does not increase its amount. Error detection 

 and correction: redundant information; theorem of the noisy channel ; economics of error 

 checking. Actual communication systems: signals and channels as physical entities. Exercises. 



V. Organization (systems, structures, pattern): Organization, communication, redundancy. 

 Systems analysis: informational analysis, informational challenge and performance; general 

 limitations on information-processing. Multi-part systems. Unitization. Conclusion. 



Appendix I: The evaluation of information content: Typical difficulties. The relativity of 

 information measures; arbitrariness of selections which determine actual values. Approxima- 

 tion methods. Examples: rate of information transmission in conversation; information 

 content per printed letter. 



Appendix II." Answers to exercises. 



* This paper is based on a report of the same name issued as Office of Ordnance Research 

 Technical Memorandum 56-1, in January 1956. The memorandum was written at the sugges- 

 tion of Dr Sherwood Githens, Jr, Director of the Physical Sciences Division of the Office of 

 Ordnance Research; he and his staff were very helpful at all stages of the execution of this 

 project, and it is a pleasure to extend them thanks. The present revised version was prepared at 

 Brookhaven National Laboratory, under the auspices of the Atomic Energy Commission. 



3 



