A Primer on Information Theory 5 



which are more and more complex; they depend for their functioning on 

 successful and efficient communication, and information theory offers general 

 principles about communication. 



Information theory is related to a group of specialties which are either 

 new or have recently increased tremendously in popularity, as has information 

 theory itself. To name but a few: operations analysis; the theory of experi- 

 mental design; decision theory; theory of linear programming; cybernetics; 

 game theory; theories of administration; group dynamics; and others. 



A little pondering over this list suggests a few generalizations: 



(i) all of these sciences are mathematical, the fields of probability and 

 statistics being referred to most frequently; 



(ii) each employs a system of evaluation* of something; 



(iii) each deals with complex situations; in every one, there is a multi- 

 plicity of possible choices as arrangements of some sort, and in 

 most of them, a large number of interrelated factors affect the 

 choices. Thus, they all can properly be called system theories; 



(iv) none of these sciences is primarily concerned either with the physical 

 nature of the system considered or with the mechanisms by which 

 its parts are interrelated. 



These are the common features. However, each of the endeavors named 

 is a special science since each deals with a different aspect of systems. It is an 

 open question whether because of their similarities these different sciences 

 can be gathered under one common discipline that could be called General 

 Systems Theory (and is the basis of a society formed in 1954). 



Information theory, thus, is only one of several system theories. The parti- 

 cular concern which characterizes it in contrast to the related specialities is 

 measurement of the degree to which a thing (or a condition, or an event) is 

 specified; that is, of the degree to which it differs from other possible things 

 (conditions, events). Communication and organization are treated in terms 

 of a mutual specification. One way to illustrate the essence of information 

 theory is to compare it with statistics. Statistics and information theory both 

 deal with the diversity among the elements of a set, but in different ways. 

 Statistics treats diversity as a nuisance, and tries to establish what can be 

 stated or done in spite of it. Information theory treats diversity as an asset 

 without which operations such as selection, communication, representation, 

 specification, would not be possible; it tries to establish what can be achieved 

 because of a certain degree of diversity. The 'information' evaluated in Infor- 

 mation Theory is thus not the every-day information. The 'information' in 

 a message, for example, as a type of event, is the measure of the amount of 

 knowledge (intelligence) which a message of this sort ideally can convey through 

 the medium of symbolic representation. 



To many people, information theory looks highly promising on first contact; 

 to some people, it still looks promising after serious study. Information theory 

 has become well established among engineers. In psychology, it has achieved 



* In some cases values are frankly imposed, in others they are inferred from observed 

 behavior. This does not necessarily mean that the values must be consciously imposed; 

 goal-directed behavior can occur without any conscious act of fixing values. 



