462 Information Theory and Biology /25 : 2 



it will not be removed completely. In the language of information 

 theory, the answer is masked by the "noise" caused by the student's 

 inability to assess his own work. If, instead of asking the student, the 

 teacher had "asked" his test paper (that is, corrected it), all uncertainty 

 would have been removed. 



Again, suppose the teacher asks a student if he likes ice cream. It is 

 extremely likely that he does, so the teacher can receive only very little 

 information if the student answers "yes." This time the answer will 

 remove all uncertainties. (Unless, for instance, the student has such 

 poor diction that the teacher cannot understand his speech.) 



From these examples, one can note that the more uncertain the 

 a priori choice, the more information there is that can be supplied by 

 the answer. Further, the more certainty that exists after the answer, 

 the greater the amount of information received. The quantitative form 

 of information must reduce to zero if the initial probability of the answer 

 p t is one, and must be a maximum if the final (or output) probability p is 

 one. The ratio p /pi varies as the information but does not go to zero 

 when pi (and therefore p ) are unity. A function which does behave as 

 information, and which is always positive, is log (/> /A)- For 

 historical reasons, information is defined in information theory as 



I = log 2 {Pol Pi) (1) 



The unit of / is called a bit, an abbreviation for binary integer. Because 

 p Q is greater than, or at worst equal to, p u information / as defined by 

 Equation 1 will always be positive. 



As an example of Equation 1 , one may ask if a given neuron is con- 

 ducting a spike potential at a given time. The two possible answers are 

 yes and no, so that the a priori probability p t is 



Pi = 1/2 



If the question is asked with suitable measuring equipment, the answer 

 may be definitely yes; that is, the output probability p is one. The 

 information gained is 



/ = log 2 ^° = log 2 r = log 2 2 = lbit 



Pi 2 



Electrical engineers were the first to use information theory. Many 

 electronic circuits exist in one of two stable positions. In digital 

 electronic computers, all decimal numbers are reduced to binary 

 numbers which involve making several yes-no choices. The base 2 

 logarithm appears to be the natural one, not only for the electronics 

 engineer, but also for the physiologists and biophysicists who work with 

 nerves which follow a yes-no pattern. (Historically, the physiologists 

 have preferred the words "all-or-none.") 



Table I compares decimal and binary integers. It also gives the 



