JVhat is Information Theory? 7 



a system (in this case eight), the number of ahernatives resolved 

 by each question (two, because of the binary nature of the ques- 

 tion), and the minimum number of questions necessary to de- 

 termine tlie state of the system (three in this case). It is easily 

 seen that tiie relationship between these numbers is 2'^ = 8. In 

 the vernacular of information theory, we say that three bits of 

 information are necessary to determine the state of such a sys- 

 tem; i.e., three appropriately chosen questions, each of which 

 resolves two alternatives, usually designated as 1 or 0, correspond- 

 ing" to yes or no, are all that is necessary to reduce indeterminacy 

 to certainty. The problem of choosing the appropriate questions 

 is analogous to that of choosing a good code. For example, asking 

 the question: "Is the number 3?" would correspond to very 

 inefficient coding of information. Phrasing or coding questions in 

 this way would require that you be allowed to ask eight questions 

 in order to be certain to determine the state of the system. In this 

 illustration a correct way of coding or phrasing the c^uestions 

 would be as follows: Question 1: "Is the number greater than 4?" 

 If yes, then ask Question 2: "Is the number greater than 6?" 

 If no, then ask Question 3: "Is the number 5?"' If no, then the 

 system must be in state 6. 



As previously mentioned, the probability of having guessed the 

 correct state before receiving these three bits of information was 

 1/8 in the example used. After the first bit of information is re- 

 ceived the probability of guessing" correctly is increased from 

 1/8 to 1/4, after the second bit from 1/4 to 1/2, and after the 

 third bit from 1 ;'2 to 1 . Thus, each successive bit received has 

 reduced our uncertainty as to the state of the system until all the 

 uncertainty is removed. In this example, the receipt of any more 

 information is unnecessary or redundant. However, as we shall 

 discuss later, the redundancy may be useful in correcting errors 

 due to noise in the communication channel. 



In order to use a more general illustration which is not restricted 

 to a system with equally probable states, let us consider the game 

 of Twenty Questions. In most situations the probabilities of some 

 states, i.e., the possible set of objects to be identified, are higher 

 than the probabilities of others. A good information theorist with 

 some a priori knowledge of the probabilities of these states would 



