A Primer on Information Theory 13 



portions of unequal size. In the case of three categories the first partition 

 separates category 'A' or 33 per cent of all categories, from 'B or C\ representing 

 67 per cent. This situation can be improved when it is allowed to represent 

 pairs of events, instead of single events. There are nine pairs of the events A, B, 

 and C, designated AA, AB, AC, BA, etc. We use the first symbol to subdivide 

 them into two groups of four and five, respectively; each of these groups is 

 subdivided by the second symbol, etc. We obtain: 



Pairs of real events Symbolic representation 



AA ' 111 



AB 110 



AC 10 1 



BA 10 



BB Oil 



BC 10 



CA 1 



CB 1 



CC 



29 

 Average: — = 3.22 symbols per pair of events, or 1.61 per single event. 



Excess digits = .03 < 2 per cent 



By going from pairs to triplets, the limiting value can be approached still 

 closer. In general, if the group of events to be represented can be made as large 

 as desired, then the limiting value can be approached as closely as desired. 



Unequal Probabilities — In general, categories occur with unequal probabili- 

 ties. In this case, subdividing the categories into sub-sets containing equal 

 numbers of categories will not result in a minimum-bulk code. Consider, for 

 instance, the three Fano codes for a set of five categories. Suppose category 

 A accounts for 80 per cent of all occurrences, and the other four for 5 per cent 

 each. In this case, code (a) will yield minimum bulk with an average of 1.4 

 digits per word, followed by code (c) with 1.45 and code (b) with 2.1 digits. 

 The general rule to obtain a minimum bulk code, with any number of categories 

 and equal or unequal probabilities, is as follows : all divisions and subdivisions 

 should be between groups of categories of as nearly as possible equal aggregate 

 probabilities. 



The average number of digits in a minimum bulk code is found by the 

 following consideration: let p{i) be the probability of an event falling into 

 /'th category, where / may stand for A, B, C, . . . , if the categories are designated 

 by letters, or for 1, 2, 3, ... , if the categories are numbered. For the time being, 

 we consider only probabilities which are integral powers of 1/2, i.e., 1/2, (1/2)^ = 

 1/4, (1/2)3 ^ 1/8^ (1/2)4 _ 1/16^ etc.; i.e. we set p{i) = {Xjiyi where z^ is a 

 positive integer. In such a case, each step in the coding procedure can be a 

 partition into groups of equal aggregate probabihty; then, the code word for 



