PREDICTION AND ENTROPY OF PRINTED ENGLISH 



63 



of the general theorem that Hy{x) < n{x) for any chance variables x and y. 

 The equality holds only if the distributions being added are proportional. 

 Now we may add the different components of the same width without 

 changing the entropy (since in this case the distributions are proportional). 

 The result is that we have arrived at the rectangular decomposition of the 

 qi , by a series of processes which decrease or leave constant the entropy, 

 starting with the original iV-gram probabilities. Consequently the entropy 

 of the original system Fn is greater than or equal to that of the rectangular 

 decomposition of the qi . This proves the desired result. 



It will be noted that the lower bound is definitely less than Fn unless each 

 row of the table has a rectangular distribution. This requires that for each 



12 3 



5 6 7 8 9 10 11 12 13 14 15 

 NUMBER OF LETTERS 



100 



Fig. 4 — Upper and lower experimental bounds for the entropy of 27-letter English. 



possible (iY-1) gram there is a set of possible next letters each with equal 

 probability, while all other next letters have zero probabiHty. 



It will now be shown that the upper and lower bounds for Fn given by 

 (17) are monotonic decreasing functions of N. This is true of the upper bound 

 since the ql^^ majorize the q^ and any equalizing flow in a set of probabilities 

 increases the entropy. To prove that the lower bound is also monotonic de- 

 creasing we will show that the quantity 



U =Y^ i(qi - qi+i) log i (20) 



is increased by an equalizing flow among the qi . Suppose a flow occurs from 

 qi to ^,+1 , the first decreased by Aq and the latter increased by the same 

 amount. Then three terms in the sum change and the change in U is given by 



A^ =[-{i- 1) log (i - 1) + 2i log ^ - (^ + 1) log {i + 1)1A^ (21) 



