MATHEMATICAL THEORY OF COMMUNICATION 421 



work is traversed in a long sequence of length N is about proportional to the 

 probability of being at i and then choosing this path, PipijN. If N is large 

 enough the probability of percentage error ± 5 in this is less than € so that 

 for all but a set of small probability the actual numbers lie within the limits 



{PiPij ± b)N 



Hence nearly all sequences have a probability p given by 



p = Tip\Y'''^'''' 



loff P 

 and — — is limited by 



i^ = 2(P,^y±6)log^•y 



logp 



N 



- ZPiPij log Pa 



<V- 



This proves theorem 3. 



Theorem 4 follows immediately from this on calculating upper and lower 

 bounds for n{q) based on the possible range of values of p in Theorem 3. 



In the mixed (not ergodic) case if 



L = i:piLi 

 and the entropies of the components are Hi > H2 ^ . . . > Hnwe have the 

 Theorem: Lim -^^r— = <piq) is a decreasing step function, 



N->oo N 



8 — 1 8 



(p{q) = Hs in the interval ^ oli < q <^ Ui. 



1 1 



To prove theorems 5 and 6 first note that Fn is monotonic decreasing be- 

 cause increasing N adds a subscript to a conditional entropy. A simple 

 substitution for pBi (Sj) in the definition of Fn shows that 



Fn = N Gn - {N - 1) Gs-i 



and summing this for all N gives On = -r= S Fiv . Hence On ^ Fs and Gx 



monotonic decreasing. Also they must approach the same limit. By using 

 theorem 3 we see that Lim Gy = H. 



JV-*oo 



APPENDIX 4 



Maximizing the Rate for a System of Constraints 



Suppose we have a set of constraints on sequences of symbols that is of 

 the finite state type and can be represented therefore by a linear graph. 



