MATHEMATICAL THEORY OF COMMUNICATION 631 



then the maximum entropy occurs when 



P\x) = - e 

 a 



and is equal to log ea. 



8. There is one important difference between the continuous and discrete 

 entropies. In the discrete case the entropy measures in an absolute 

 way the randomness of the chance variable. In the continuous case the 

 measurement is relative to the coordinate system. If we change coordinates 

 the entropy will in general change. In fact if we change to coordinates 

 >'i • • • jn the new entropy is given by 



H{y) = j ■ - j p{xi • • • Xn)J l-j log p{xi ■ ■ • Xn)J (-) dji - • ■ dyn 



where 7 f - j is the Jacobian of the coordinate transformation. On ex- 

 panding the logarithm and changing variables to Xi - - - Xn , we obtain: 



H(y) = H(x) - j • ' • j P(^i , -■ ,Xn)logJ l-j dxi-" dxn . 



Thus the new entropy is the old entropy less the expected logarithm of 

 the Jacobian. In the continuous case the entropy can be considered a 

 measure of randomness relative to an assumed standard, namely the co- 

 ordinate system chosen with each small volume element dxi • • • dXn given 

 equal weight. When we change the coordinate system the entropy in 

 the new system measures the randomness when equal volume elements 

 dyi • • • dyn in the new system are given equal weight. 



In spite of this dependence on the coordinate system the entropy 

 concept is as important in the continuous case as the discrete case. This 

 is due to the fact that the derived concepts of information rate and 

 channel capacity depend on the difference of two entropies and this 

 difference does not depend on the coordinate frame, each of the two terms 

 being changed by the same amount. 



The entropy of a continuous distribution can be negative. The scale 

 of measurements sets an arbitrary zero corresponding to a uniform dis- 

 tribution over a unit volume. A distribution which is more confined than 

 this has less entropy and will be negative. The rates and capacities will, 

 however, always be non-negative. 



9. A particular case of changing coordinates is the linear transformation 



