A Primer on Information Theory 33 



Other outputs than the required ones appear; in other words, a noisy channel 

 produces errors. Errors lead to loss of information, and a reduction in the 

 rate of transmission; in a noisy channel, 



Tix;y)<H(x) 



Hy(x) > 0. 



This loss is unavoidable. However, it is at least possible to spot and correct 

 the errors which have occurred. It is one of the main endeavours of information 

 theory to devise methods to do this efficiently. 



An error in a message can never be found unless the message contains some 

 extra information which can be used for this purpose. For instance, if the 

 message consists of a string of four digits chosen without any constraint: 



5 3 8 7, 



one has absolutely no possibility of knowing whether or not it contains any 

 errors. If it has been agreed upon that the message will be repeated, then one 

 can detect errors : 



5 4 8 7 



5 3 7 7, 



and if the message is repeated several times, these errors can be detected and 

 corrected, with arbitrary certainty if the number of replications can be made 

 sufficiently large: 



5 3 8 7 



5 3 7 7 



5 3 8 7 



5 4 8 7 



5 3 8 1. 



In the second case, the possibility of error detection was bought at the price 

 of making two digits do the work of one; the message is said to be 50 per cent 

 redundant. In the last case, the price of error correction is the use of five digits 

 to transmit a single one, or a redundancy of 80 per cent. 



Introducing redundant information in the fonn of a simple replication is 

 straight-forward and eiTective, but not very economical. Error detection could be 

 achieved more efficiently by simply adding the sum of the digits to the message: 

 be achieved more efficiently by simply adding the sum of the digits to the message : 



5 3 8 7 2 3. 



Here, the redundant information is only one-third of the total. In fact, giving 

 only the last digit of the sum as 'signature' is almost as effective, and requires 

 only 1 digit in 5, or 20 per cent redundant infonnation. The signature check 

 illustrates a general principle: a given amount of redundant infonnation in a 



