October 31, 1913] 



SCIENCE 



631 



papers were graded after each instructor had 

 graded the papers from his own sections. 



(1) The table reveals an exceedingly wide 

 range of marks, a range just as large as that 

 of the English and mathematics papers re- 

 ferred to above. The average of the mean 

 variations is 5.3 as compared with an average 

 of 5.4 of the English and mathematics papers. 

 (2) The mean variations are fairly uniform 

 for all papers except 4 and 9. These two, no 

 doubt, vary so much more widely than the 

 others because both have an average below the 

 passing grade. Judgments of such papers are 

 more apt to be haphazard since, from the prac- 

 tical point of view, it makes no difference what 

 the grade is, so long as the paper is consid- 

 ered a failure. But the matter is quite serious 

 in case of a paper like number 9 which is con- 

 sidered above passing by six and below pass- 

 ing by four instructors. (3) A third point of 

 interest is the fact that the teachers under 

 whom the students took the course grades in 

 column 1, did not succeed in grading the 

 papers any more accurately than the other in- 

 structors who did not know the students at all. 

 The mean variation of the grades in column 1 

 from the average of each paper is practically 

 as large, 4.7, as the mean variation of all to- 

 gether, 5.3. (4) There is a very noticeable 

 diiference in the standard of grading. Two 

 instructors, 4 and 5, graded on the whole very 

 much lower than the average and Nos. 7 and 

 8 graded higher than the average. These 

 deviations can be found readily by comparing 

 each instructor's average with the general 



tion is smaller, though not as much smaller as 

 one might anticipate, being 4.3 as compared 

 with 5.3 in Table I. 



In order to eliminate the variation in the 

 marks due to this difference in standards 

 among the instructors, all the marks in Table 

 I. were weighted by the amount that each in- 

 structor's average differed from the general 

 average. The weighted values thus obtained 

 are presented in Table II. The decimals were 

 dropped in the transposition. 



The differences in Table II. therefore repre- 

 sent the differences in the relative evaluation 

 of the papers themselves irrespective of 

 whether an instructor marks severely or leni- 

 ently. It will be noticed that the mean varia- 



The next step is to separate the third and 

 fourth factors, i. e., how much of the variation 

 is due to the inability to distinguish between 

 closely allied degrees of merit, and how much 

 is due to differences in relative value placed 

 by different instructors upon various aspects 

 of a given paper, such as form, neatness, 

 clearness, etc. 



The accuracy of the ability to distinguish 

 between various shades of merit may be ascer- 

 tained by having the same person give two or 

 more evaluations of the same papers sepa- 

 rated by sufficiently long intervals of time, so 

 that the details and identity of the papers have 

 been forgotten. I have tested this point by 

 determining how closely an instructor is able 

 to agree with his own grades. Table III. gives 

 pairs of grades assigned at different intervals 

 to the same papers by the same instructor. In 

 each case the papers were from the instructors' 

 own classes. The aim was to have ten papers 

 re-graded, but in some instances not that many 

 were available. 



Table III. shows that the difference in the 

 marks assigned to the same papers by the same 

 instructor is on the average 4.4 points, or in 

 terms of mean variation 2.2 points. This dif- 

 ference is as large in one sort of papers as in 

 another. It is as large in mathematics as in 

 or in science. This was to be ex- 



