630 



SCIENCE 



[N. S. Vol. XXXVIII. No. 983 



versity of Christiania, has retired. Mr. Askel 

 S. Steen succeeds him in these capacities. 

 Charles F. Brooks 

 Harvaed University 



SPECIAL AMIICLES 



RELIABILITY AND DISTRIBUTION OF GRADES 



If we consider grades scientifically as a 

 scale of measurements, two important ques- 

 tions arise: (1) How fine a scale of units is 

 distinguishable, and (2) What proportion of 

 persons will ordinarily fall under each unit? 



First, let us examine the question as to the 

 size of distinguishable steps. The answer to 

 this question can be determined by the relia- 

 bility with which marks can be assigned. Re- 

 cent studies have revealed an exceedingly wide 

 divergence in the grades assigned by different 

 teachers to the same papers. Starch and 

 Elliott' found that the grades assigned to two 

 English papers by 142 teachers of English 

 ranged in the ease of one paper from 64 to 98 

 with a probable error of 4.0, and in the case 

 of the other paper from 50 to 98, with a prob- 

 able error of 4.8. This wide range is not due 

 to the fact that these were language papers, 

 since the grades of a mathematics paper as- 

 signed by 118 teachers of mathematics ranged 

 from 28 to 92, with a probable error of 7.5 

 points." 



What bearing do these facts have upon the 

 reliability of marks and how are we to explain 



such wide ranges of differences? Four major 

 factors enter into the problem which, I believe, 

 fully account for the situation: (1) Differ- 

 ences among the standards of different schools, 

 (2) Differences among the standards of differ- 

 ent teachers, (3) Differences in the relative 

 values placed by different teachers upon vari- 

 ous elements in a paper, and (4) Differences 

 due to the pure inability to distinguish be- 

 tween closely allied degrees of merit. 



How much of the variation is due to each 

 factor? To determine the strength of the first 

 factor we must find out the range of variation 

 in the grades assigned by teachers in the same 

 institution and departments instead of differ- 

 ent institutions. To this end I obtained ten 

 papers written in the final examination in 

 freshman English at the University of Wis- 

 consin, and had them graded independently 

 by ten instructors of the various sections of 

 freshman English. An effort is made by co- 

 operation among the instructors concerned to 

 have as much uniformity as possible in the 

 conduct of these sections. The same final ex- 

 amination is given to all. 



Table I. gives the marks assigned by each 

 instructor to each paper. The first column 

 contains the grades assigned by the teachers 

 under whom the students took the course. 

 Papers 6 and 10 were obtained from the class 

 of one instructor and all the other papers from 

 the class of another instructor. These ten 



General average 78.7. 



' D. Starch and E. C. Elliott, School Beview, 20 : 2 D. Starch and E. C. Elliott, School Seview, 21 : 



