Robustness of Inter-Rater Reliability Estimators to Rater Leniency/Severity Effects In An Absolute Decision Setting - A Monte Carlo Study of Performance Ratings

<p>Inter-rater reliability coefficients are often reported in performance assessment as a measure of rating quality. Rater effects, as a systematic error in rating, may change the decision made about an examinee when an absolute interpretation of scores, such as a pass/fail decision, is used. Little research, however, examines the robustness of different inter-rater reliability estimators to rater effects. The purpose of the present research is to inform assessment practitioners or researchers reporting inter-rater reliability of (a) the more appropriate estimator in the presence of rater effects to achieve the most accurate estimates of inter-rater reliability and (b) the possible biases of the estimators under various conditions, such as number of scale points, initial inter-rater reliability, and number of papers. Results from this study indicate that the leniency/severity effects and the number of scale points substantially affect the accuracy of different inter-rater reliability estimators. The phi coefficient (or index of dependability) is not always the most accurate estimator, and it becomes the best choice only when the leniency/severity effect is large and the number of scale points is large. However, for other estimators, the increase in the number of scale points does not necessarily help the improvement of estimation accuracy.</p>

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Campus Access Dissertation

Discipline thesis:degree_discipline

Educational Studies

Year

2011

Author dc:creator

• Zhu, Min

Contributors dc:contributor

• Robert Johnson

• Education
• Educational Psychology
• inter-rater reliability estimator
• performance assessment
• rater effect

Statement dc:rights

• © 2011, Min Zhu

Repository record dc:identifier

https://scholarcommons.sc.edu/etd/1040

OAI identifier oai:identifier

oai:scholarcommons.sc.edu:etd-2041