Further Evaluating the Effect of Behavioral Observability and Overall Impressions on Rater Agreement: A Replication Study

<p>This replication study sought to analyze the effects of behavioral observability and overall impressions on rater agreement, as recently examined by Roch, Paquin, & Littlejohn (2009) and Scott (2012). Results from the study performed by Roch et al. indicated that raters are more likely to agree when items are either more difficult to rate or less observable. In the replication study conducted by Scott, the results did not support the relationship which Roch et al. found between observability and rater agreement, but did support the relationship previously found between item difficulty and rater agreement. The four objectives of this replication study were to determine whether rater agreement is negatively related to item observability (Hypothesis 1) and positively related to difficulty (Hypothesis 2), as well as to determine whether item performance ratings are closer to overall impressions when items are less observable (Hypothesis 3) and more difficult to rate (Hypothesis 4). The sample was comprised of 152 undergraduate students tasked with providing performance ratings on an individual depicted in a video of a discussion group. Results indicated that agreement was negatively correlated with both observability (supporting Hypothesis 1) and difficulty (not supporting Hypothesis 2), and that ratings were closer to overall impressions when items were less observable (supporting Hypothesis 3), but not when items were more difficult to rate (not supporting Hypothesis 4).</p>

Name thesis:degree_name

Master of Science

Discipline thesis:degree_discipline

Department of Psychological Sciences

Year

2015

Author dc:creator

• Sizemore, Patrick

Contributors dc:contributor

• Anthony R. Paquin (Director), Reagan Brown, and Aaron Wichman

• rater agreement
• item characteristics
• Applied Behavior Analysis
• Experimental Analysis of Behavior
• Psychology

Repository record dc:identifier

https://digitalcommons.wku.edu/theses/1498

OAI identifier oai:identifier

oai:digitalcommons.wku.edu:theses-2502