The Relationship Between Rater Agreement, Behavioral Observability and Overall Impressions

<p>This study examined two item characteristics believed to influence rater agreement: observability and difficulty. The first goal of this study was to replicate the findings of Roch, Paquin and Littlejohn (2009), which found that rater agreement was negatively related to item observability (Hypothesis 1) and rating difficulty (Hypothesis 2). The study also explored whether participants had closer item performance ratings to their overall impression when items were less observable (Hypothesis 3) and more difficult to rate (Hypothesis 4). A sample of 254 Undergraduate psychology students viewed a video of a leaderless group discussion and then filled out a rating form assessing performance of one of the individuals in the video and rating difficulty. Results were that rater agreement was positively related to observability (not supporting Hypothesis 1) and negatively related to difficulty (supporting Hypothesis 2). RDS, a distance score between participant’s overall impression and the item performance rating was computed to assess Hypotheses 3 and 4. RDS was positively related to observability (supporting Hypothesis 3) and not related to difficulty (not supporting Hypothesis 4). The positive relationship between observability and rater agreement was surprising given that it was the opposite of previous findings. Not hypothesized but of interest to the study was that observability and difficulty were not correlated. In previous studies, these variables were negatively correlated. Implications of these findings are discussed along with directions for further research.</p>

Name thesis:degree_name

Master of Arts

Discipline thesis:degree_discipline

Department of Psychology

Year

2012

Author dc:creator

• Scott, Jennifer N.

Contributors dc:contributor

• Anthony R. Paquin, Director, Reagan Brown, Aaron Wichman

• Performance rating
• Industrial and Organizational Psychology
• Psychology

Repository record dc:identifier

https://digitalcommons.wku.edu/theses/1152

OAI identifier oai:identifier

oai:digitalcommons.wku.edu:theses-2155