A STUDY TO IDENTIFY FACTORS OF RATER COGNITION FROM DIFFERENT RATER PERSPECTIVES DURING PERFORMANCE-BASED ASSESSMENTS
type: CreativeWork
description: Performance-based assessments (PBA) are used in healthcare education and for the evaluation of performance in professional practice. In nursing, PBAs are used with observation to assess competency of students and professionals. Rating using a PBA is highly complex, and the literature shows high levels of rater variance in rater scores and the ability of a rater to accurately assess and rate a performance. The field of rater cognition studies the variety of cognitive factors influencing a rater’s ability to accurately observe, process, and integrate a judgement about the performance into a scale. The purpose of this mixed-methods study was to explore rater cognition from three perspectives of raters who watched and rated the same performance of nurse practitioner students participating in a telehealth exam with a standardized patient. The student, standardized patient, and advanced practice faculty evaluated the student from their rating perspective through evaluation of the PBA scores, reflection questions, and interviews. A framework developed by Gauthier et al. (2016) was used to guide the exploration of rater cognition. Five student performances were rated by a total of five students, three faculty, and three standardized patients. Each triad of raters completed the PBA, reflection questions, and semi-structured interviews. Quantitative analysis of the PBA scores for each rater triad (scoring the same student performance) was completed using pairwise comparisons, triad comparisons, and individual items. The most disagreement occurred with the student and faculty pair. One of the triads had a higher number of disagreements relative to the other triads, and two PBA items had disagreement with at least three different triads. Qualitative analysis was performed on the reflection questions and interview questions. Keyword analysis was completed on the reflection questions and identified 18 keywords. Content analysis of the interviews found five themes with fifteen cat…
abstract: Performance-based assessments (PBA) are used in healthcare education and for the evaluation of performance in professional practice. In nursing, PBAs are used with observation to assess competency of students and professionals. Rating using a PBA is highly complex, and the literature shows high levels of rater variance in rater scores and the ability of a rater to accurately assess and rate a performance. The field of rater cognition studies the variety of cognitive factors influencing a rater’s ability to accurately observe, process, and integrate a judgement about the performance into a scale. The purpose of this mixed-methods study was to explore rater cognition from three perspectives of raters who watched and rated the same performance of nurse practitioner students participating in a telehealth exam with a standardized patient. The student, standardized patient, and advanced practice faculty evaluated the student from their rating perspective through evaluation of the PBA scores, reflection questions, and interviews. A framework developed by Gauthier et al. (2016) was used to guide the exploration of rater cognition. Five student performances were rated by a total of five students, three faculty, and three standardized patients. Each triad of raters completed the PBA, reflection questions, and semi-structured interviews. Quantitative analysis of the PBA scores for each rater triad (scoring the same student performance) was completed using pairwise comparisons, triad comparisons, and individual items. The most disagreement occurred with the student and faculty pair. One of the triads had a higher number of disagreements relative to the other triads, and two PBA items had disagreement with at least three different triads. Qualitative analysis was performed on the reflection questions and interview questions. Keyword analysis was completed on the reflection questions and identified 18 keywords. Content analysis of the interviews found five themes with fifteen cat…
author: Lewis, Heather A
language: en
keywords: Nursing, Health education, Education, nursing education, PBA, performance-based assessments, rater cognition, student assessment
date published: 2021-01-01
date modified: 2026-07-24T02:44:56Z
provider: University of Kansas
publisher: Networked Digital Library of Theses and Dissertations
type: WebPage
source organization: University of Kansas

PropertyValue
property ID: Handle
value: https://hdl.handle.net/1808/39129