The Effect Of Instrument-Specific Rater Training On Interrater Reliability And Counseling Skills Performance Differentiation

The purpose of this study was to explore the effect of instrument-specific rater training on interrater reliability (IRR) and counseling skills performance differentiation. Strong IRR is of primary concern to effective program evaluation (McCullough, Kuhn, Andrews, Valen, Hatch, & Osimo, 2003; Schanche, Nielsen, McCullough, Valen, & Mykletun, 2010) and counselor education (Baker, Daniels, & Greeley, 1990; Jennings, Goh, Skovholt, & Banerje-Steevens, 2003; Lepkowski, Packman, Smaby, & Maddux, 2009). The ability to differentiate between low and high performances of counseling skills is central to informing the classroom instruction of counseling students and the supervision of early clinical experiences (Byrne & Hartley, 2010; Fitch, Gillam, & Baltimore, 2004; Paladino, Barrio-Minton, & Kern, 2011). Participants were randomly assigned to one of four groups defined by whether they received instrument-specific training and the performance level of the counseling skills they assessed. Data was collected using the Universal Counseling Skills Assessment (UCSA) administered traditionally and through the Dynamic Scoring Interface (DSI). The researcher used a 2 X 2 factorial ANOVA, independent samples t-tests, intraclass correlation coefficients, and Fisher’s r to z transformations to analyze the data’s validity across the groups and reliability within the groups. Results that brief instrument-specific training and a structure scoring procedure can significantly strengthen IRR. The results of the analyses are discussed within the context of their implications for counselor education and future research possibilities.

Name thesis:degree_name

Ph.D. in Counselor Education

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Leadership and Counselor Education

Year dc:date.available

2013

Author dc:creator

• Meacham, Paul Douglas

Contributors dc:contributor

• Kevin Stoltz
• Marc Showalter
• Matthew Reysen

• Counseling Skills Assessment
• Counseling Skills Development
• Interrater Reliability
• Counseling Psychology

Repository record dc:identifier

https://egrove.olemiss.edu/etd/463

OAI identifier oai:identifier

oai:egrove.olemiss.edu:etd-1462