{"id":{"repo_id":"mississippi","oai_identifier":"oai:egrove.olemiss.edu:etd-1462"},"canonical_url":"https://search.dev.ndltd.org/etd/mississippi/oai:egrove.olemiss.edu:etd-1462","repository":{"repo_id":"mississippi","name":"University of Mississippi","base_url":"https://egrove.olemiss.edu/do/oai/"},"display":{"title":"The Effect Of Instrument-Specific Rater Training On Interrater Reliability And Counseling Skills Performance Differentiation","abstract":"The purpose of this study was to explore the effect of instrument-specific rater training on interrater reliability (IRR) and counseling skills performance differentiation. Strong IRR is of primary concern to effective program evaluation (McCullough, Kuhn, Andrews, Valen, Hatch, & Osimo, 2003; Schanche, Nielsen, McCullough, Valen, & Mykletun, 2010) and counselor education (Baker, Daniels, & Greeley, 1990; Jennings, Goh, Skovholt, & Banerje-Steevens, 2003; Lepkowski, Packman, Smaby, & Maddux, 2009). The ability to differentiate between low and high performances of counseling skills is central to informing the classroom instruction of counseling students and the supervision of early clinical experiences (Byrne & Hartley, 2010; Fitch, Gillam, & Baltimore, 2004; Paladino, Barrio-Minton, & Kern, 2011). Participants were randomly assigned to one of four groups defined by whether they received instrument-specific training and the performance level of the counseling skills they assessed. Data was collected using the Universal Counseling Skills Assessment (UCSA) administered traditionally and through the Dynamic Scoring Interface (DSI). The researcher used a 2 X 2 factorial ANOVA, independent samples t-tests, intraclass correlation coefficients, and Fisher’s r to z transformations to analyze the data’s validity across the groups and reliability within the groups. Results that brief instrument-specific training and a structure scoring procedure can significantly strengthen IRR. The results of the analyses are discussed within the context of their implications for counselor education and future research possibilities.","abstract_html":"The purpose of this study was to explore the effect of instrument-specific rater training on interrater reliability (IRR) and counseling skills performance differentiation. Strong IRR is of primary concern to effective program evaluation (McCullough, Kuhn, Andrews, Valen, Hatch, & Osimo, 2003; Schanche, Nielsen, McCullough, Valen, & Mykletun, 2010) and counselor education (Baker, Daniels, & Greeley, 1990; Jennings, Goh, Skovholt, & Banerje-Steevens, 2003; Lepkowski, Packman, Smaby, & Maddux, 2009). The ability to differentiate between low and high performances of counseling skills is central to informing the classroom instruction of counseling students and the supervision of early clinical experiences (Byrne & Hartley, 2010; Fitch, Gillam, & Baltimore, 2004; Paladino, Barrio-Minton, & Kern, 2011). Participants were randomly assigned to one of four groups defined by whether they received instrument-specific training and the performance level of the counseling skills they assessed. Data was collected using the Universal Counseling Skills Assessment (UCSA) administered traditionally and through the Dynamic Scoring Interface (DSI). The researcher used a 2 X 2 factorial ANOVA, independent samples t-tests, intraclass correlation coefficients, and Fisher’s r to z transformations to analyze the data’s validity across the groups and reliability within the groups. Results that brief instrument-specific training and a structure scoring procedure can significantly strengthen IRR. The results of the analyses are discussed within the context of their implications for counselor education and future research possibilities.","abstract_has_math":false,"creators":["Meacham, Paul Douglas"],"institution":null,"degree_name":"Ph.D. in Counselor Education","degree_level":"Dissertation","degree_discipline":"Leadership and Counselor Education","degree_department":null,"school":null,"contributors":["Kevin Stoltz","Marc Showalter","Matthew Reysen"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2013,"date_issued":"2013-01-01T08:00:00Z","date_published":"2013-01-01T08:00:00Z","updated_at":"2026-07-24T03:05:36Z","subjects":["Counseling Skills Assessment","Counseling Skills Development","Interrater Reliability","Counseling Psychology"],"languages":[],"rights":[],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://egrove.olemiss.edu/etd/463","outbound_label":"Repository record","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Kevin Stoltz","Marc Showalter","Matthew Reysen"]},{"key":"dc:creator","label":"Author","values":["Meacham, Paul Douglas"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.available","label":"Dc Date Available","values":["2019-06-27T07:00:00Z"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Leadership and Counselor Education"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Dissertation"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Ph.D. in Counselor Education"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Counseling Skills Assessment","Counseling Skills Development","Interrater Reliability","Counseling Psychology"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://egrove.olemiss.edu/etd/463"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["The purpose of this study was to explore the effect of instrument-specific rater training on interrater reliability (IRR) and counseling skills performance differentiation. Strong IRR is of primary concern to effective program evaluation (McCullough, Kuhn, Andrews, Valen, Hatch, & Osimo, 2003; Schanche, Nielsen, McCullough, Valen, & Mykletun, 2010) and counselor education (Baker, Daniels, & Greeley, 1990; Jennings, Goh, Skovholt, & Banerje-Steevens, 2003; Lepkowski, Packman, Smaby, & Maddux, 2009). The ability to differentiate between low and high performances of counseling skills is central to informing the classroom instruction of counseling students and the supervision of early clinical experiences (Byrne & Hartley, 2010; Fitch, Gillam, & Baltimore, 2004; Paladino, Barrio-Minton, & Kern, 2011). Participants were randomly assigned to one of four groups defined by whether they received instrument-specific training and the performance level of the counseling skills they assessed. Data was collected using the Universal Counseling Skills Assessment (UCSA) administered traditionally and through the Dynamic Scoring Interface (DSI). The researcher used a 2 X 2 factorial ANOVA, independent samples t-tests, intraclass correlation coefficients, and Fisher’s r to z transformations to analyze the data’s validity across the groups and reliability within the groups. Results that brief instrument-specific training and a structure scoring procedure can significantly strengthen IRR. The results of the analyses are discussed within the context of their implications for counselor education and future research possibilities."]},{"key":"dc:title","label":"Title","values":["The Effect Of Instrument-Specific Rater Training On Interrater Reliability And Counseling Skills Performance Differentiation"]}]}],"canonical_facts":{"dc:contributor":["Kevin Stoltz","Marc Showalter","Matthew Reysen"],"dc:creator":["Meacham, Paul Douglas"],"dc:date.available":["2019-06-27T07:00:00Z"],"dc:description.abstract":["The purpose of this study was to explore the effect of instrument-specific rater training on interrater reliability (IRR) and counseling skills performance differentiation. Strong IRR is of primary concern to effective program evaluation (McCullough, Kuhn, Andrews, Valen, Hatch, & Osimo, 2003; Schanche, Nielsen, McCullough, Valen, & Mykletun, 2010) and counselor education (Baker, Daniels, & Greeley, 1990; Jennings, Goh, Skovholt, & Banerje-Steevens, 2003; Lepkowski, Packman, Smaby, & Maddux, 2009). The ability to differentiate between low and high performances of counseling skills is central to informing the classroom instruction of counseling students and the supervision of early clinical experiences (Byrne & Hartley, 2010; Fitch, Gillam, & Baltimore, 2004; Paladino, Barrio-Minton, & Kern, 2011). Participants were randomly assigned to one of four groups defined by whether they received instrument-specific training and the performance level of the counseling skills they assessed. Data was collected using the Universal Counseling Skills Assessment (UCSA) administered traditionally and through the Dynamic Scoring Interface (DSI). The researcher used a 2 X 2 factorial ANOVA, independent samples t-tests, intraclass correlation coefficients, and Fisher’s r to z transformations to analyze the data’s validity across the groups and reliability within the groups. Results that brief instrument-specific training and a structure scoring procedure can significantly strengthen IRR. The results of the analyses are discussed within the context of their implications for counselor education and future research possibilities."],"dc:identifier":["https://egrove.olemiss.edu/etd/463"],"dc:subject":["Counseling Skills Assessment","Counseling Skills Development","Interrater Reliability","Counseling Psychology"],"dc:title":["The Effect Of Instrument-Specific Rater Training On Interrater Reliability And Counseling Skills Performance Differentiation"],"thesis:degree_discipline":["Leadership and Counselor Education"],"thesis:degree_level":["Dissertation"],"thesis:degree_name":["Ph.D. in Counselor Education"]},"updated_at":"2026-07-24T03:05:36Z"}