{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/68972"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/68972","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Techniques for Assessing the Quality of Ratings","abstract":"U of I Only","abstract_html":"U of I Only","abstract_has_math":false,"creators":["Burns, Mary Daugherty"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"Ph.D.","degree_level":"Dissertation","degree_discipline":"Education","degree_department":null,"school":null,"contributors":[],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2014,"date_issued":"2014-12-15T18:50:19Z","date_published":"2014-12-15T18:50:19Z","updated_at":"2026-07-22T22:25:59Z","subjects":["Education, Tests and Measurements"],"languages":[],"rights":[],"rights_urls":[],"identifier_entries":[{"key":"dc:identifier","label":"Identifier","values":["(UMI)AAI8511589"],"render_values":[{"text":"(UMI)AAI8511589","href":null,"code":true}]}]},"links":{"outbound_url":"http://hdl.handle.net/2142/68972","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:creator","label":"Author","values":["Burns, Mary Daugherty"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2014-12-15T18:50:19Z","10000-01-01","1985"]},{"key":"dc:type","label":"Dc Type","values":["text"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Education"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Dissertation"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Ph.D."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Education, Tests and Measurements"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["http://hdl.handle.net/2142/68972","(UMI)AAI8511589"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["U of I Only","232 p.","Thesis (Ph.D.)--University of Illinois at Urbana-Champaign, 1985.","This thesis identified and studied nine analytical techniques used to compute individual rater reliability estimates. These techniques were analyzed to compare reliability estimates, to assess validity, and to determine the effects achieved by modifying the underlying distribution.","To study these analytical techniques, two categories of data were analyzed. The first category consisted of simulated rating data such that individual rater reliabilities were predetermined to serve as the criterion and the underlying distribution was predetermined. Data were simulated for three underlying distributions, normal, bimodal, and negatively skewed. The second category consisted of empirical rating data for "novice" raters and for "expert" raters who served as the criterion.","Analyses of the simulated rating data indicated that the individual rater reliability estimates, computed by each analytical technique, were not comparable to the simulated reliabilities for each rater. This was true regardless of the underlying distribution. The underlying distribution affected the magnitude of the individual rater reliability estimates and the agreement between each analytical technique and the criterion. Generally, the analytical techniques were more in agreement with one another than any one technique was in agreement with the criterion. In terms of rank ordering raters, none of the analytical techniques ranked the raters identically to the predetermined simulation. However, all the techniques, regardless of the underlying distribution were able to identify the two most reliable raters, but were unable to identify the rater with the lowest reliability.","Due to the limited reliability achieved by the "expert" rater criterion, no conclusion was drawn regarding the validity of the individual rater reliability techniques as applied to the empirical rating data.","Made available in DSpace on 2014-12-15T18:50:19Z (GMT). No. of bitstreams: 1 8511589.pdf: 6135462 bytes, checksum: d725f9bfdc4de3919b7d752804b26d26 (MD5) Previous issue date: 1985","Embargo set by: Seth Robbins for item 69138 Lift date: Forever Reason: Restricted to the U of I community idenfinitely during batch ingest of legacy ETDs","Restricted to the U of I community idenfinitely during batch ingest of legacy ETDs"]},{"key":"dc:title","label":"Title","values":["Techniques for Assessing the Quality of Ratings"]}]}],"canonical_facts":{"dc:creator":["Burns, Mary Daugherty"],"dc:date":["2014-12-15T18:50:19Z","10000-01-01","1985"],"dc:description":["U of I Only","232 p.","Thesis (Ph.D.)--University of Illinois at Urbana-Champaign, 1985.","This thesis identified and studied nine analytical techniques used to compute individual rater reliability estimates. These techniques were analyzed to compare reliability estimates, to assess validity, and to determine the effects achieved by modifying the underlying distribution.","To study these analytical techniques, two categories of data were analyzed. The first category consisted of simulated rating data such that individual rater reliabilities were predetermined to serve as the criterion and the underlying distribution was predetermined. Data were simulated for three underlying distributions, normal, bimodal, and negatively skewed. The second category consisted of empirical rating data for "novice" raters and for "expert" raters who served as the criterion.","Analyses of the simulated rating data indicated that the individual rater reliability estimates, computed by each analytical technique, were not comparable to the simulated reliabilities for each rater. This was true regardless of the underlying distribution. The underlying distribution affected the magnitude of the individual rater reliability estimates and the agreement between each analytical technique and the criterion. Generally, the analytical techniques were more in agreement with one another than any one technique was in agreement with the criterion. In terms of rank ordering raters, none of the analytical techniques ranked the raters identically to the predetermined simulation. However, all the techniques, regardless of the underlying distribution were able to identify the two most reliable raters, but were unable to identify the rater with the lowest reliability.","Due to the limited reliability achieved by the "expert" rater criterion, no conclusion was drawn regarding the validity of the individual rater reliability techniques as applied to the empirical rating data.","Made available in DSpace on 2014-12-15T18:50:19Z (GMT). No. of bitstreams: 1 8511589.pdf: 6135462 bytes, checksum: d725f9bfdc4de3919b7d752804b26d26 (MD5) Previous issue date: 1985","Embargo set by: Seth Robbins for item 69138 Lift date: Forever Reason: Restricted to the U of I community idenfinitely during batch ingest of legacy ETDs","Restricted to the U of I community idenfinitely during batch ingest of legacy ETDs"],"dc:identifier":["http://hdl.handle.net/2142/68972","(UMI)AAI8511589"],"dc:subject":["Education, Tests and Measurements"],"dc:title":["Techniques for Assessing the Quality of Ratings"],"dc:type":["text"],"thesis:degree_discipline":["Education"],"thesis:degree_level":["Dissertation"],"thesis:degree_name":["Ph.D."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:25:59Z"}