This project explores how to increase the reliability of subjective language assessments. What are the major factors that affect reliability, and what can educators do to maximize it? Research in the field suggests four key factors that have the greatest effect on reliability. The first factor is the procedures used to administer and score the assessment. The more standardized the process is, the higher the reliability will be. The second factor is rater experience, both in the profession and with the assessment in question. Experience is positively correlated with reliability. Novice teachers who get training and feedback in a scoring rubric can become almost as reliable as veteran educators, so training cycles should be a component of any assessment program. The third factor that affects reliability is the type of task the students are asked to do. Tasks that integrate more than one language skill are more reliable than those that use only one skill. The fourth factor is student experience with the testing format. More experience is positively correlated with better achievement, so educators must make sure their students are familiar with the tasks they are asked to do.
","abstract_html":"<p>This project explores how to increase the reliability of subjective language assessments. What are the major factors that affect reliability, and what can educators do to maximize it? Research in the field suggests four key factors that have the greatest effect on reliability. The first factor is the procedures used to administer and score the assessment. The more standardized the process is, the higher the reliability will be. The second factor is rater experience, both in the profession and with the assessment in question. Experience is positively correlated with reliability. Novice teachers who get training and feedback in a scoring rubric can become almost as reliable as veteran educators, so training cycles should be a component of any assessment program. The third factor that affects reliability is the type of task the students are asked to do. Tasks that integrate more than one language skill are more reliable than those that use only one skill. The fourth factor is student experience with the testing format. More experience is positively correlated with better achievement, so educators must make sure their students are familiar with the tasks they are asked to do.</p>","abstract_has_math":false,"creators":["Armes, Jeremy W"],"institution":null,"degree_name":"Master of Arts in Teaching English to Speakers of Other Languages (TESOL)","degree_level":"Project/Capstone - Global access","degree_discipline":"International and Multicultural Education (IME)","degree_department":null,"school":null,"contributors":["Dr. Sedique Popal"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2016,"date_issued":"2016-05-20T07:00:00Z","date_published":"2016-05-20T07:00:00Z","updated_at":"2026-07-24T05:43:31Z","subjects":["Assessment","Reliability","Language Testing","Curriculum and Instruction","Elementary Education and Teaching","International and Comparative Education","Other Teacher Education and Professional Development"],"languages":[],"rights":[],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://repository.usfca.edu/capstone/338","outbound_label":"Repository record","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Dr. Sedique Popal"]},{"key":"dc:creator","label":"Author","values":["Armes, Jeremy W"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.available","label":"Dc Date Available","values":["2016-05-19T07:00:00Z"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["International and Multicultural Education (IME)"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Project/Capstone - Global access"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Master of Arts in Teaching English to Speakers of Other Languages (TESOL)"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Assessment","Reliability","Language Testing","Curriculum and Instruction","Elementary Education and Teaching","International and Comparative Education","Other Teacher Education and Professional Development"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://repository.usfca.edu/capstone/338"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["This project explores how to increase the reliability of subjective language assessments. What are the major factors that affect reliability, and what can educators do to maximize it? Research in the field suggests four key factors that have the greatest effect on reliability. The first factor is the procedures used to administer and score the assessment. The more standardized the process is, the higher the reliability will be. The second factor is rater experience, both in the profession and with the assessment in question. Experience is positively correlated with reliability. Novice teachers who get training and feedback in a scoring rubric can become almost as reliable as veteran educators, so training cycles should be a component of any assessment program. The third factor that affects reliability is the type of task the students are asked to do. Tasks that integrate more than one language skill are more reliable than those that use only one skill. The fourth factor is student experience with the testing format. More experience is positively correlated with better achievement, so educators must make sure their students are familiar with the tasks they are asked to do.
"]},{"key":"dc:title","label":"Title","values":["Quantifying the Qualitative: Increasing the Reliability of Subjective Language Assessments"]}]}],"canonical_facts":{"dc:contributor":["Dr. Sedique Popal"],"dc:creator":["Armes, Jeremy W"],"dc:date.available":["2016-05-19T07:00:00Z"],"dc:description.abstract":["This project explores how to increase the reliability of subjective language assessments. What are the major factors that affect reliability, and what can educators do to maximize it? Research in the field suggests four key factors that have the greatest effect on reliability. The first factor is the procedures used to administer and score the assessment. The more standardized the process is, the higher the reliability will be. The second factor is rater experience, both in the profession and with the assessment in question. Experience is positively correlated with reliability. Novice teachers who get training and feedback in a scoring rubric can become almost as reliable as veteran educators, so training cycles should be a component of any assessment program. The third factor that affects reliability is the type of task the students are asked to do. Tasks that integrate more than one language skill are more reliable than those that use only one skill. The fourth factor is student experience with the testing format. More experience is positively correlated with better achievement, so educators must make sure their students are familiar with the tasks they are asked to do.
"],"dc:identifier":["https://repository.usfca.edu/capstone/338"],"dc:subject":["Assessment","Reliability","Language Testing","Curriculum and Instruction","Elementary Education and Teaching","International and Comparative Education","Other Teacher Education and Professional Development"],"dc:title":["Quantifying the Qualitative: Increasing the Reliability of Subjective Language Assessments"],"thesis:degree_discipline":["International and Multicultural Education (IME)"],"thesis:degree_level":["Project/Capstone - Global access"],"thesis:degree_name":["Master of Arts in Teaching English to Speakers of Other Languages (TESOL)"]},"updated_at":"2026-07-24T05:43:31Z"}