{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/24224"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/24224","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"\"Validation of an academic listening test: Effects of \"\"breakdown\"\" tests and test takers' cognitive awareness of listening processes\"","abstract":"This study investigated the breakdown effect of a listening comprehension test, whether test takers are affected in comprehending lectures by impediments, and collected test takers’ cognitive awareness on test tasks which contain listening breakdown factors how they perceived these impediments. In this context of the study, a “Breakdown” is a test task that contains impeding factors which could limit test takers’ listening comprehension. Impeding factors used in this study are British accent, fast speech rate, and noise. The listening comprehension test contained four different types of lectures with two topics: dinosaurs and weather changes. In total, ninety-six test takers took the test and each group of twenty-four test takers were assigned to following test groups: the “regular”, the “British”, the “speech rate”, and the “noise” groups. The name of each group represented the source of interference or impediment in the listening task. Along with test takers’ performance on listening test, the design of the study was informed by test takers’ responses to a questionnaire on their cognitive awareness of listening, metacognitive process, and test-taking strategy uses. Both quantitative and qualitative analyses were used on test scores and responses on questionnaire. To estimate validity and reliability of the test, several quantitative analyses were conducted. Cronbach’s alphas and confidence intervals of alphas were calculated for each test and the coefficient values were observed to vary among the tests. A Generalizability study and an analysis of variance (ANOVA) were conducted to measure the breakdown effect on the test. The result of generalizability study showed that variance components for interaction between person and items were larger than variance components for person, which indicated confounding effect. However, the analysis of variance showed that breakdown effect on test was not detected, but the effect of talk types was significantly different. Classical item analyses were also iii conducted to observe differences on item functions among four tests. A few items were flagged as easy items, however difficulty and discrimination indices were varied among groups. To understand test takers’ perspectives on second language listening, both quantitative and qualitative analyses were conducted. Test takers’ responses on questionnaire showed test takers from different groups demonstrated similar metacognitive strategy uses, monitoring behaviors, and test-taking strategies. Findings showed that there was no intentional breakdown effect on all groups. Low reliability coefficient and generalizability analysis showed that there was insignificant evidence to support breakdown effect of three impediments on test takers’ listening proficiency. However, one of impediments, noise functioned as an impediment in the test. Some test takers from the noise group reported that their comprehension was impeded by noise. At the test level, talk type was identified as breakdown factor from both quantitative and qualitative analyses. At item level, regardless of item difficulty indices, lower item discrimination indices indicated that items functioned differently when an impediment was added in the test. Listening passages functioned as a more influential factor in comprehension than the breakdown factors. Test takers who already had some background knowledge could perform better than those who do not have the knowledge. Furthermore, test takers with higher scores were able to use various types of metacognitive process than those with lower scores regardless of groups. This study suggests that several considerations are needed to make authentic listening tasks when listening impediments are included in the second language test development such as degrees of impediments, threshold of the breakdown factor, and breakdown effects on both item iv and test levels. Moreover, metacognitive process embedded listening items could contribute to understand second language learners’ listening comprehension.","abstract_html":"This study investigated the breakdown effect of a listening comprehension test, whether test takers are affected in comprehending lectures by impediments, and collected test takers’ cognitive awareness on test tasks which contain listening breakdown factors how they perceived these impediments. In this context of the study, a “Breakdown” is a test task that contains impeding factors which could limit test takers’ listening comprehension. Impeding factors used in this study are British accent, fast speech rate, and noise. The listening comprehension test contained four different types of lectures with two topics: dinosaurs and weather changes. In total, ninety-six test takers took the test and each group of twenty-four test takers were assigned to following test groups: the “regular”, the “British”, the “speech rate”, and the “noise” groups. The name of each group represented the source of interference or impediment in the listening task. Along with test takers’ performance on listening test, the design of the study was informed by test takers’ responses to a questionnaire on their cognitive awareness of listening, metacognitive process, and test-taking strategy uses. Both quantitative and qualitative analyses were used on test scores and responses on questionnaire. To estimate validity and reliability of the test, several quantitative analyses were conducted. Cronbach’s alphas and confidence intervals of alphas were calculated for each test and the coefficient values were observed to vary among the tests. A Generalizability study and an analysis of variance (ANOVA) were conducted to measure the breakdown effect on the test. The result of generalizability study showed that variance components for interaction between person and items were larger than variance components for person, which indicated confounding effect. However, the analysis of variance showed that breakdown effect on test was not detected, but the effect of talk types was significantly different. Classical item analyses were also iii conducted to observe differences on item functions among four tests. A few items were flagged as easy items, however difficulty and discrimination indices were varied among groups. To understand test takers’ perspectives on second language listening, both quantitative and qualitative analyses were conducted. Test takers’ responses on questionnaire showed test takers from different groups demonstrated similar metacognitive strategy uses, monitoring behaviors, and test-taking strategies. Findings showed that there was no intentional breakdown effect on all groups. Low reliability coefficient and generalizability analysis showed that there was insignificant evidence to support breakdown effect of three impediments on test takers’ listening proficiency. However, one of impediments, noise functioned as an impediment in the test. Some test takers from the noise group reported that their comprehension was impeded by noise. At the test level, talk type was identified as breakdown factor from both quantitative and qualitative analyses. At item level, regardless of item difficulty indices, lower item discrimination indices indicated that items functioned differently when an impediment was added in the test. Listening passages functioned as a more influential factor in comprehension than the breakdown factors. Test takers who already had some background knowledge could perform better than those who do not have the knowledge. Furthermore, test takers with higher scores were able to use various types of metacognitive process than those with lower scores regardless of groups. This study suggests that several considerations are needed to make authentic listening tasks when listening impediments are included in the second language test development such as degrees of impediments, threshold of the breakdown factor, and breakdown effects on both item iv and test levels. Moreover, metacognitive process embedded listening items could contribute to understand second language learners’ listening comprehension.","abstract_has_math":false,"creators":["Chi, Youngshin"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"Ph.D.","degree_level":"Dissertation","degree_discipline":"Educational Psychology","degree_department":null,"school":null,"contributors":["Davidson, Frederick G.","Anderson, Carolyn J.","Grindrod, Christopher M.","Golato, Peter"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2011,"date_issued":"2011-05-25T14:55:30Z","date_published":"2011-05-25T14:55:30Z","updated_at":"2026-07-22T22:25:23Z","subjects":["language testing","validation","test development","second language listening"],"languages":["en"],"rights":["Copyright 2011 Youngshin Chi"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"http://hdl.handle.net/2142/24224","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Davidson, Frederick G.","Anderson, Carolyn J.","Grindrod, Christopher M.","Golato, Peter"]},{"key":"dc:creator","label":"Author","values":["Chi, Youngshin"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2011-05-25T14:55:30Z","2013-06-25T10:00:16Z","2011-05"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Educational Psychology"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Dissertation"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Ph.D."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["language testing","validation","test development","second language listening"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2011 Youngshin Chi"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["http://hdl.handle.net/2142/24224"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["This study investigated the breakdown effect of a listening comprehension test, whether test takers are affected in comprehending lectures by impediments, and collected test takers’ cognitive awareness on test tasks which contain listening breakdown factors how they perceived these impediments. In this context of the study, a “Breakdown” is a test task that contains impeding factors which could limit test takers’ listening comprehension. Impeding factors used in this study are British accent, fast speech rate, and noise. The listening comprehension test contained four different types of lectures with two topics: dinosaurs and weather changes. In total, ninety-six test takers took the test and each group of twenty-four test takers were assigned to following test groups: the “regular”, the “British”, the “speech rate”, and the “noise” groups. The name of each group represented the source of interference or impediment in the listening task. Along with test takers’ performance on listening test, the design of the study was informed by test takers’ responses to a questionnaire on their cognitive awareness of listening, metacognitive process, and test-taking strategy uses. Both quantitative and qualitative analyses were used on test scores and responses on questionnaire. To estimate validity and reliability of the test, several quantitative analyses were conducted. Cronbach’s alphas and confidence intervals of alphas were calculated for each test and the coefficient values were observed to vary among the tests. A Generalizability study and an analysis of variance (ANOVA) were conducted to measure the breakdown effect on the test. The result of generalizability study showed that variance components for interaction between person and items were larger than variance components for person, which indicated confounding effect. However, the analysis of variance showed that breakdown effect on test was not detected, but the effect of talk types was significantly different. Classical item analyses were also iii conducted to observe differences on item functions among four tests. A few items were flagged as easy items, however difficulty and discrimination indices were varied among groups. To understand test takers’ perspectives on second language listening, both quantitative and qualitative analyses were conducted. Test takers’ responses on questionnaire showed test takers from different groups demonstrated similar metacognitive strategy uses, monitoring behaviors, and test-taking strategies. Findings showed that there was no intentional breakdown effect on all groups. Low reliability coefficient and generalizability analysis showed that there was insignificant evidence to support breakdown effect of three impediments on test takers’ listening proficiency. However, one of impediments, noise functioned as an impediment in the test. Some test takers from the noise group reported that their comprehension was impeded by noise. At the test level, talk type was identified as breakdown factor from both quantitative and qualitative analyses. At item level, regardless of item difficulty indices, lower item discrimination indices indicated that items functioned differently when an impediment was added in the test. Listening passages functioned as a more influential factor in comprehension than the breakdown factors. Test takers who already had some background knowledge could perform better than those who do not have the knowledge. Furthermore, test takers with higher scores were able to use various types of metacognitive process than those with lower scores regardless of groups. This study suggests that several considerations are needed to make authentic listening tasks when listening impediments are included in the second language test development such as degrees of impediments, threshold of the breakdown factor, and breakdown effects on both item iv and test levels. Moreover, metacognitive process embedded listening items could contribute to understand second language learners’ listening comprehension.","Item withdrawn by Rebecca Bryant (rabryant@illinois.edu) on 2011-04-21T19:30:04Z Item was in collections: University of Illinois Theses & Dissertations (ID: 1) No. of bitstreams: 1 Chi_Youngshin.pdf: 1131537 bytes, checksum: 20567f2be388497c7a257d1a70510d0c (MD5)","Made available in DSpace on 2011-05-25T14:55:30Z (GMT). No. of bitstreams: 2 Chi_Youngshin.pdf: 1144136 bytes, checksum: 93c950446f19f9a98e44e41a9e6f2bfb (MD5) license.txt: 4060 bytes, checksum: d4168b6fac621d69b433a2048157cd98 (MD5)","Item marked as restricted to the 'UIUC Users [automated]' Group (id=2) by William Ingram (wingram2@illinois.edu) on 2011-06-24T19:12:37Z Item is restricted until 2013-06-24T19:12:37Z","Item reinstated by Sarah Shreeves (sshreeve@illinois.edu) on 2013-06-25T10:00:15Z Item was in collections: University of Illinois Dissertations and Theses (ID: 204) College of Education Dissertations and Theses (ID: 434) No. of bitstreams: 2 Chi_Youngshin.pdf: 1144136 bytes, checksum: 93c950446f19f9a98e44e41a9e6f2bfb (MD5) license.txt: 4060 bytes, checksum: d4168b6fac621d69b433a2048157cd98 (MD5)","Item released from any restrictions by Sarah Shreeves (sshreeve@illinois.edu) on 2013-06-25T10:00:16Z"]},{"key":"dc:title","label":"Title","values":["\"Validation of an academic listening test: Effects of \"\"breakdown\"\" tests and test takers' cognitive awareness of listening processes\""]}]}],"canonical_facts":{"dc:contributor":["Davidson, Frederick G.","Anderson, Carolyn J.","Grindrod, Christopher M.","Golato, Peter"],"dc:creator":["Chi, Youngshin"],"dc:date":["2011-05-25T14:55:30Z","2013-06-25T10:00:16Z","2011-05"],"dc:description":["This study investigated the breakdown effect of a listening comprehension test, whether test takers are affected in comprehending lectures by impediments, and collected test takers’ cognitive awareness on test tasks which contain listening breakdown factors how they perceived these impediments. In this context of the study, a “Breakdown” is a test task that contains impeding factors which could limit test takers’ listening comprehension. Impeding factors used in this study are British accent, fast speech rate, and noise. The listening comprehension test contained four different types of lectures with two topics: dinosaurs and weather changes. In total, ninety-six test takers took the test and each group of twenty-four test takers were assigned to following test groups: the “regular”, the “British”, the “speech rate”, and the “noise” groups. The name of each group represented the source of interference or impediment in the listening task. Along with test takers’ performance on listening test, the design of the study was informed by test takers’ responses to a questionnaire on their cognitive awareness of listening, metacognitive process, and test-taking strategy uses. Both quantitative and qualitative analyses were used on test scores and responses on questionnaire. To estimate validity and reliability of the test, several quantitative analyses were conducted. Cronbach’s alphas and confidence intervals of alphas were calculated for each test and the coefficient values were observed to vary among the tests. A Generalizability study and an analysis of variance (ANOVA) were conducted to measure the breakdown effect on the test. The result of generalizability study showed that variance components for interaction between person and items were larger than variance components for person, which indicated confounding effect. However, the analysis of variance showed that breakdown effect on test was not detected, but the effect of talk types was significantly different. Classical item analyses were also iii conducted to observe differences on item functions among four tests. A few items were flagged as easy items, however difficulty and discrimination indices were varied among groups. To understand test takers’ perspectives on second language listening, both quantitative and qualitative analyses were conducted. Test takers’ responses on questionnaire showed test takers from different groups demonstrated similar metacognitive strategy uses, monitoring behaviors, and test-taking strategies. Findings showed that there was no intentional breakdown effect on all groups. Low reliability coefficient and generalizability analysis showed that there was insignificant evidence to support breakdown effect of three impediments on test takers’ listening proficiency. However, one of impediments, noise functioned as an impediment in the test. Some test takers from the noise group reported that their comprehension was impeded by noise. At the test level, talk type was identified as breakdown factor from both quantitative and qualitative analyses. At item level, regardless of item difficulty indices, lower item discrimination indices indicated that items functioned differently when an impediment was added in the test. Listening passages functioned as a more influential factor in comprehension than the breakdown factors. Test takers who already had some background knowledge could perform better than those who do not have the knowledge. Furthermore, test takers with higher scores were able to use various types of metacognitive process than those with lower scores regardless of groups. This study suggests that several considerations are needed to make authentic listening tasks when listening impediments are included in the second language test development such as degrees of impediments, threshold of the breakdown factor, and breakdown effects on both item iv and test levels. Moreover, metacognitive process embedded listening items could contribute to understand second language learners’ listening comprehension.","Item withdrawn by Rebecca Bryant (rabryant@illinois.edu) on 2011-04-21T19:30:04Z Item was in collections: University of Illinois Theses & Dissertations (ID: 1) No. of bitstreams: 1 Chi_Youngshin.pdf: 1131537 bytes, checksum: 20567f2be388497c7a257d1a70510d0c (MD5)","Made available in DSpace on 2011-05-25T14:55:30Z (GMT). No. of bitstreams: 2 Chi_Youngshin.pdf: 1144136 bytes, checksum: 93c950446f19f9a98e44e41a9e6f2bfb (MD5) license.txt: 4060 bytes, checksum: d4168b6fac621d69b433a2048157cd98 (MD5)","Item marked as restricted to the 'UIUC Users [automated]' Group (id=2) by William Ingram (wingram2@illinois.edu) on 2011-06-24T19:12:37Z Item is restricted until 2013-06-24T19:12:37Z","Item reinstated by Sarah Shreeves (sshreeve@illinois.edu) on 2013-06-25T10:00:15Z Item was in collections: University of Illinois Dissertations and Theses (ID: 204) College of Education Dissertations and Theses (ID: 434) No. of bitstreams: 2 Chi_Youngshin.pdf: 1144136 bytes, checksum: 93c950446f19f9a98e44e41a9e6f2bfb (MD5) license.txt: 4060 bytes, checksum: d4168b6fac621d69b433a2048157cd98 (MD5)","Item released from any restrictions by Sarah Shreeves (sshreeve@illinois.edu) on 2013-06-25T10:00:16Z"],"dc:identifier":["http://hdl.handle.net/2142/24224"],"dc:language":["en"],"dc:rights":["Copyright 2011 Youngshin Chi"],"dc:subject":["language testing","validation","test development","second language listening"],"dc:title":["\"Validation of an academic listening test: Effects of \"\"breakdown\"\" tests and test takers' cognitive awareness of listening processes\""],"thesis:degree_discipline":["Educational Psychology"],"thesis:degree_level":["Dissertation"],"thesis:degree_name":["Ph.D."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:25:23Z"}