{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/42124"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/42124","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Implementing a cognitive diagnostic assessment in an institutional test: a new networking model in language testing and experiment with a new psychometric model and task type","abstract":"This dissertation is based on two major projects, cognitive diagnostic model comparison and implementing cognitive diagnostic assessment in an institutional test. In the first project, five cognitive diagnostic models are empirically compared for language test data under a unified general modeling framework. The models are applied to three sets of data, TOEFL Reading, TOEFL Listening and ECPE Grammar and examined in terms of their model fit to the data and functioning. The item-association root mean squared error values and multiple information criteria all indicate that the general model (LCDM) and the compensatory-RUM model are the best fit to all three test data sets used. The functioning of the models examined through multiple indices also unanimously confirms these fit indices. Based on these results, a discussion follows to argue that the general modeling framework is optimal for language assessment data due to its much greater flexibility. The behaviors of the compensatory RUM and non-compensatory RUM (Fusion) models are also compared and the differences are analyzed. Cognitive diagnostic assessment (CDA) has gained attention in language testing since late 90’s. A few models of this new assessment method have been applied to the response data of different language tests, yielding encouraging results in general. Most of these empirical studies used large-scale, standardized tests and retrofitted to these existing tests. The second research study in this proposal was an effort to go beyond this limited research context of previous studies of CDA in language testing. Using a new psychometric cognitive diagnostic model (Log-linear cognitive diagnostic model) and task type (elicited imitation task) as well as constructing the EIT test with a CDA implementation in mind from the outset (thus not retrofitting), this project tried to implement a cognitive diagnostic assessment in an institutional placement test. In doing so, the study employed a statistical method (resampling technique) as a way to resolve the issue caused by a small size of a test-taker pool at an institution, which is usually smaller than the optimal size for CDA implementation. It also tried to analyze polytomously scored response data. The study achieved a success at estimating with polytomous response data that were scored with a three-point scale (i.e., zero to two points). Though it was a limited success (in that more complex rating scale could not be analyzed), it was the first success in estimating with polytomous response data in the context of CDA research in language testing. The analysis results of the study also provide many insights and implications for the process of Q-matrix construction, grain size of attributes, appropriate task types and item types for cognitive diagnostic assessment as well as appropriate cognitive psychometric models for differing contexts of CDA implementation. The study also rediscovers or introduces the usefulness of bootstrap resampling method as an approach that is gaining popularity even in areas where only traditional quantitative methods are usually employed. Also, the networking occurred in this project between students in different specializations could be established as a new networking model in language testing. Considering such a collaboration is very much needed for implementing a relatively new measurement method in a specific knowledge domain, the co-work attempted in this project could serve as a model for implementing CDA in language testing.","abstract_html":"This dissertation is based on two major projects, cognitive diagnostic model comparison and implementing cognitive diagnostic assessment in an institutional test. In the first project, five cognitive diagnostic models are empirically compared for language test data under a unified general modeling framework. The models are applied to three sets of data, TOEFL Reading, TOEFL Listening and ECPE Grammar and examined in terms of their model fit to the data and functioning. The item-association root mean squared error values and multiple information criteria all indicate that the general model (LCDM) and the compensatory-RUM model are the best fit to all three test data sets used. The functioning of the models examined through multiple indices also unanimously confirms these fit indices. Based on these results, a discussion follows to argue that the general modeling framework is optimal for language assessment data due to its much greater flexibility. The behaviors of the compensatory RUM and non-compensatory RUM (Fusion) models are also compared and the differences are analyzed. Cognitive diagnostic assessment (CDA) has gained attention in language testing since late 90’s. A few models of this new assessment method have been applied to the response data of different language tests, yielding encouraging results in general. Most of these empirical studies used large-scale, standardized tests and retrofitted to these existing tests. The second research study in this proposal was an effort to go beyond this limited research context of previous studies of CDA in language testing. Using a new psychometric cognitive diagnostic model (Log-linear cognitive diagnostic model) and task type (elicited imitation task) as well as constructing the EIT test with a CDA implementation in mind from the outset (thus not retrofitting), this project tried to implement a cognitive diagnostic assessment in an institutional placement test. In doing so, the study employed a statistical method (resampling technique) as a way to resolve the issue caused by a small size of a test-taker pool at an institution, which is usually smaller than the optimal size for CDA implementation. It also tried to analyze polytomously scored response data. The study achieved a success at estimating with polytomous response data that were scored with a three-point scale (i.e., zero to two points). Though it was a limited success (in that more complex rating scale could not be analyzed), it was the first success in estimating with polytomous response data in the context of CDA research in language testing. The analysis results of the study also provide many insights and implications for the process of Q-matrix construction, grain size of attributes, appropriate task types and item types for cognitive diagnostic assessment as well as appropriate cognitive psychometric models for differing contexts of CDA implementation. The study also rediscovers or introduces the usefulness of bootstrap resampling method as an approach that is gaining popularity even in areas where only traditional quantitative methods are usually employed. Also, the networking occurred in this project between students in different specializations could be established as a new networking model in language testing. Considering such a collaboration is very much needed for implementing a relatively new measurement method in a specific knowledge domain, the co-work attempted in this project could serve as a model for implementing CDA in language testing.","abstract_has_math":false,"creators":["Yi, Yeonsook"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"Ph.D.","degree_level":"Dissertation","degree_discipline":"Educational Psychology","degree_department":null,"school":null,"contributors":["Davidson, Frederick G.","Zhang, Jinming","Bowles, Melissa A.","Templin, Jonathan","Tremblay, Annie"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2013,"date_issued":"2013-02-03T19:16:27Z","date_published":"2013-02-03T19:16:27Z","updated_at":"2026-07-22T22:25:31Z","subjects":["cognitive diagnostic assessment","cognitive diagnostic model comparison","general modeling framework","language test data","Test of English as a Foreign Language (TOEFL) Reading and Listening","university French placement test","elicited imitation task","resampling","polytomous","networking in language testing","Examination for the Certificate of Proficiency in English (ECPE) Grammar"],"languages":["en"],"rights":["Copyright 2012 Yeonsook Yi"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"http://hdl.handle.net/2142/42124","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Davidson, Frederick G.","Zhang, Jinming","Bowles, Melissa A.","Templin, Jonathan","Tremblay, Annie"]},{"key":"dc:creator","label":"Author","values":["Yi, Yeonsook"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2013-02-03T19:16:27Z","2015-02-03T11:00:32Z","2012-12"]},{"key":"dc:type","label":"Dc Type","values":["text"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Educational Psychology"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Dissertation"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Ph.D."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["cognitive diagnostic assessment","cognitive diagnostic model comparison","general modeling framework","language test data","Test of English as a Foreign Language (TOEFL) Reading and Listening","university French placement test","elicited imitation task","resampling","polytomous","networking in language testing","Examination for the Certificate of Proficiency in English (ECPE) Grammar"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2012 Yeonsook Yi"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["http://hdl.handle.net/2142/42124"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["This dissertation is based on two major projects, cognitive diagnostic model comparison and implementing cognitive diagnostic assessment in an institutional test. In the first project, five cognitive diagnostic models are empirically compared for language test data under a unified general modeling framework. The models are applied to three sets of data, TOEFL Reading, TOEFL Listening and ECPE Grammar and examined in terms of their model fit to the data and functioning. The item-association root mean squared error values and multiple information criteria all indicate that the general model (LCDM) and the compensatory-RUM model are the best fit to all three test data sets used. The functioning of the models examined through multiple indices also unanimously confirms these fit indices. Based on these results, a discussion follows to argue that the general modeling framework is optimal for language assessment data due to its much greater flexibility. The behaviors of the compensatory RUM and non-compensatory RUM (Fusion) models are also compared and the differences are analyzed. Cognitive diagnostic assessment (CDA) has gained attention in language testing since late 90’s. A few models of this new assessment method have been applied to the response data of different language tests, yielding encouraging results in general. Most of these empirical studies used large-scale, standardized tests and retrofitted to these existing tests. The second research study in this proposal was an effort to go beyond this limited research context of previous studies of CDA in language testing. Using a new psychometric cognitive diagnostic model (Log-linear cognitive diagnostic model) and task type (elicited imitation task) as well as constructing the EIT test with a CDA implementation in mind from the outset (thus not retrofitting), this project tried to implement a cognitive diagnostic assessment in an institutional placement test. In doing so, the study employed a statistical method (resampling technique) as a way to resolve the issue caused by a small size of a test-taker pool at an institution, which is usually smaller than the optimal size for CDA implementation. It also tried to analyze polytomously scored response data. The study achieved a success at estimating with polytomous response data that were scored with a three-point scale (i.e., zero to two points). Though it was a limited success (in that more complex rating scale could not be analyzed), it was the first success in estimating with polytomous response data in the context of CDA research in language testing. The analysis results of the study also provide many insights and implications for the process of Q-matrix construction, grain size of attributes, appropriate task types and item types for cognitive diagnostic assessment as well as appropriate cognitive psychometric models for differing contexts of CDA implementation. The study also rediscovers or introduces the usefulness of bootstrap resampling method as an approach that is gaining popularity even in areas where only traditional quantitative methods are usually employed. Also, the networking occurred in this project between students in different specializations could be established as a new networking model in language testing. Considering such a collaboration is very much needed for implementing a relatively new measurement method in a specific knowledge domain, the co-work attempted in this project could serve as a model for implementing CDA in language testing.","Item withdrawn by Mark Zulauf (zulauf@illinois.edu) on 2012-12-07T13:54:19Z Item was in collections: University of Illinois Theses & Dissertations (ID: 1) No. of bitstreams: 1 Yi_Yeonsook.pdf: 2634008 bytes, checksum: e8d4e1ecfce599960fefeebf2036cdc4 (MD5)","Made available in DSpace on 2013-02-03T19:16:27Z (GMT). No. of bitstreams: 2 Yeonsook_Yi.pdf: 2810469 bytes, checksum: c5d8631f2b2714cb4a92e7f419bff55a (MD5) license.txt: 4060 bytes, checksum: 16b773af5d93263bfb1db73ba19b840b (MD5)","Restriction data tranferred 2014-07-01T11:11:49-05:00 Original Data Group with Access UIUC Users [automated] Release Date: 2015-02-03 13:18:53 UTC Reason: Author requested U of Illinois access only (OA after 2yrs) in Vireo ETD system","Item marked as restricted to the 'UIUC Users [automated]' Group (id=2) by Seth Robbins (srobbins@illinois.edu) on 2013-02-03T19:18:55Z Item is restricted until 2015-02-03T19:18:53Z","U of I Only Restriction Lifted for Item 42071 on 2015-02-03T11:00:32Z."]},{"key":"dc:title","label":"Title","values":["Implementing a cognitive diagnostic assessment in an institutional test: a new networking model in language testing and experiment with a new psychometric model and task type"]}]}],"canonical_facts":{"dc:contributor":["Davidson, Frederick G.","Zhang, Jinming","Bowles, Melissa A.","Templin, Jonathan","Tremblay, Annie"],"dc:creator":["Yi, Yeonsook"],"dc:date":["2013-02-03T19:16:27Z","2015-02-03T11:00:32Z","2012-12"],"dc:description":["This dissertation is based on two major projects, cognitive diagnostic model comparison and implementing cognitive diagnostic assessment in an institutional test. In the first project, five cognitive diagnostic models are empirically compared for language test data under a unified general modeling framework. The models are applied to three sets of data, TOEFL Reading, TOEFL Listening and ECPE Grammar and examined in terms of their model fit to the data and functioning. The item-association root mean squared error values and multiple information criteria all indicate that the general model (LCDM) and the compensatory-RUM model are the best fit to all three test data sets used. The functioning of the models examined through multiple indices also unanimously confirms these fit indices. Based on these results, a discussion follows to argue that the general modeling framework is optimal for language assessment data due to its much greater flexibility. The behaviors of the compensatory RUM and non-compensatory RUM (Fusion) models are also compared and the differences are analyzed. Cognitive diagnostic assessment (CDA) has gained attention in language testing since late 90’s. A few models of this new assessment method have been applied to the response data of different language tests, yielding encouraging results in general. Most of these empirical studies used large-scale, standardized tests and retrofitted to these existing tests. The second research study in this proposal was an effort to go beyond this limited research context of previous studies of CDA in language testing. Using a new psychometric cognitive diagnostic model (Log-linear cognitive diagnostic model) and task type (elicited imitation task) as well as constructing the EIT test with a CDA implementation in mind from the outset (thus not retrofitting), this project tried to implement a cognitive diagnostic assessment in an institutional placement test. In doing so, the study employed a statistical method (resampling technique) as a way to resolve the issue caused by a small size of a test-taker pool at an institution, which is usually smaller than the optimal size for CDA implementation. It also tried to analyze polytomously scored response data. The study achieved a success at estimating with polytomous response data that were scored with a three-point scale (i.e., zero to two points). Though it was a limited success (in that more complex rating scale could not be analyzed), it was the first success in estimating with polytomous response data in the context of CDA research in language testing. The analysis results of the study also provide many insights and implications for the process of Q-matrix construction, grain size of attributes, appropriate task types and item types for cognitive diagnostic assessment as well as appropriate cognitive psychometric models for differing contexts of CDA implementation. The study also rediscovers or introduces the usefulness of bootstrap resampling method as an approach that is gaining popularity even in areas where only traditional quantitative methods are usually employed. Also, the networking occurred in this project between students in different specializations could be established as a new networking model in language testing. Considering such a collaboration is very much needed for implementing a relatively new measurement method in a specific knowledge domain, the co-work attempted in this project could serve as a model for implementing CDA in language testing.","Item withdrawn by Mark Zulauf (zulauf@illinois.edu) on 2012-12-07T13:54:19Z Item was in collections: University of Illinois Theses & Dissertations (ID: 1) No. of bitstreams: 1 Yi_Yeonsook.pdf: 2634008 bytes, checksum: e8d4e1ecfce599960fefeebf2036cdc4 (MD5)","Made available in DSpace on 2013-02-03T19:16:27Z (GMT). No. of bitstreams: 2 Yeonsook_Yi.pdf: 2810469 bytes, checksum: c5d8631f2b2714cb4a92e7f419bff55a (MD5) license.txt: 4060 bytes, checksum: 16b773af5d93263bfb1db73ba19b840b (MD5)","Restriction data tranferred 2014-07-01T11:11:49-05:00 Original Data Group with Access UIUC Users [automated] Release Date: 2015-02-03 13:18:53 UTC Reason: Author requested U of Illinois access only (OA after 2yrs) in Vireo ETD system","Item marked as restricted to the 'UIUC Users [automated]' Group (id=2) by Seth Robbins (srobbins@illinois.edu) on 2013-02-03T19:18:55Z Item is restricted until 2015-02-03T19:18:53Z","U of I Only Restriction Lifted for Item 42071 on 2015-02-03T11:00:32Z."],"dc:identifier":["http://hdl.handle.net/2142/42124"],"dc:language":["en"],"dc:rights":["Copyright 2012 Yeonsook Yi"],"dc:subject":["cognitive diagnostic assessment","cognitive diagnostic model comparison","general modeling framework","language test data","Test of English as a Foreign Language (TOEFL) Reading and Listening","university French placement test","elicited imitation task","resampling","polytomous","networking in language testing","Examination for the Certificate of Proficiency in English (ECPE) Grammar"],"dc:title":["Implementing a cognitive diagnostic assessment in an institutional test: a new networking model in language testing and experiment with a new psychometric model and task type"],"dc:type":["text"],"thesis:degree_discipline":["Educational Psychology"],"thesis:degree_level":["Dissertation"],"thesis:degree_name":["Ph.D."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:25:31Z"}