{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/98150"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/98150","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Q-matrix optimization for cognitive diagnostic assessment","abstract":"Cognitive diagnostic assessment is a growing area in psychological and educational measurement with the advantage of providing diagnostic profiles (mastery/non-mastery of measured attributes) for examinees, giving insights for classroom teaching and student learning. Central to the successful implementation of a cognitive diagnostic assessment is the Q-matrix, the structure that specifies item-attribute relationships. However, the Q-matrix is prone to misspecification, given that it is often constructed based solely on human opinions. This thesis uses three research studies to investigate key issues of Q-matrix optimization for cognitive diagnostic assessments. The first study investigates the effects of Q-matrix misspecification on the classification accuracy and consistency of diagnostic results. The two types of Q-matrix misspecifications examined are Q-entry misspecification (which includes three levels of misspecification: 10%, 20% and 30%), and attribute misspecification (which includes attribute exclusion and attribute inclusion). The results of a simulation study show that both Q-entry and attribute misspecification significantly deteriorate the accuracy of classification and the consistency of diagnostic results. In addition, the two classification accuracy and consistency indices have the potential to be useful in identifying possible attribute misspecification (e.g., attribute inclusion) of Q-matrix in empirical analyses. The second study provides a systematic performance evaluation of the three most commonly used Q-matrix validation methods: the sequential EM based δ-method (de la Torre, 2008), the Bayesian estimation method (DeCarlo, 2012), and the nonparametric Q-matrix refinement method (Chiu, 2013), with both basic and complex assessment design factors. The results of two simulation studies reveal that the Bayesian estimation method outperforms the other two methods in terms of recovering the misspecified Q-entries across various conditions. The performance of the three Q-matrix validation methods is also affected to different degrees by various assessment design factors, among which the data generation model is the most critical. The third study proposes a two-stage cross-validation method that combines the strengths of the nonparametric refinement method and Bayesian estimation techniques for improving Q-matrix validation accuracy and computation efficiency. The results show that the proposed method can effectively optimize Q-matrices that are possibly misspecified in both simulation and empirical data settings.","abstract_html":"Cognitive diagnostic assessment is a growing area in psychological and educational measurement with the advantage of providing diagnostic profiles (mastery/non-mastery of measured attributes) for examinees, giving insights for classroom teaching and student learning. Central to the successful implementation of a cognitive diagnostic assessment is the Q-matrix, the structure that specifies item-attribute relationships. However, the Q-matrix is prone to misspecification, given that it is often constructed based solely on human opinions. This thesis uses three research studies to investigate key issues of Q-matrix optimization for cognitive diagnostic assessments. The first study investigates the effects of Q-matrix misspecification on the classification accuracy and consistency of diagnostic results. The two types of Q-matrix misspecifications examined are Q-entry misspecification (which includes three levels of misspecification: 10%, 20% and 30%), and attribute misspecification (which includes attribute exclusion and attribute inclusion). The results of a simulation study show that both Q-entry and attribute misspecification significantly deteriorate the accuracy of classification and the consistency of diagnostic results. In addition, the two classification accuracy and consistency indices have the potential to be useful in identifying possible attribute misspecification (e.g., attribute inclusion) of Q-matrix in empirical analyses. The second study provides a systematic performance evaluation of the three most commonly used Q-matrix validation methods: the sequential EM based δ-method (de la Torre, 2008), the Bayesian estimation method (DeCarlo, 2012), and the nonparametric Q-matrix refinement method (Chiu, 2013), with both basic and complex assessment design factors. The results of two simulation studies reveal that the Bayesian estimation method outperforms the other two methods in terms of recovering the misspecified Q-entries across various conditions. The performance of the three Q-matrix validation methods is also affected to different degrees by various assessment design factors, among which the data generation model is the most critical. The third study proposes a two-stage cross-validation method that combines the strengths of the nonparametric refinement method and Bayesian estimation techniques for improving Q-matrix validation accuracy and computation efficiency. The results show that the proposed method can effectively optimize Q-matrices that are possibly misspecified in both simulation and empirical data settings.","abstract_has_math":false,"creators":["Chen, Cong"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"Ph.D.","degree_level":"Dissertation","degree_discipline":"Educational Psychology","degree_department":null,"school":null,"contributors":["Zhang, Jinming","Anderson, Carolyn","Chang, Hua-Hua","Culpepper, Steven"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2017,"date_issued":"2017-09-29T17:45:08Z","date_published":"2017-09-29T17:45:08Z","updated_at":"2026-07-22T22:24:35Z","subjects":["Q-matrix","Cognitive diagnostic assessment"],"languages":["en"],"rights":["Copyright 2017 Cong Chen"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"http://hdl.handle.net/2142/98150","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Zhang, Jinming","Anderson, Carolyn","Chang, Hua-Hua","Culpepper, Steven"]},{"key":"dc:creator","label":"Author","values":["Chen, Cong"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2017-09-29T17:45:08Z","2020-03-03T10:15:29Z","2017-05-03","2017-08"]},{"key":"dc:type","label":"Dc Type","values":["text"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Educational Psychology"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Dissertation"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Ph.D."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Q-matrix","Cognitive diagnostic assessment"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2017 Cong Chen"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["http://hdl.handle.net/2142/98150"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Cognitive diagnostic assessment is a growing area in psychological and educational measurement with the advantage of providing diagnostic profiles (mastery/non-mastery of measured attributes) for examinees, giving insights for classroom teaching and student learning. Central to the successful implementation of a cognitive diagnostic assessment is the Q-matrix, the structure that specifies item-attribute relationships. However, the Q-matrix is prone to misspecification, given that it is often constructed based solely on human opinions. This thesis uses three research studies to investigate key issues of Q-matrix optimization for cognitive diagnostic assessments. The first study investigates the effects of Q-matrix misspecification on the classification accuracy and consistency of diagnostic results. The two types of Q-matrix misspecifications examined are Q-entry misspecification (which includes three levels of misspecification: 10%, 20% and 30%), and attribute misspecification (which includes attribute exclusion and attribute inclusion). The results of a simulation study show that both Q-entry and attribute misspecification significantly deteriorate the accuracy of classification and the consistency of diagnostic results. In addition, the two classification accuracy and consistency indices have the potential to be useful in identifying possible attribute misspecification (e.g., attribute inclusion) of Q-matrix in empirical analyses. The second study provides a systematic performance evaluation of the three most commonly used Q-matrix validation methods: the sequential EM based δ-method (de la Torre, 2008), the Bayesian estimation method (DeCarlo, 2012), and the nonparametric Q-matrix refinement method (Chiu, 2013), with both basic and complex assessment design factors. The results of two simulation studies reveal that the Bayesian estimation method outperforms the other two methods in terms of recovering the misspecified Q-entries across various conditions. The performance of the three Q-matrix validation methods is also affected to different degrees by various assessment design factors, among which the data generation model is the most critical. The third study proposes a two-stage cross-validation method that combines the strengths of the nonparametric refinement method and Bayesian estimation techniques for improving Q-matrix validation accuracy and computation efficiency. The results show that the proposed method can effectively optimize Q-matrices that are possibly misspecified in both simulation and empirical data settings.","Submission published under a 24 month embargo labeled 'U of I Access', the embargo will last until 2019-08-01","The student, Cong Chen, accepted the attached license on 2017-05-03 at 00:02.","The student, Cong Chen, submitted this Dissertation for approval on 2017-05-03 at 00:35.","This Dissertation was approved for publication on 2017-05-03 at 10:51.","DSpace SAF Submission Ingestion Package generated from Vireo submission #11145 on 2017-09-29 at 10:45:30","Made available in DSpace on 2017-09-29T17:45:08Z (GMT). No. of bitstreams: 2 CHEN-DISSERTATION-2017.pdf: 1129702 bytes, checksum: 68fd2221d97586cb6accbd42dcc96151 (MD5) LICENSE.txt: 4206 bytes, checksum: 9d874d521e0e28eca885e0f254e678f6 (MD5) Previous issue date: 2017-05-03","Embargo set by: Colleen Fallaw for item 103438 Lift date: 2019-09-29T17:48:06Z Reason: Author requested U of Illinois access only (OA after 2yrs) in Vireo ETD system","Embargo set by: Seth Robbins for item 103438 Lift date: 2020-03-02T19:56:41Z Reason: Author requested U of Illinois access only (OA after 2yrs) in Vireo ETD system","Embargo set by: Seth Robbins for item 103438 Lift date: 2020-03-02T19:59:52Z Reason: Author requested U of Illinois access only (OA after 2yrs) in Vireo ETD system","Embargo set by: Seth Robbins for item 103438 Lift date: 2020-03-02T20:02:46Z Reason: Author requested U of Illinois access only (OA after 2yrs) in Vireo ETD system","U of I Only Restriction Lifted for Item 103438 on 2020-03-03T10:15:29Z."]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Q-matrix optimization for cognitive diagnostic assessment"]}]}],"canonical_facts":{"dc:contributor":["Zhang, Jinming","Anderson, Carolyn","Chang, Hua-Hua","Culpepper, Steven"],"dc:creator":["Chen, Cong"],"dc:date":["2017-09-29T17:45:08Z","2020-03-03T10:15:29Z","2017-05-03","2017-08"],"dc:description":["Cognitive diagnostic assessment is a growing area in psychological and educational measurement with the advantage of providing diagnostic profiles (mastery/non-mastery of measured attributes) for examinees, giving insights for classroom teaching and student learning. Central to the successful implementation of a cognitive diagnostic assessment is the Q-matrix, the structure that specifies item-attribute relationships. However, the Q-matrix is prone to misspecification, given that it is often constructed based solely on human opinions. This thesis uses three research studies to investigate key issues of Q-matrix optimization for cognitive diagnostic assessments. The first study investigates the effects of Q-matrix misspecification on the classification accuracy and consistency of diagnostic results. The two types of Q-matrix misspecifications examined are Q-entry misspecification (which includes three levels of misspecification: 10%, 20% and 30%), and attribute misspecification (which includes attribute exclusion and attribute inclusion). The results of a simulation study show that both Q-entry and attribute misspecification significantly deteriorate the accuracy of classification and the consistency of diagnostic results. In addition, the two classification accuracy and consistency indices have the potential to be useful in identifying possible attribute misspecification (e.g., attribute inclusion) of Q-matrix in empirical analyses. The second study provides a systematic performance evaluation of the three most commonly used Q-matrix validation methods: the sequential EM based δ-method (de la Torre, 2008), the Bayesian estimation method (DeCarlo, 2012), and the nonparametric Q-matrix refinement method (Chiu, 2013), with both basic and complex assessment design factors. The results of two simulation studies reveal that the Bayesian estimation method outperforms the other two methods in terms of recovering the misspecified Q-entries across various conditions. The performance of the three Q-matrix validation methods is also affected to different degrees by various assessment design factors, among which the data generation model is the most critical. The third study proposes a two-stage cross-validation method that combines the strengths of the nonparametric refinement method and Bayesian estimation techniques for improving Q-matrix validation accuracy and computation efficiency. The results show that the proposed method can effectively optimize Q-matrices that are possibly misspecified in both simulation and empirical data settings.","Submission published under a 24 month embargo labeled 'U of I Access', the embargo will last until 2019-08-01","The student, Cong Chen, accepted the attached license on 2017-05-03 at 00:02.","The student, Cong Chen, submitted this Dissertation for approval on 2017-05-03 at 00:35.","This Dissertation was approved for publication on 2017-05-03 at 10:51.","DSpace SAF Submission Ingestion Package generated from Vireo submission #11145 on 2017-09-29 at 10:45:30","Made available in DSpace on 2017-09-29T17:45:08Z (GMT). No. of bitstreams: 2 CHEN-DISSERTATION-2017.pdf: 1129702 bytes, checksum: 68fd2221d97586cb6accbd42dcc96151 (MD5) LICENSE.txt: 4206 bytes, checksum: 9d874d521e0e28eca885e0f254e678f6 (MD5) Previous issue date: 2017-05-03","Embargo set by: Colleen Fallaw for item 103438 Lift date: 2019-09-29T17:48:06Z Reason: Author requested U of Illinois access only (OA after 2yrs) in Vireo ETD system","Embargo set by: Seth Robbins for item 103438 Lift date: 2020-03-02T19:56:41Z Reason: Author requested U of Illinois access only (OA after 2yrs) in Vireo ETD system","Embargo set by: Seth Robbins for item 103438 Lift date: 2020-03-02T19:59:52Z Reason: Author requested U of Illinois access only (OA after 2yrs) in Vireo ETD system","Embargo set by: Seth Robbins for item 103438 Lift date: 2020-03-02T20:02:46Z Reason: Author requested U of Illinois access only (OA after 2yrs) in Vireo ETD system","U of I Only Restriction Lifted for Item 103438 on 2020-03-03T10:15:29Z."],"dc:format":["application/pdf"],"dc:identifier":["http://hdl.handle.net/2142/98150"],"dc:language":["en"],"dc:rights":["Copyright 2017 Cong Chen"],"dc:subject":["Q-matrix","Cognitive diagnostic assessment"],"dc:title":["Q-matrix optimization for cognitive diagnostic assessment"],"dc:type":["text"],"thesis:degree_discipline":["Educational Psychology"],"thesis:degree_level":["Dissertation"],"thesis:degree_name":["Ph.D."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:24:35Z"}