Q-matrix optimization for cognitive diagnostic assessment

Cognitive diagnostic assessment is a growing area in psychological and educational measurement with the advantage of providing diagnostic profiles (mastery/non-mastery of measured attributes) for examinees, giving insights for classroom teaching and student learning. Central to the successful implementation of a cognitive diagnostic assessment is the Q-matrix, the structure that specifies item-attribute relationships. However, the Q-matrix is prone to misspecification, given that it is often constructed based solely on human opinions. This thesis uses three research studies to investigate key issues of Q-matrix optimization for cognitive diagnostic assessments. The first study investigates the effects of Q-matrix misspecification on the classification accuracy and consistency of diagnostic results. The two types of Q-matrix misspecifications examined are Q-entry misspecification (which includes three levels of misspecification: 10%, 20% and 30%), and attribute misspecification (which includes attribute exclusion and attribute inclusion). The results of a simulation study show that both Q-entry and attribute misspecification significantly deteriorate the accuracy of classification and the consistency of diagnostic results. In addition, the two classification accuracy and consistency indices have the potential to be useful in identifying possible attribute misspecification (e.g., attribute inclusion) of Q-matrix in empirical analyses. The second study provides a systematic performance evaluation of the three most commonly used Q-matrix validation methods: the sequential EM based δ-method (de la Torre, 2008), the Bayesian estimation method (DeCarlo, 2012), and the nonparametric Q-matrix refinement method (Chiu, 2013), with both basic and complex assessment design factors. The results of two simulation studies reveal that the Bayesian estimation method outperforms the other two methods in terms of recovering the misspecified Q-entries across various conditions. The performance of the three Q-matrix validation methods is also affected to different degrees by various assessment design factors, among which the data generation model is the most critical. The third study proposes a two-stage cross-validation method that combines the strengths of the nonparametric refinement method and Bayesian estimation techniques for improving Q-matrix validation accuracy and computation efficiency. The results show that the proposed method can effectively optimize Q-matrices that are possibly misspecified in both simulation and empirical data settings.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Educational Psychology

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2017

Author dc:creator

• Chen, Cong

Contributors dc:contributor

• Zhang, Jinming
• Anderson, Carolyn
• Chang, Hua-Hua
• Culpepper, Steven

• Q-matrix
• Cognitive diagnostic assessment

Statement dc:rights

• Copyright 2017 Cong Chen

Language dc:language

en

Handle dc:identifier

http://hdl.handle.net/2142/98150

OAI identifier oai:identifier

oai:www.ideals.illinois.edu:2142/98150