Modeling improvement for educational assessments: comparisons and extensions

Item response theory models (IRTMs) and cognitive diagnostic models (CDMs) are two well-known families of theory-driven educational assessment models. IRTMs focus on the relative position of students in an ability continuum, while CDMs focus on the absolute diagnostic classification of students’ latent traits. Multidimensional IRTMs (MIRTMs) share similarities with CDMs. Both can measure the multidimensional abilities of students through dichotomously scored items. And they can be divided into compensatory and noncompensatory branches. The application of CDMs in real assessment practices is less prevalent than IRTMs, and one possible reason is the assessment stability among calibration samples. If CDMs do not perfectly match the data which is common in real practices, the item invariance property is violated. Chapter 2 explores the robustness of CDMs assuming underlying attributes are continuous, studying factors that influence the definition of mastery in MIRTM-based assessments. Results showed that 0 was not the universal cut point of continuous attributes for mastery vs non-mastery. Indeed, the scaling issue occurred when CDMs were used to handle MIRTM-based datasets. Additionally, because of the development of computing power and data science, researchers adopt machine learning techniques to improve educational assessment models. Specifically, the neural network (NN) is a nonparametric and model-agnostic approach that does not require any assumptions of the parameter distribution, which is powerful in approximating complicated nonlinear functions. Previous neural assessment models constricted hidden layers of the NN in different ways to improve the interpretability of the internal working of the model. A novel neural CDM (NN-CDM) was introduced in Chapter 3, where a small sample of test takers’ responses and attribute mastery states were used for training. Simulation studies were conducted in DINA-based datasets and MIRTM-based datasets with predetermined cut points. Also, this chapter filled the research gap in the application of neural assessment models in attribute hierarchy conditions. The NN-CDM outperformed the existing supervised neural CDM in all conditions. The other important extension of traditional assessment models is to deal with longitudinal assessments. Longitudinal assessments can track students’ stepwise performance, which fits the need for e-learning platforms. There is a limited discussion on model weaknesses and strengths, as well as practical concerns. Most previous literature focuses on the model performance of conditions with less than 4 time points. In Chapter 4, simulation studies were conducted to compare the performance of four longitudinal assessment models (three L-IRTMs and one L-CDM) delivering continuous latent ability estimates. The strengths and weaknesses of each model were discussed, from both conceptual and practical perspectives. In terms of L-IRTMs, more time points, larger sample sizes, and longer tests slightly improved the accuracy of ability parameter estimation. The L-UIRTM obtained the most accurate estimates in all conditions.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Educational Psychology

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2023

Author dc:creator

• Hu, Mingqi

Contributors dc:contributor

• Zhang, Jinming
• Anderson, Carolyn J.
• Chang, Hua-hua
• Xia, Yan

• Item Response Theory
• Cognitive Diagnostic Models
• Neural Network

Statement dc:rights

• Copyright 2023 Mingqi Hu

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/121337

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F121337