Modeling item bias in fixed-item tests and computerized adaptive tests

Education equity and fairness of assessment are two of the most important topics in education. Bias is a related concept, where there is construct bias, method bias, and item bias (e.g., van de Vijver & Poortinga, 1997; Werner & Campbell, 1970). Measurement invariance is another one, essential in modern standardized assessments based on item response theory (IRT; Richardson, 1936). It includes (1) configural invariance, (2) weak invariance (or metric invariance), (3) strong invariance (or scalar invariance), and (4) strict invariance (Wu et al., 2007). It ensures the comparability of scores across different examinee groups. An operational definition of the scalar invariance is precisely the definition of differential item functioning (DIF; Berk, 1982), a framework that underpins modern methods to investigate bias that possibly exists in assessment items. Despite its long-standing literature and wide application in the testing industry, DIF remains an important area for psychometric research and the discussion concerning DIF detection methods keeps evolving. This dissertation aimed to highlight the limitations of a few commonly used DIF methods in psychometric practice and suggest alternative approaches to improve DIF detection accuracy. Chapter 1 provides an extensive review of the DIF methods having been widely applied in either the testing industry or academic literature. The methods include those following the single-level or the multilevel framework. Chapter 2 reveals key patterns associated with the cutoff value of the RMSD approach, which is a popular DIF method for analyzing cross-country score comparability in international large-scale assessments. Based on Type-I error rates from simulation, a polynomial regression was proposed to predict the appropriate cutoff given the number of groups to be analyzed and significance level. Chapter 3 compares the performance of the RMSD approach with four other approaches designed for multi-group DIF analysis. When compared with a few acceptable methods in the evolving multi-group DIF literature, the RMSD approach was demonstrated to be optimal when used along with the predicted cutoff. Chapter 4 concerns a multilevel framework for DIF detection with data from computerized adaptive tests (CAT), where DIF remains understudied. It conceptualized how between-item dependency could be reframed into between-examinee dependency, and proposes a multilevel model to obtain more accurate DIF results after accounting for this dependency. Chapter 5 lays out a few directions for future research, including developing a global effect size measure across DIF methods, a general DIF approach based on a versatile framework named structural equation modeling, and a shift of the statistical framework for DIF detection.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Educational Psychology

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2023

Author dc:creator

• Chen, Dandan

Contributors dc:contributor

• Zhang, Jinming
• Anderson, Carolyn
• Kern, Justin
• Xia, Yan
• Shin, David

• Differential Item Functioning
• Dif
• Fixed-item Tests
• Computerized Adaptive Tests
• Cat

Statement dc:rights

• Copyright 2023 Dandan Chen

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/120189

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F120189