Impact of Ignoring Nested Data Structures on Ability Estimation

The literature is clear that intentional or unintentional clustering of data elements typically results in the inflation of the estimated standard error of fixed parameter estimates. This study is unique in that it examines the impact of multilevel data structures on subject ability which are random effect predictions known as empirical Bayes estimates in the one-parameter IRT / Rasch model. The literature on the impact of complex survey design on latent trait models is mixed and there is no "best practice" established regarding how to handle this situation. A simulation study was conducted to address two questions related to ability estimation. First, what impacts does design based clustering have with respect to desirable statistical properties when estimating subject ability with the one-parameter IRT / Rasch model? Second, since empirical Bayes estimators have shrinkage properties, what impacts does clustering of first-stage sampling units have on measurement validity-does the first-stage sampling unit impact the ability estimate, and if so, is this desirable and equitable? Two models were fit to a factorial experimental design where the data were simulated over various conditions. The first model Rasch model formulated as a HGLM ignores the sample design (incorrect model) while the second incorporates a first-stage sampling unit (correct model). Study findings generally showed that the two models were comparable with respect to desirable statistical properties under a majority of the replicated conditions-more measurement error in ability estimation is found when the intra-class correlation is high and the item pool is small. In practice this is the exception rather than the norm. However, it was found that the empirical Bayes estimates were dependent upon the first-stage sampling unit raising the issue of equity and fairness in educational decision making. A real-world complex survey design with binary outcome data was also fit with both models. Analysis of the data supported the simulation design results which lead to the conclusion that modeling binary Rasch data may resort to a policy tradeoff between desirable statistical properties and measurement validity.

Name thesis:degree_name

Ph. D.

Level thesis:degree_level

doctoral

Discipline thesis:degree_discipline

Educational Research and Evaluation

Department dc:contributor.department

Educational Leadership and Policy Studies

Grantor dc:publisher

Virginia Tech

Year dc:date.issued

2014

Author dc:creator

• Shropshire, Kevin O'Neil

Chair dc:contributor.committeechair

• Miyazaki, Yasuo

Committee members dc:contributor.committeemember

• House, Leanna L.
• Singh, Kusum
• Skaggs, Gary E.
• Savla, Jyoti S.

• Complex survey designs
• clustering
• PSU
• nested data
• multilevel data
• hierarchical data
• two-level HGLM
• three-level HGLM
• Rasch
• ability estimation

Statement dc:rights

• In Copyright

Licence dc:rights.uri

• http://rightsstatements.org/vocab/InC/1.0/

Dc Identifier Other

vt_gsexam:3063

OAI identifier oai:identifier

oai:vtechworks.lib.vt.edu:10919/64197