Implementing a cognitive diagnostic assessment in an institutional test: a new networking model in language testing and experiment with a new psychometric model and task type

This dissertation is based on two major projects, cognitive diagnostic model comparison and implementing cognitive diagnostic assessment in an institutional test. In the first project, five cognitive diagnostic models are empirically compared for language test data under a unified general modeling framework. The models are applied to three sets of data, TOEFL Reading, TOEFL Listening and ECPE Grammar and examined in terms of their model fit to the data and functioning. The item-association root mean squared error values and multiple information criteria all indicate that the general model (LCDM) and the compensatory-RUM model are the best fit to all three test data sets used. The functioning of the models examined through multiple indices also unanimously confirms these fit indices. Based on these results, a discussion follows to argue that the general modeling framework is optimal for language assessment data due to its much greater flexibility. The behaviors of the compensatory RUM and non-compensatory RUM (Fusion) models are also compared and the differences are analyzed. Cognitive diagnostic assessment (CDA) has gained attention in language testing since late 90’s. A few models of this new assessment method have been applied to the response data of different language tests, yielding encouraging results in general. Most of these empirical studies used large-scale, standardized tests and retrofitted to these existing tests. The second research study in this proposal was an effort to go beyond this limited research context of previous studies of CDA in language testing. Using a new psychometric cognitive diagnostic model (Log-linear cognitive diagnostic model) and task type (elicited imitation task) as well as constructing the EIT test with a CDA implementation in mind from the outset (thus not retrofitting), this project tried to implement a cognitive diagnostic assessment in an institutional placement test. In doing so, the study employed a statistical method (resampling technique) as a way to resolve the issue caused by a small size of a test-taker pool at an institution, which is usually smaller than the optimal size for CDA implementation. It also tried to analyze polytomously scored response data. The study achieved a success at estimating with polytomous response data that were scored with a three-point scale (i.e., zero to two points). Though it was a limited success (in that more complex rating scale could not be analyzed), it was the first success in estimating with polytomous response data in the context of CDA research in language testing. The analysis results of the study also provide many insights and implications for the process of Q-matrix construction, grain size of attributes, appropriate task types and item types for cognitive diagnostic assessment as well as appropriate cognitive psychometric models for differing contexts of CDA implementation. The study also rediscovers or introduces the usefulness of bootstrap resampling method as an approach that is gaining popularity even in areas where only traditional quantitative methods are usually employed. Also, the networking occurred in this project between students in different specializations could be established as a new networking model in language testing. Considering such a collaboration is very much needed for implementing a relatively new measurement method in a specific knowledge domain, the co-work attempted in this project could serve as a model for implementing CDA in language testing.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Educational Psychology

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2013

Author dc:creator

• Yi, Yeonsook

Contributors dc:contributor

• Davidson, Frederick G.
• Zhang, Jinming
• Bowles, Melissa A.
• Templin, Jonathan
• Tremblay, Annie

• cognitive diagnostic assessment
• cognitive diagnostic model comparison
• general modeling framework
• language test data
• Test of English as a Foreign Language (TOEFL) Reading and Listening
• university French placement test
• elicited imitation task
• resampling
• polytomous
• networking in language testing
• Examination for the Certificate of Proficiency in English (ECPE) Grammar

Statement dc:rights

• Copyright 2012 Yeonsook Yi

Language dc:language

en

Handle dc:identifier

http://hdl.handle.net/2142/42124

OAI identifier oai:identifier

oai:www.ideals.illinois.edu:2142/42124