A Bayesian change-point approach to item compromise detection

Item compromise has been a longstanding issue that threatens test validity and test security in continuous tests. The literature in compromised item detection thus far has been primarily focused on non-Bayesian methods. Most of the existing detection methods monitor the item compromise process at the item level; each item is monitored individually through a summary statistic, such as the correct or incorrect response proportion. What's more, most of the existing detection methods are primarily based on response accuracy and response time information has not been efficiently utilized. In this study, a Bayesian change-point method that incorporates responses and response times is proposed to detect compromised items in continuous tests. In Chapter 1, the background of continuous tests (using computerized tests as an example) and test security is briefly overviewed. The strengths and weaknesses of continuous tests and the challenges that they have imposed on test security will be discussed. Next, the history of utilizing response times in testing as well as why and how response times can help uncover the underlying response process will be presented. Within this context, the problem that this dissertation is trying to resolve, i.e., detecting item compromise in continuous tests with responses and response times, will be clearly stated. The significance of addressing this problem will be discussed as well. In Chapter 2, the methods related to item compromise prevention and detection will be carefully reviewed. With an emphasis on the detection methods, most recent detection methods will be reviewed and synthesized based on the nature of the methods. Meanwhile, the Bayesian change-point framework will also be illustrated. Here, both Phase I and Phase II detection will be demonstrated, with their definitions and application context being explained. Chapter 3 presents a pilot study which aims at employing Bayesian change-point approach to detect the item compromise with responses only. The main purpose of this pilot study is to show that Bayesian change-point approach has great potential in item compromise detection and can be employed to enhance test security. Three simulation studies are conducted to show its performance. Simulation I aims to show the stationary Bayesian change-point model. Specifically, convergence and parameter recovery of the proposed Metropolis-Hastings-within-Gibbs algorithms with respect to the change-point model are evaluated. Simulation II demonstrates the performance of Shiryaev procedure in detecting compromised items in real time while using true person parameters. Simulation III also presents the performance of Shiryaev procedure, but using the EAP estimates of latent ability parameters. In Chapter 4, the proposed Bayesian change-point model with responses and response times and the Shiryaev procedure are explained in detail. The simulation designs in conjunction with evaluation criteria are explained as well. Chapter 5 presents the results of the three simulations. Convergence plot and parameter recovery of the Bayesian change-point model are presented in simulation I. Detection accuracy and efficiency are presented for simulation II. The comparison of the Bayesian change-point method with and without RT is presented in simulation III. Finally, the conclusions of the current study are presented in Chapter 6. Limitations as well as the directions for future studies are also discussed.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Educational Psychology

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2022

Author dc:creator

• Du, Yang

Contributors dc:contributor

• Cromley, Jennifer G
• Chang, Hua-Hua
• Kern, Justin L
• Xia, Yan

• test security
• item compromise
• continuous tests
• Bayesian change-point

Statement dc:rights

• Copyright 2022 Yang Du.

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/115512

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F115512