Properties of a Resampling Validation Technique for Empirically Scoring Psychological Assessments

"Four studies were conducted to examine the psychometric properties of a new, resampling-based validity estimator which does not require a crossvalidation sample in order to produce unbiased estimates of validity. The first and second experiments were Monte Carlo simulations while the third and fourth applied the method to ""real"" data. In Experiment 1, a purely empirical method was found to be almost as effective as crossvalidation in debiasing the validity estimate; samples of 200 or more were found to be needed to produce valid empirical keys. In Experiment 2, a hybrid method which combines a priori model information was examined and found not to substantially improve the procedure unless the model information was of excellent quality (but the resampling estimator was also shown to be fairly robust to poor model information). Experiment 3 replicated the simulation results of Experiment 1 using ""real"" personality data and Experiment 4 compared the purely empirical keying to a hybrid method that blended empirical and a priori information for a biodata instrument. The results of these two experiments using empirical data confirmed many of the aspects of the simulations while pointing out some limitations of the simulated context. In general, the use of thresholds (minimum option-validities or number endorsing an option) to improve the robustness of the estimate was not strongly supported. The addition of a priori, model information in the hybrid keying only slightly improved upon the pure empirical approach. The hybrid keying method that relies most heavily on empirical information paired with a good model key allows samples as small as N = 100 to be keyed."

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Psychology

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2015

Author dc:creator

• Mead, Alan David

Contributors dc:contributor

• Drasgow, Fritz

• Psychology, Psychometrics

Language dc:language

eng

Identifier

(MiAaPQ)AAI9971134

OAI identifier oai:identifier

oai:www.ideals.illinois.edu:2142/82307