Effects of respondent training on self-report personality assessment: an item response theory approach

Within the item response theory (IRT) framework and inspired by the rater training literature, this study explored the effects of short online respondent training on personality item interpretation and responding and the number of response categories (i.e. polytomous vs. dichotomous) on item performance, model-data fit, and criterion-related validity. Participants recruited from MTurk (n = 1977) were randomly assigned to 1 of the 4 groups differing in training (i.e. training vs. no training) and response scale (i.e. 4-point Likert scale vs. dichotomous), and their responses to dominance and ideal-point personality measures were analyzed with GGUM, SGR, and 2PL. Results indicated that training was associated with more well-performing and more discriminating and informative intermediate items on the ideal-point scales when a dichotomous response scale was used. The dichotomous scale in general was related to better fit, while criterion-related validity stayed unaffected by both training and the response scale. Participants reported that they had been confused about personality items before, and were positive about the online training, which was consistent with the finding that trained participants on average spent 32 seconds less finishing the ideal-point surveys. Implications for future research and practice are discussed.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Psychology

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2019

Author dc:creator

• Zhang, Luyao

Contributors dc:contributor

• Drasgow, Fritz
• Chang, Hua-Hua
• Fraley, R. Chris
• Newman, Daniel
• Roberts, Brent
• Rounds, James

• Rater training
• item response theory (IRT)
• self-report personality assessment
• the ideal-point model
• intermediate items

Statement dc:rights

• Copyright 2018 Luyao Zhang

Language dc:language

en

Handle dc:identifier

http://hdl.handle.net/2142/102913

OAI identifier oai:identifier

oai:www.ideals.illinois.edu:2142/102913