IRTree selection validity in presence of extreme response styles

The measurement of psychological constructs is frequently based on self-report tests, often with Likert-type items rated from “Strongly Disagree” to “Strongly Agree.” However, previous research has suggested that responses to these types of items are often not solely a function of the content trait of interest, but also of other systematic response tendencies due to item format. These tendencies, called response styles, can introduce noise into the measurement of a content trait. Previous research also demonstrates demographic group differences in response styles, potentially introducing bias into the scoring of tests. In recent years, a family of item response theory (IRT) models called IRTree models have been developed that can allow researchers to parse out content traits (e.g., personality traits) from noise traits (e.g., response styles). IRTree models, in which IRT models are organized into a decision tree structure, allow for unique parameters at each node of a theorized response process. While there has been research in evaluating the efficacy of IRTree models in modeling response processes in comparison to other models, there are few studies regarding the practical implications of decisions made based on this structured model. In this study, we analyze the selection validity and adverse impact of classification decisions made based on an IRTree model that controls for response style tendencies compared to an IRT model that does not control for response style tendencies. Here, we consider situations where respondents may display an extreme response style, or a tendency to systematically prefer extreme response categories (e.g., “Strongly Disagree” or “Strongly Agree”), though the approach could be expanded to use for any theorized response process. We first present a simulation in which we demonstrate that in cases where extreme response styles do exist, the IRTree model has greater selection validity than the generalized partial credit model (GPCM) while in situations where extreme response styles do not exist, the models perform nearly equally well. We also find that, in situations where extreme response styles do exist, the IRTree model presents far fewer instances of adverse impact. Additionally, we demonstrate that both models are robust to item-level directional invariance (i.e., cases where the threshold to endorse an extreme response category is dependent upon the agree or disagree direction). To consider the practical applicability of our findings, we also present an example using real data collected from the Open-Source Psychometrics Project Fisher Temperament Inventory dataset. We found the IRTree model to have better fit, but little variation in the selection decision made by each model; thus, adverse impact was virtually identical between the two models. We offer considerations for practical application of the IRTree model and future directions for research.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Educational Psychology

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2023

Author dc:creator

• Quirk, Victoria L.

Contributors dc:contributor

• Kern, Justin L.
• Zhang, Jinming

• Item Response Theory
• Item Response Trees
• Irtree Models
• Response Process
• Non-content Variability
• Extreme Response Styles
• Likert Scales
• Selection Validity
• Classification Accuracy
• Adverse Impact

Statement dc:rights

• Copyright 2023 Victoria Quirk

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/122044

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F122044