Assessment of Method Effects of Keying and Wording in Instruments: A Mixed-Methods Explanatory Sequential Study

<p>This dissertation presents an innovative approach to examining the keying method, wording method, and construct validity on psychometric instruments. By employing a mixed methods explanatory sequential design, the effects of keying and wording in two psychometric assessments were examined and validated. Those two self-report psychometric assessments were the Effortful Control assessment (Ellis & Rothbart, 2001) and the Grit assessment (Duckworth & Quinn, 2009). Moreover, the quantitative phase utilized structural equation modeling to analyze 2,104 students’ responses and assess the construct of keying and wording. Various hypothetical models were investigated and evaluated. The reliability of each construct in each method was analyzed using several omega coefficients. Simultaneously, construct validity was discussed through model performance and parameter estimates. Following the quantitative phase, six one-on-one interviews were conducted in the qualitative phase. The purpose of the second phase was to gather participants’ opinions and gain insights to further interpret and understand the results obtained in the first quantitative phase. By integrating quantitative and qualitative approaches, this study aims to enhance the validity and meaningfulness of its conclusions.</p> <p>The overarching purpose of this study was to assess the keying structure and method effects of keying and wording through a mixed methods explanatory sequential study. To achieve this goal, the study investigated different method factors that impacted the construct validity of psychometric assessments among adolescents. Findings from the first quantitative strand revealed the detection of the keying, although the structure of keying displayed inconsistent probability of variance associated with the reliability across different instruments. Additionally, method effects of keying and wording were identified, and problematic item questions were further examined in the qualitative phase. Results from the qualitative strand led to the conclusion that three factors influenced adolescents’ understanding of assessment questions. These were complex item questions, student’s characteristics, and school course plan. This study therefore addressed research gaps regarding the effects of keying and wording in psychometric instruments and contributed to understanding their impact on construct validity. The findings draw attention to the keying and wording methods in self-report psychometric instruments. This also impacted the accuracy of assessing the construct of interest and the further parameter estimates. Based on these findings, suggestions for future research were discussed and summarized.</p>

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Year

2024

Author dc:creator

• Ma, Lin

Contributors dc:contributor

• Nicholas Cutforth
• Antonio Olmos
• Lilian Chimuma
• Ruth Chao

• Factor analysis
• Keying and wording
• Method effects
• Mixed-methods explanatory sequential study
• Psychometric instruments
• Structural equation modeling
• Education
• Educational Assessment, Evaluation, and Research
• Physical Sciences and Mathematics
• Statistical Methodology
• Statistical Models
• Statistics and Probability

Statement dc:rights

• <p>Copyright is held by the author. User is responsible for all copyright compliance.</p>

Language dc:language

English (eng)

Repository record dc:identifier

https://digitalcommons.du.edu/etd/2379

OAI identifier oai:identifier

oai:digitalcommons.du.edu:etd-3362