Developing an AI Teacher Assistant: A Mixed Methods Study of Artificial Intelligence Scoring and Feedback Capabilities

<p>This mixed methods study investigated the capacity of customized versions of ChatGPT-4o to generate scores and feedback for student responses to AP English Literature and Composition free response questions. This study entailed creating a customized version of ChatGPT-4o using researcher-created model instructions and released materials from the College Board. The quantitative phase tested the ability of the model to reliably score 56 released essays with official College Board scores; the AI-generated scores were compared to official scores using Cohen’s kappa and quadratic weighted kappa. Next, the four public high school AP English teachers participated in a 30-minute exploration of the customized version of ChatGPT-4o before answering semi-structured interview questions on the usefulness, ease of use, and accuracy of the scores and writing feedback generated by the model. Thematic analysis of the semi-structured interview transcript revealed five major themes. The quantitative and qualitative results were convergent for the accuracy of scoring.</p>

Name thesis:degree_name

Doctor of Education

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Human Services

Year dc:date.available

2025

Author dc:creator

• Harrell, Aaron M

Contributors dc:contributor

• Brian Uriegas
• Ali Hachem
• Luis Aguerrevere

• Artificial intelligence
• Automated essay scoring
• ChatGPT
• Feedback
• AP English
• Mixed methods
• Educational Leadership

Repository record dc:identifier

https://scholarworks.sfasu.edu/etds/638

OAI identifier oai:identifier

oai:scholarworks.sfasu.edu:etds-1675