This mixed methods study investigated the capacity of customized versions of ChatGPT-4o to generate scores and feedback for student responses to AP English Literature and Composition free response questions. This study entailed creating a customized version of ChatGPT-4o using researcher-created model instructions and released materials from the College Board. The quantitative phase tested the ability of the model to reliably score 56 released essays with official College Board scores; the AI-generated scores were compared to official scores using Cohen’s kappa and quadratic weighted kappa. Next, the four public high school AP English teachers participated in a 30-minute exploration of the customized version of ChatGPT-4o before answering semi-structured interview questions on the usefulness, ease of use, and accuracy of the scores and writing feedback generated by the model. Thematic analysis of the semi-structured interview transcript revealed five major themes. The quantitative and qualitative results were convergent for the accuracy of scoring.
","abstract_html":"<p>This mixed methods study investigated the capacity of customized versions of ChatGPT-4o to generate scores and feedback for student responses to AP English Literature and Composition free response questions. This study entailed creating a customized version of ChatGPT-4o using researcher-created model instructions and released materials from the College Board. The quantitative phase tested the ability of the model to reliably score 56 released essays with official College Board scores; the AI-generated scores were compared to official scores using Cohen’s kappa and quadratic weighted kappa. Next, the four public high school AP English teachers participated in a 30-minute exploration of the customized version of ChatGPT-4o before answering semi-structured interview questions on the usefulness, ease of use, and accuracy of the scores and writing feedback generated by the model. Thematic analysis of the semi-structured interview transcript revealed five major themes. The quantitative and qualitative results were convergent for the accuracy of scoring.</p>","abstract_has_math":false,"creators":["Harrell, Aaron M"],"institution":null,"degree_name":"Doctor of Education","degree_level":"Dissertation","degree_discipline":"Human Services","degree_department":null,"school":null,"contributors":["Brian Uriegas","Ali Hachem","Luis Aguerrevere"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2025,"date_issued":"2025-07-01T07:00:00Z","date_published":"2025-07-01T07:00:00Z","updated_at":"2026-07-24T04:30:49Z","subjects":["Artificial intelligence","Automated essay scoring","ChatGPT","Feedback","AP English","Mixed methods","Educational Leadership"],"languages":[],"rights":[],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://scholarworks.sfasu.edu/etds/638","outbound_label":"Repository record","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Brian Uriegas","Ali Hachem","Luis Aguerrevere"]},{"key":"dc:creator","label":"Author","values":["Harrell, Aaron M"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.available","label":"Dc Date Available","values":["2027-07-24T07:00:00Z"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Human Services"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Dissertation"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Doctor of Education"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Artificial intelligence","Automated essay scoring","ChatGPT","Feedback","AP English","Mixed methods","Educational Leadership"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://scholarworks.sfasu.edu/etds/638"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["This mixed methods study investigated the capacity of customized versions of ChatGPT-4o to generate scores and feedback for student responses to AP English Literature and Composition free response questions. This study entailed creating a customized version of ChatGPT-4o using researcher-created model instructions and released materials from the College Board. The quantitative phase tested the ability of the model to reliably score 56 released essays with official College Board scores; the AI-generated scores were compared to official scores using Cohen’s kappa and quadratic weighted kappa. Next, the four public high school AP English teachers participated in a 30-minute exploration of the customized version of ChatGPT-4o before answering semi-structured interview questions on the usefulness, ease of use, and accuracy of the scores and writing feedback generated by the model. Thematic analysis of the semi-structured interview transcript revealed five major themes. The quantitative and qualitative results were convergent for the accuracy of scoring.
"]},{"key":"dc:title","label":"Title","values":["Developing an AI Teacher Assistant: A Mixed Methods Study of Artificial Intelligence Scoring and Feedback Capabilities"]}]}],"canonical_facts":{"dc:contributor":["Brian Uriegas","Ali Hachem","Luis Aguerrevere"],"dc:creator":["Harrell, Aaron M"],"dc:date.available":["2027-07-24T07:00:00Z"],"dc:description.abstract":["This mixed methods study investigated the capacity of customized versions of ChatGPT-4o to generate scores and feedback for student responses to AP English Literature and Composition free response questions. This study entailed creating a customized version of ChatGPT-4o using researcher-created model instructions and released materials from the College Board. The quantitative phase tested the ability of the model to reliably score 56 released essays with official College Board scores; the AI-generated scores were compared to official scores using Cohen’s kappa and quadratic weighted kappa. Next, the four public high school AP English teachers participated in a 30-minute exploration of the customized version of ChatGPT-4o before answering semi-structured interview questions on the usefulness, ease of use, and accuracy of the scores and writing feedback generated by the model. Thematic analysis of the semi-structured interview transcript revealed five major themes. The quantitative and qualitative results were convergent for the accuracy of scoring.
"],"dc:identifier":["https://scholarworks.sfasu.edu/etds/638"],"dc:subject":["Artificial intelligence","Automated essay scoring","ChatGPT","Feedback","AP English","Mixed methods","Educational Leadership"],"dc:title":["Developing an AI Teacher Assistant: A Mixed Methods Study of Artificial Intelligence Scoring and Feedback Capabilities"],"thesis:degree_discipline":["Human Services"],"thesis:degree_level":["Dissertation"],"thesis:degree_name":["Doctor of Education"]},"updated_at":"2026-07-24T04:30:49Z"}