What makes an argument effective? Triangulating task characteristics, essay performances, and rating processes in an L2 integrated argumentative writing test

Integrated argumentative tasks are commonly used to assess learners’ writing ability in second language (L2) writing assessment. While abundant research has addressed performance characteristics and rating processes of integrated argumentative tests (e.g., Gebril & Plakans, 2009, 2014), our understanding of the nature of L2 argumentation and the impact of source use on argumentation quality remains relatively limited. This dissertation study thus investigates essay performances and rating processes of an operational integrated argumentative writing test to delineate how features related to task, examinee, and rater interact, in the hope of gathering validity evidence for the use of such tests in an L2 context. Specifically, it examines (1) whether argumentation effectiveness exhibits systematic differences across proficiency levels; (2) how source use influences argumentation effectiveness; (3) whether raters consider source use characteristics when scoring argumentation effectiveness; (4) what cognitive processes are involved in scoring argumentation effectiveness. To answer the four main research questions, I analyzed test takers’ written products and raters’ rating performances qualitatively and quantitatively. I first conducted text analysis to examine features pertaining to argumentation and source use. Then, I designed an offline experiment to examine if different source use characteristics impact raters’ scoring of argumentation. I further investigated raters’ cognitive processes using the eye-tracking methodology. Results from test takers’ writing performances show that (1) both argument structure and reasoning quality differed across proficiency levels and (2) source use characteristics had an impact on argument structure and reasoning quality. The offline experiment further indicates that the quantity and quality of source use could determine argumentation scores assigned by raters. Finally, the eye-tracking experiment reveals the cognitive processes involved in scoring integrated argumentative writing performances. Raters paid the most attention to sentences with source text use, and their eye movements were related to the scores and comments they provided. This dissertation provides a deeper understanding of how L2 test takers construct arguments and how raters evaluate integrated argumentative writing performances. It also stands to have practical implications for L2 writing pedagogy and assessment, including how argumentation can be taught in writing courses and how rating scales and rater training programs can be developed to guide the scoring of argumentation effectiveness in academic contexts.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Linguistics

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2024

Author dc:creator

• Chuang, Ping-Lin

Contributors dc:contributor

• Yan, Xun
• Bowles, Melissa
• Christianson, Kiel
• Plakans, Lia

• L2 Writing Assessment
• Integrated Writing
• Argumentation
• Source Use
• Rater Behavior

Statement dc:rights

• Copyright 2024 Ping-Lin Chuang

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/125529

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F125529