Proposal of a semi-adaptive local Chinese placement test: Design, development, and validation

As technology develops and due to the strike of the Covid-19 pandemic, online instruction and language testing have become the new trend in language learning and teaching. The computerized adaptive testing (CAT) is utilized by many large-scale standardized language tests such as the Test of English as a Foreign Language (TOEFL) because of its many advantages (e.g., efficiency, accuracy, preventing cheating...etc.) Nevertheless, there has been limited discussion and utilization of CAT in local testing contexts, perhaps discouraged by its somewhat challenging requirements, such as powerful computer programs and algorithms for item scoring and selection. Hence, the current work proposes a CAT-based online Chinese assessment (OCEAN), which is a local placement test situated in the Chinese program at the University of Illinois at Urbana-Champaign (UIUC). With its semi-adaptive test design, OCEAN serves as an example and demonstrates how CAT principles can be simplified and used in local contexts to increase local test efficiency and accuracy. Though this dissertation does not involve the implementation of the semi-adaptive testing system, it provides detailed descriptions regarding the design of the semi-adaptive test system, and the development and validation of test items and tasks to be used in the proposed testing system. OCEAN places a student into the course level that best matches their Chinese ability through its three-phase test flow: a self-assessment task that determines the initial test difficulty of the following adaptive language tests, stage-adaptive listening and reading tests, and integrated speaking and writing tasks whose difficulty levels are based on a student’s listening and reading performance. The flow of the test was so designed so that most students can be placed into a course most appropriate for them through the preliminary identification of the self-assessment tasks and confirmation of the automated-graded listening and reading test. Only a small portion of students whose Chinese proficiency needs further examination (e.g., at the borderline) would need to take the integrated and independent speaking and writing tests, which not only measure a student’s four basic skills (i.e., listening, reading, speaking, and writing) in Chinese, but also would be graded by experienced local instructors who are experts of the curriculum. Besides the test flow design, the efficiency of OCEAN highly relies on the validity and reliability of its test items and tasks, which is the primary focus of the current work. Psychometric qualities of test items and tasks were examined by Rasch analyses. The Rasch results indicated that the self-assessment tasks, and listening and reading test items had excellent validity and can reliably separate students with different levels of Chinese proficiency. The speaking and writing tasks were validated based on criteria describing students’ performance in each type of task (i.e., integrated or independent). According to Rasch measurements, the difficulty of the test items defined by task type and criteria was appropriate and can separate competent students from those who need more practice. The Rasch measurements also suggested that the performance grading rubrics can be used reliably and consistently by raters. Standards for passing and failing each test in OCEAN were carefully set through the collaborative work of local instructors via the criterion-centered Angoff method and the examinee-based borderline regression method. The former asks instructors to rate the likelihood of a student who has the minimal ability required to continue to the next course level answering each test item correctly, while the latter identifies the cut score of a test by comparing students’ numeric and categorical scores that describe their performance in a test in a linear regression line. By applying the carefully decided cut scores, students who had received at least 11 weeks of instruction in the Chinese program, which match the program’s definition of borderline passing and passing students, tended to pass a test, separating them from those who had not. Overall, the test items and tasks designed for the purposes of the current work had satisfying performance, especially considering the smaller-than-desired scope of the work. The psychometric qualities of the test items and tasks when they were applied to a non-targeted student population were also explored, and the results suggested that their good quality remained although new sets of cut scores were required for placement purposes. At the end of the study, suggestions for future research, including the implementation of the proposed semi-adaptive testing system, and practical implications of the current work for other foreign language programs, especially programs of Chinese and Less Commonly Taught Languages (LCTLs), are discussed.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

E Asian Languages & Cultures

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2023

Author dc:creator

• Lien, Zhi-Ling

Contributors dc:contributor

• Shih, Chilin
• Yan, Xun
• Bowles, Melissa
• Sadler, Misumi

• Local Language Testing
• Adaptive Language Testing
• Self-assessment
• Less Commonly Taught Languages (LCTLS)

Statement dc:rights

• Copyright 2023 Zhi-Ling Lien

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/120369

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F120369