Assessing the Validity of Multiple-Choice Questions, Using them to Undertake Comparative Analysis on Student Cohort Performance, and Evaluating the Methodologies Used

Multiple-choice questions (MCQs) have been used in all four first-year Chemistry courses at The University of Adelaide for many years to assess the students both during the semester and within the final examinations. Ensuring that the assessment tasks and the items used within them are a valid method of determining student competency is an important part of reviewing the results of the assessments. Throughout this research all the MCQ assessment tasks and items used in first-year Chemistry courses between 2012-2015 were reviewed and analysed using Classical Test Theory (CTT) and Rasch Analysis. Out of 261 unique items that were utilised in MCQ assessment tasks across the four first-year Chemistry courses over the 4-year period analysed most of the items were found to be performing appropriately. However, 12 of them were found to be consistently problematic by CTT (4 major issues, 4 potentially major issues, and 4 minor issues), and 83 items were seen to be consistently problematic using Rasch analysis (41 major issues, 9 potentially major issues, and 33 minor issues), showing a large difference in the items that are identified by each methodology. Due to the fact that these problematic items are spread over a large number of assessment tasks that took place during different years none of the individual assessment tasks contained enough problematic items that it impacted upon the validity of those assessment tasks. After excluding these problematic items when analysing for differences in how male and female students perform 27 unique items were identified to contain gender bias using CTT (18 male, 4 female, 5 alternating) out of 249 unique MCQ items while only 14 unique items were identified to contain gender bias by Rasch analysis (7 male, 6 female, 1 alternating) out of 178 unique items considered. The performance of student cohorts over different time intervals was also explored within this research. This involved the comparison of students’ results between the lecture tests undertaken during the semester to their results within the MCQ section of the final examination. It also involved comparing student cohorts from the same course over different years to determine if student competency changes significantly between. On average students performed to a higher level on MCQ assessment tasks later in the semester years (i.e. a ‘test-retest’ strategy is beneficial for students). There were no discernible trends identified when comparing between yearly student cohorts within the same course, thus indicating that student competency did not change significantly within the period studied (2012-2015). Comparing between Biology and Chemistry courses was also attempted within this research; however, due to complications in linking the results of the assessment tasks it was determined that this was not possible using the available data. As the entirety of this research was based around two different methodologies that can evaluate MCQ assessment tasks and items in different ways (Classical Test Theory and Rasch Analysis) it gave the opportunity to compare the two methodologies. Classical Test Theory was found to be a highly approachable methodology that provided a high-level overview of an assessment task and its items, making it a suitable methodology for analysing low stakes assessment tasks to improve upon any large issues within those tasks. Rasch Analysis could be used to analyse the performance of students as well as the assessment tasks and items, allowing for greater versatility in how the analysis can be applied and the information that it can provide. This makes Rasch Analysis a more appropriate methodology for high stakes assessment tasks where it is important that the most accurate reflection of the students’ abilities and any issues present within the task need to be identified and addressed.

Author dc:creator

• Marchant, Jacob Rhys

Advisors dc:contributor.advisor

• Pyke, Simon
• Williamson, Natalie

• Student Assessment
• Classical Test Theory
• Rasch Analysis
• Multiple-Choice Questions

Language dc:language.iso

en

Handle dc:identifier.uri

https://hdl.handle.net/2440/132705

OAI identifier oai:identifier

oai:digital.library.adelaide.edu.au:2440/132705