Rater Implicit Gender Bias in a Multi-Center Simulation-Based Resident Milestone Assessment

Background: Gender bias in graduate medical education assessments may disadvantage female residents. With simulation increasingly used in high-stakes assessments, such as those used to assess emergency medicine residents, understanding potential bias is essential. Objective: To evaluate the presence of gender bias in Simtastic, a large multi-center simulation-based milestone assessment of second year residents to better understand the presence of gender bias in simulation-based assessments. Methods: This retrospective study analyzed 1011 Simtastic scores of PGY2 residents from 2015–2025. Residents were scored using case-specific checklists and global assessments. The data was analyzed for scoring bias based on gender concordance between rater and resident by comparing global assessments to the checklist score on each case. Hierarchical linear modeling accounted for clustering by residency program, site, and rater. Results: Checklist and global assessments (GA) scores were strongly correlated. However, female residents received lower GA and checklist scores than male residents across all years. Male raters awarded higher scores to male residents, while female raters showed no such difference. Over time, GA scores from male raters converged with those from female raters, suggesting reduced inflation with rater experience. Despite this trend, score disparities for female residents persisted. Conclusions: Although female residents consistently received lower scores in Simtastic, such trends likely reflect systemic bias rather than performance differences. Despite male rater bias diminishing over time, inequities remained. Structured rater training and refinement of checklist design are needed to promote equity in high-stakes simulation-based evaluations.

Author dc:creator

• Clare Desmond (24400265)

• Rater Gender Bias
• Gender Bias in Simulation-Based Assessment

Statement dc:rights

• In Copyright
• Open Access after 2028-05-01

DOI dc:identifier

https://doi.org/10.25417/uic.32995307.v1

OAI identifier oai:identifier

oai:figshare.com:article/32995307