An Analysis of Bias Towards Women in Large Language Models Using Likert Scale Evaluations

<p>Closed-source large language models (LLMs) developed by large technology companies continue to grow in popularity. However, ethical conversations surrounding the safety of model outputs have been a prominent topic of discussion. This project aims to assess three leading closed-source LLMs: OpenAI’s ChatGPT, Google’s Gemini, and Anthropic’s Claude, to analyze how their outputs perform when treated as a subject of several psychological evaluation scales measuring biased behaviors against women. The Ambivalent Sexism Index, Modern Sexism Scale, and Belief in Sexism Shift evaluations were used to get descriptions of how the LLMs respond to traditional and modern prompts involving sexism and gender bias. The three evaluations used Likert scale response scores, providing quantitative scoring data. Results from evaluation trials were obtained using each LLMs API, collecting Likert scores in response to the evaluation prompts. Free-response data was also collected to understand output reasoning. Ordinal regression modeling with mixed effects aim to further understand how certain variables affect scoring. To understand patterns in reasoning, thematic analysis of the free response data was completed. Three significant themes were found across all model responses:<em> recognizing women’s challenges, understanding variation in gender experience, </em>and <em>feminism and progressive initiatives. </em>These themes emphasize the ways in which LLMs respond to biased statements against women.</p>

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Electrical Engineering and Computer Science

Year

2025

Author dc:creator

• Fieck, Sarah T

Contributors dc:contributor

• LouAnne Boyd, Ph.D.
• Chelsea Parlett, Ph.D.
• Elizabeth Stevens, Ph.D.

• Machine Learning
• Large Language Models
• Ethics
• Prompt Engineering
• Sexism
• Bias
• Data Science
• Social Statistics

Repository record dc:identifier

https://digitalcommons.chapman.edu/eecs_theses/7

OAI identifier oai:identifier

oai:digitalcommons.chapman.edu:eecs_theses-1007