{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/105104"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/105104","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Exposing and correcting the gender bias in image captioning datasets and models","abstract":"The task of image captioning implicitly involves gender identification. However, due to the gender bias in data, gender identification by an image captioning model suffers. Also, due to the word-by-word prediction, the gender-activity bias in the data tends to influence the other words in the caption, resulting in the well know problem of label bias. In this work, we investigate gender bias in the COCO captioning dataset, and show that it engenders not only from the statistical distribution of genders with contexts but also from the flawed per instance annotation provided by the human annotators. We then look at the issues created by this bias in the models trained on the data. We propose a technique to get rid of the bias by splitting the task into 2 subtasks: gender-neutral image captioning and gender classification. By this decoupling, the gender-context influence can be eradicated. We train a gender neutral image captioning model, which does not exhibit the language model based bias arising from the gender and gives good quality captions. This model gives comparable results to a gendered model even when evaluating against a dataset that possesses similar bias as the training data. Interestingly, the predictions by this model on images without humans, are also visibly different from the one trained on gendered captions. For injecting gender into the captions, we train gender classifiers using cropped portions of images that contain only the person. This allows us to get rid of the context and focus on the person to predict the gender. We train bounding box based and body mask based classifiers, giving a much higher accuracy in gender prediction than an image captioning model implicitly attempting to classify the gender from the full image. By substituting the genders into the gender-neutral captions, we get the final gendered predictions. Our predictions achieve similar performance to a model trained with gender, and at the same time are devoid of gender bias. Finally, our main result is that on an anti-stereotypical dataset, our model outperforms a popular image captioning model which is trained with gender.","abstract_html":"The task of image captioning implicitly involves gender identification. However, due to the gender bias in data, gender identification by an image captioning model suffers. Also, due to the word-by-word prediction, the gender-activity bias in the data tends to influence the other words in the caption, resulting in the well know problem of label bias. In this work, we investigate gender bias in the COCO captioning dataset, and show that it engenders not only from the statistical distribution of genders with contexts but also from the flawed per instance annotation provided by the human annotators. We then look at the issues created by this bias in the models trained on the data. We propose a technique to get rid of the bias by splitting the task into 2 subtasks: gender-neutral image captioning and gender classification. By this decoupling, the gender-context influence can be eradicated. We train a gender neutral image captioning model, which does not exhibit the language model based bias arising from the gender and gives good quality captions. This model gives comparable results to a gendered model even when evaluating against a dataset that possesses similar bias as the training data. Interestingly, the predictions by this model on images without humans, are also visibly different from the one trained on gendered captions. For injecting gender into the captions, we train gender classifiers using cropped portions of images that contain only the person. This allows us to get rid of the context and focus on the person to predict the gender. We train bounding box based and body mask based classifiers, giving a much higher accuracy in gender prediction than an image captioning model implicitly attempting to classify the gender from the full image. By substituting the genders into the gender-neutral captions, we get the final gendered predictions. Our predictions achieve similar performance to a model trained with gender, and at the same time are devoid of gender bias. Finally, our main result is that on an anti-stereotypical dataset, our model outperforms a popular image captioning model which is trained with gender.","abstract_has_math":false,"creators":["Bhargava, Shruti"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"M.S.","degree_level":"Thesis","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":["Forsyth, David Alexander"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2019,"date_issued":"2019-08-23T20:36:13Z","date_published":"2019-08-23T20:36:13Z","updated_at":"2026-07-22T22:24:44Z","subjects":["Image captioning","gender bias","machine learning","deep learning","fairness"],"languages":["en"],"rights":["Copyright 2019 Shruti Bhargava"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"http://hdl.handle.net/2142/105104","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Forsyth, David Alexander"]},{"key":"dc:creator","label":"Author","values":["Bhargava, Shruti"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2019-08-23T20:36:13Z","2021-08-24T09:15:20Z","2019-04-26","2019-05"]},{"key":"dc:type","label":"Dc Type","values":["text"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["M.S."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Image captioning","gender bias","machine learning","deep learning","fairness"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2019 Shruti Bhargava"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["http://hdl.handle.net/2142/105104"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["The task of image captioning implicitly involves gender identification. However, due to the gender bias in data, gender identification by an image captioning model suffers. Also, due to the word-by-word prediction, the gender-activity bias in the data tends to influence the other words in the caption, resulting in the well know problem of label bias. In this work, we investigate gender bias in the COCO captioning dataset, and show that it engenders not only from the statistical distribution of genders with contexts but also from the flawed per instance annotation provided by the human annotators. We then look at the issues created by this bias in the models trained on the data. We propose a technique to get rid of the bias by splitting the task into 2 subtasks: gender-neutral image captioning and gender classification. By this decoupling, the gender-context influence can be eradicated. We train a gender neutral image captioning model, which does not exhibit the language model based bias arising from the gender and gives good quality captions. This model gives comparable results to a gendered model even when evaluating against a dataset that possesses similar bias as the training data. Interestingly, the predictions by this model on images without humans, are also visibly different from the one trained on gendered captions. For injecting gender into the captions, we train gender classifiers using cropped portions of images that contain only the person. This allows us to get rid of the context and focus on the person to predict the gender. We train bounding box based and body mask based classifiers, giving a much higher accuracy in gender prediction than an image captioning model implicitly attempting to classify the gender from the full image. By substituting the genders into the gender-neutral captions, we get the final gendered predictions. Our predictions achieve similar performance to a model trained with gender, and at the same time are devoid of gender bias. Finally, our main result is that on an anti-stereotypical dataset, our model outperforms a popular image captioning model which is trained with gender.","Submission published under a 24 month embargo labeled 'U of I Access', the embargo will last until 2021-05-01","The student, Shruti Bhargava, accepted the attached license on 2019-04-25 at 16:11.","The student, Shruti Bhargava, submitted this Thesis for approval on 2019-04-25 at 16:20.","This Thesis was approved for publication on 2019-04-26 at 08:29.","DSpace SAF Submission Ingestion Package generated from Vireo submission #13927 on 2019-08-22 at 15:08:46","Made available in DSpace on 2019-08-23T20:36:13Z (GMT). No. of bitstreams: 2 BHARGAVA-THESIS-2019.pdf: 6140852 bytes, checksum: 4de862cc12a2bc1c2879fdebf62bac27 (MD5) LICENSE.txt: 4212 bytes, checksum: d8a74d2c97863fcdc54052370992c353 (MD5) Previous issue date: 2019-04-26","Embargo set by: Seth Robbins for item 112223 Lift date: 2021-08-23T20:36:18Z Reason: Author requested U of Illinois access only (OA after 2yrs) in Vireo ETD system","U of I Only Restriction Lifted for Item 112223 on 2021-08-24T09:15:20Z."]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Exposing and correcting the gender bias in image captioning datasets and models"]}]}],"canonical_facts":{"dc:contributor":["Forsyth, David Alexander"],"dc:creator":["Bhargava, Shruti"],"dc:date":["2019-08-23T20:36:13Z","2021-08-24T09:15:20Z","2019-04-26","2019-05"],"dc:description":["The task of image captioning implicitly involves gender identification. However, due to the gender bias in data, gender identification by an image captioning model suffers. Also, due to the word-by-word prediction, the gender-activity bias in the data tends to influence the other words in the caption, resulting in the well know problem of label bias. In this work, we investigate gender bias in the COCO captioning dataset, and show that it engenders not only from the statistical distribution of genders with contexts but also from the flawed per instance annotation provided by the human annotators. We then look at the issues created by this bias in the models trained on the data. We propose a technique to get rid of the bias by splitting the task into 2 subtasks: gender-neutral image captioning and gender classification. By this decoupling, the gender-context influence can be eradicated. We train a gender neutral image captioning model, which does not exhibit the language model based bias arising from the gender and gives good quality captions. This model gives comparable results to a gendered model even when evaluating against a dataset that possesses similar bias as the training data. Interestingly, the predictions by this model on images without humans, are also visibly different from the one trained on gendered captions. For injecting gender into the captions, we train gender classifiers using cropped portions of images that contain only the person. This allows us to get rid of the context and focus on the person to predict the gender. We train bounding box based and body mask based classifiers, giving a much higher accuracy in gender prediction than an image captioning model implicitly attempting to classify the gender from the full image. By substituting the genders into the gender-neutral captions, we get the final gendered predictions. Our predictions achieve similar performance to a model trained with gender, and at the same time are devoid of gender bias. Finally, our main result is that on an anti-stereotypical dataset, our model outperforms a popular image captioning model which is trained with gender.","Submission published under a 24 month embargo labeled 'U of I Access', the embargo will last until 2021-05-01","The student, Shruti Bhargava, accepted the attached license on 2019-04-25 at 16:11.","The student, Shruti Bhargava, submitted this Thesis for approval on 2019-04-25 at 16:20.","This Thesis was approved for publication on 2019-04-26 at 08:29.","DSpace SAF Submission Ingestion Package generated from Vireo submission #13927 on 2019-08-22 at 15:08:46","Made available in DSpace on 2019-08-23T20:36:13Z (GMT). No. of bitstreams: 2 BHARGAVA-THESIS-2019.pdf: 6140852 bytes, checksum: 4de862cc12a2bc1c2879fdebf62bac27 (MD5) LICENSE.txt: 4212 bytes, checksum: d8a74d2c97863fcdc54052370992c353 (MD5) Previous issue date: 2019-04-26","Embargo set by: Seth Robbins for item 112223 Lift date: 2021-08-23T20:36:18Z Reason: Author requested U of Illinois access only (OA after 2yrs) in Vireo ETD system","U of I Only Restriction Lifted for Item 112223 on 2021-08-24T09:15:20Z."],"dc:format":["application/pdf"],"dc:identifier":["http://hdl.handle.net/2142/105104"],"dc:language":["en"],"dc:rights":["Copyright 2019 Shruti Bhargava"],"dc:subject":["Image captioning","gender bias","machine learning","deep learning","fairness"],"dc:title":["Exposing and correcting the gender bias in image captioning datasets and models"],"dc:type":["text"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["Thesis"],"thesis:degree_name":["M.S."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:24:44Z"}