{"id":{"repo_id":"mit","oai_identifier":"oai:dspace.mit.edu:1721.1/124078"},"canonical_url":"https://search.dev.ndltd.org/etd/mit/oai:dspace.mit.edu:1721.1/124078","repository":{"repo_id":"mit","name":"MIT","base_url":"https://dspace.mit.edu/oai/request"},"display":{"title":"Affective computing and crowdsourcing : subjective labels and sequential effects","abstract":"Online platforms displaying long streams of examples are often employed to gather labels from both experts and crowd workers. While previous work in crowdsourcing has focused on objective tasks and estimating error parameters of annotators, collecting labels in a subjective setting (e.g. emotion recognition) is more complicated due to different interpretations of a stimulus (i.e. emotion). These interpretations could be influenced by many factors such as annotator mood, previously observed examples, and previously produced labels. This thesis investigates subjectivity and sequential effects in emotion recognition using a generative neural network to synthetically generate faces and empathetic dialog evaluation through an interactive platform. Comparing annotator agreement reveals that inter-rater agreement is lower for female faces and faces displaying negative emotion. Shuffling examples such that all annotators see a different randomized order further decreases annotator agreement, yet our experiments also reveal positive auto-correlation between sequential labels when annotators see all examples are in the same order; likely due to affective priming. To model these effects, matrix factorization is found to be effective in improving aggregation of dialog evaluation ratings (low-annotator agreement) while the Dawid-Skene model [1] improves label aggregation for both dialog and emotion recognition ratings.","abstract_html":"Online platforms displaying long streams of examples are often employed to gather labels from both experts and crowd workers. While previous work in crowdsourcing has focused on objective tasks and estimating error parameters of annotators, collecting labels in a subjective setting (e.g. emotion recognition) is more complicated due to different interpretations of a stimulus (i.e. emotion). These interpretations could be influenced by many factors such as annotator mood, previously observed examples, and previously produced labels. This thesis investigates subjectivity and sequential effects in emotion recognition using a generative neural network to synthetically generate faces and empathetic dialog evaluation through an interactive platform. Comparing annotator agreement reveals that inter-rater agreement is lower for female faces and faces displaying negative emotion. Shuffling examples such that all annotators see a different randomized order further decreases annotator agreement, yet our experiments also reveal positive auto-correlation between sequential labels when annotators see all examples are in the same order; likely due to affective priming. To model these effects, matrix factorization is found to be effective in improving aggregation of dialog evaluation ratings (low-annotator agreement) while the Dawid-Skene model [1] improves label aggregation for both dialog and emotion recognition ratings.","abstract_has_math":false,"creators":["Shen, Judy Hanwen."],"institution":"Massachusetts Institute of Technology","degree_name":"Master","degree_level":null,"degree_discipline":null,"degree_department":"Program in Media Arts and Sciences (Massachusetts Institute of Technology)","school":null,"contributors":[],"advisors":["Rosalind W. Picard."],"committee_chairs":[],"committee_members":[],"year":2019,"date_issued":"2019","date_published":"2019","updated_at":"2026-07-22T22:21:08Z","subjects":["Program in Media Arts and Sciences"],"languages":["eng"],"rights":["MIT theses are protected by copyright. They may be viewed, downloaded, or printed from this source but further reproduction or distribution in any format is prohibited without written permission."],"rights_urls":["http://dspace.mit.edu/handle/1721.1/7582"],"identifier_entries":[]},"links":{"outbound_url":"https://hdl.handle.net/1721.1/124078","outbound_label":"Handle","outbound_source":"dc:identifier.uri"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor.advisor","label":"Advisor","values":["Rosalind W. Picard."]},{"key":"dc:contributor.department","label":"Department","values":["Program in Media Arts and Sciences (Massachusetts Institute of Technology)","Media"]},{"key":"dc:contributor.other","label":"Dc Contributor Other","values":["Program in Media Arts and Sciences (Massachusetts Institute of Technology)"]},{"key":"dc:creator","label":"Author","values":["Shen, Judy Hanwen."]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.accessioned","label":"Dc Date Accessioned","values":["2020-03-09T18:52:33Z"]},{"key":"dc:date.available","label":"Dc Date Available","values":["2020-03-09T18:52:33Z"]},{"key":"dc:date.issued","label":"Date","values":["2019"]},{"key":"dc:publisher","label":"Institution","values":["Massachusetts Institute of Technology"]},{"key":"dc:type","label":"Dc Type","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Master"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Program in Media Arts and Sciences"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language.iso","label":"Language (ISO)","values":["eng"]},{"key":"dc:rights","label":"Dc Rights","values":["MIT theses are protected by copyright. They may be viewed, downloaded, or printed from this source but further reproduction or distribution in any format is prohibited without written permission."]},{"key":"dc:rights.uri","label":"Rights URI","values":["http://dspace.mit.edu/handle/1721.1/7582"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier.uri","label":"Identifier URI","values":["https://hdl.handle.net/1721.1/124078"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Thesis: S.M., Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2019","Cataloged from PDF version of thesis.","Includes bibliographical references (pages 85-91)."]},{"key":"dc:description.abstract","label":"Abstract","values":["Online platforms displaying long streams of examples are often employed to gather labels from both experts and crowd workers. While previous work in crowdsourcing has focused on objective tasks and estimating error parameters of annotators, collecting labels in a subjective setting (e.g. emotion recognition) is more complicated due to different interpretations of a stimulus (i.e. emotion). These interpretations could be influenced by many factors such as annotator mood, previously observed examples, and previously produced labels. This thesis investigates subjectivity and sequential effects in emotion recognition using a generative neural network to synthetically generate faces and empathetic dialog evaluation through an interactive platform. Comparing annotator agreement reveals that inter-rater agreement is lower for female faces and faces displaying negative emotion. Shuffling examples such that all annotators see a different randomized order further decreases annotator agreement, yet our experiments also reveal positive auto-correlation between sequential labels when annotators see all examples are in the same order; likely due to affective priming. To model these effects, matrix factorization is found to be effective in improving aggregation of dialog evaluation ratings (low-annotator agreement) while the Dawid-Skene model [1] improves label aggregation for both dialog and emotion recognition ratings."]},{"key":"dc:description.degree","label":"Dc Description Degree","values":["S.M."]},{"key":"dc:title","label":"Title","values":["Affective computing and crowdsourcing : subjective labels and sequential effects"]}]}],"canonical_facts":{"dc:contributor.advisor":["Rosalind W. Picard."],"dc:contributor.department":["Program in Media Arts and Sciences (Massachusetts Institute of Technology)","Media"],"dc:contributor.other":["Program in Media Arts and Sciences (Massachusetts Institute of Technology)"],"dc:creator":["Shen, Judy Hanwen."],"dc:date.accessioned":["2020-03-09T18:52:33Z"],"dc:date.available":["2020-03-09T18:52:33Z"],"dc:date.issued":["2019"],"dc:description":["Thesis: S.M., Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2019","Cataloged from PDF version of thesis.","Includes bibliographical references (pages 85-91)."],"dc:description.abstract":["Online platforms displaying long streams of examples are often employed to gather labels from both experts and crowd workers. While previous work in crowdsourcing has focused on objective tasks and estimating error parameters of annotators, collecting labels in a subjective setting (e.g. emotion recognition) is more complicated due to different interpretations of a stimulus (i.e. emotion). These interpretations could be influenced by many factors such as annotator mood, previously observed examples, and previously produced labels. This thesis investigates subjectivity and sequential effects in emotion recognition using a generative neural network to synthetically generate faces and empathetic dialog evaluation through an interactive platform. Comparing annotator agreement reveals that inter-rater agreement is lower for female faces and faces displaying negative emotion. Shuffling examples such that all annotators see a different randomized order further decreases annotator agreement, yet our experiments also reveal positive auto-correlation between sequential labels when annotators see all examples are in the same order; likely due to affective priming. To model these effects, matrix factorization is found to be effective in improving aggregation of dialog evaluation ratings (low-annotator agreement) while the Dawid-Skene model [1] improves label aggregation for both dialog and emotion recognition ratings."],"dc:description.degree":["S.M."],"dc:identifier.uri":["https://hdl.handle.net/1721.1/124078"],"dc:language.iso":["eng"],"dc:publisher":["Massachusetts Institute of Technology"],"dc:rights":["MIT theses are protected by copyright. They may be viewed, downloaded, or printed from this source but further reproduction or distribution in any format is prohibited without written permission."],"dc:rights.uri":["http://dspace.mit.edu/handle/1721.1/7582"],"dc:subject":["Program in Media Arts and Sciences"],"dc:title":["Affective computing and crowdsourcing : subjective labels and sequential effects"],"dc:type":["Thesis"],"thesis:degree_name":["Master"]},"updated_at":"2026-07-22T22:21:08Z"}