Usability testing, or user experience (UX) testing, is increasingly recognized as an important part of the user interface design process. However, evaluating usability tests can be expensive in terms of time and resources and can lack consistency between human evaluators. This makes automation an appealing expansion or alternative to conventional usability techniques.
Early usability automation focused on evaluating human behavior through quantitative metrics but the explosion of opinion mining and sentiment analysis applications in recent decades has led to exciting new possibilities for usability evaluation methods.
This paper presents a survey of modern, open-source sentiment analyzers’ usefulness in extracting and correctly identifying moments of semantic significance in the context of recorded mock usability evaluations. Though our results did not find a text-based sentiment analyzer that could correctly parse moments as well as human evaluators, one analyzer was found to be able to parse positive moments found through audio-only cues as well as human evaluators. Further research into adjusting settings on current sentiment analyzers for usability evaluations and using multimodal tools instead of text-based analyzers could produce valuable tools for usability evaluations when used in conjunction with human evaluators.
","abstract_html":"<p>Usability testing, or user experience (UX) testing, is increasingly recognized as an important part of the user interface design process. However, evaluating usability tests can be expensive in terms of time and resources and can lack consistency between human evaluators. This makes automation an appealing expansion or alternative to conventional usability techniques.</p> <p>Early usability automation focused on evaluating human behavior through quantitative metrics but the explosion of opinion mining and sentiment analysis applications in recent decades has led to exciting new possibilities for usability evaluation methods.</p> <p>This paper presents a survey of modern, open-source sentiment analyzers’ usefulness in extracting and correctly identifying moments of semantic significance in the context of recorded mock usability evaluations. Though our results did not find a text-based sentiment analyzer that could correctly parse moments as well as human evaluators, one analyzer was found to be able to parse positive moments found through audio-only cues as well as human evaluators. Further research into adjusting settings on current sentiment analyzers for usability evaluations and using multimodal tools instead of text-based analyzers could produce valuable tools for usability evaluations when used in conjunction with human evaluators.</p>","abstract_has_math":false,"creators":["Van Damme, Kelsi"],"institution":null,"degree_name":"MS in Computer Science","degree_level":null,"degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":["Franz Kurfess","Computer Science","College of Engineering"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2021,"date_issued":"2021-06-01T07:00:00Z","date_published":"2021-06-01T07:00:00Z","updated_at":"2026-07-24T01:32:42Z","subjects":["Usability","User Experience","Sentiment Analysis","Agreement","Automation","Affectual","Data Science","Graphics and Human Computer Interfaces","Software Engineering"],"languages":[],"rights":[],"rights_urls":[],"identifier_entries":[{"key":"dc:identifier","label":"Identifier","values":["10.15368/theses.2021.107"],"render_values":[{"text":"10.15368/theses.2021.107","href":"https://doi.org/10.15368/theses.2021.107","code":true}]}]},"links":{"outbound_url":"https://digitalcommons.calpoly.edu/theses/2609","outbound_label":"Repository record","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Franz Kurfess","Computer Science","College of Engineering"]},{"key":"dc:creator","label":"Author","values":["Van Damme, Kelsi"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.available","label":"Dc Date Available","values":["2021-06-11T07:00:00Z"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_name","label":"Degree Name","values":["MS in Computer Science"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Usability","User Experience","Sentiment Analysis","Agreement","Automation","Affectual","Data Science","Graphics and Human Computer Interfaces","Software Engineering"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://digitalcommons.calpoly.edu/theses/2609","10.15368/theses.2021.107"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["Usability testing, or user experience (UX) testing, is increasingly recognized as an important part of the user interface design process. However, evaluating usability tests can be expensive in terms of time and resources and can lack consistency between human evaluators. This makes automation an appealing expansion or alternative to conventional usability techniques.
Early usability automation focused on evaluating human behavior through quantitative metrics but the explosion of opinion mining and sentiment analysis applications in recent decades has led to exciting new possibilities for usability evaluation methods.
This paper presents a survey of modern, open-source sentiment analyzers’ usefulness in extracting and correctly identifying moments of semantic significance in the context of recorded mock usability evaluations. Though our results did not find a text-based sentiment analyzer that could correctly parse moments as well as human evaluators, one analyzer was found to be able to parse positive moments found through audio-only cues as well as human evaluators. Further research into adjusting settings on current sentiment analyzers for usability evaluations and using multimodal tools instead of text-based analyzers could produce valuable tools for usability evaluations when used in conjunction with human evaluators.
"]},{"key":"dc:title","label":"Title","values":["A Performance Survey of Text-Based Sentiment Analysis Methods for Automating Usability Evaluations"]}]}],"canonical_facts":{"dc:contributor":["Franz Kurfess","Computer Science","College of Engineering"],"dc:creator":["Van Damme, Kelsi"],"dc:date.available":["2021-06-11T07:00:00Z"],"dc:description.abstract":["Usability testing, or user experience (UX) testing, is increasingly recognized as an important part of the user interface design process. However, evaluating usability tests can be expensive in terms of time and resources and can lack consistency between human evaluators. This makes automation an appealing expansion or alternative to conventional usability techniques.
Early usability automation focused on evaluating human behavior through quantitative metrics but the explosion of opinion mining and sentiment analysis applications in recent decades has led to exciting new possibilities for usability evaluation methods.
This paper presents a survey of modern, open-source sentiment analyzers’ usefulness in extracting and correctly identifying moments of semantic significance in the context of recorded mock usability evaluations. Though our results did not find a text-based sentiment analyzer that could correctly parse moments as well as human evaluators, one analyzer was found to be able to parse positive moments found through audio-only cues as well as human evaluators. Further research into adjusting settings on current sentiment analyzers for usability evaluations and using multimodal tools instead of text-based analyzers could produce valuable tools for usability evaluations when used in conjunction with human evaluators.
"],"dc:identifier":["https://digitalcommons.calpoly.edu/theses/2609","10.15368/theses.2021.107"],"dc:subject":["Usability","User Experience","Sentiment Analysis","Agreement","Automation","Affectual","Data Science","Graphics and Human Computer Interfaces","Software Engineering"],"dc:title":["A Performance Survey of Text-Based Sentiment Analysis Methods for Automating Usability Evaluations"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_name":["MS in Computer Science"]},"updated_at":"2026-07-24T01:32:42Z"}