Decision-making for a regenerative society: voice-based emotion AI

Voice-based user interfaces are increasingly present in our daily lives. These systems rely on voice as a key identifier and biometric, sometimes drawing from aggregated data sets to facilitate decision-making based on interpretations of our emotional state, gender, ethnicity and more. However, voice as an indicator of identity and intention is already a fraught part of human interaction in light of bias and difference. The starting hypothesis is that AI-mediated voice-based decision-making and the technology’s assumptions about a user are subjective. Our voices operating in today’s complex technology ecosystem have new effects of which we are not fully aware. This research aims to understand a user’s voice journey through technology and its affective interpretations as it relates to voice-based decision-making that affects a user’s contextual identity and privacy. The dissertation starts with a new media speculative study that explores the use of sound in design as an inclusive and revelatory process of finding ways of living that rely primarily on the perception of sound. Next, the research investigates how users understand the ways in which a wearable device equipped with a voice analyzer “hears.” The study builds on precedents in affective computing, algorithm auditing frameworks, folk theories of algorithmic understanding, and human-centric AI. The outcome points out competencies involved in understanding objects that function using sound-based artificial intelligence. The research continues with a virtual “thing” ethnography for the three components of a voice-based emotion AI: interface, data, and algorithm. Using the five-emotion framework and digital research techniques (e.g., media archaeology, performative inquiry) and following AI best practices, it iterates through three script performances, analyzing the state of the field in terms of technological imaginaries in data classification, voice emotion modelling, voice emotion model evaluation, data orchestration, and agency over voice emotion in a big-data world) and its maturity today. The “thing” ethnography, coupled with the research with human participants study, point to the limitations of using standard datasets and the opportunity for creating performant voice-based emotion AI with real people’s data. The benefit of AI transfer learning, coupled with personal storytelling for data orchestration and a speculative, phenomenological exploration with small data, represents the opportunity to regain agency over voice emotion in a big-data world.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Informatics

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2022

Author dc:creator

• Porteanu, Monica

Contributors dc:contributor

• Hamilton, Kevin
• Karahalios, Kyratso
• Grosser, Benjamin
• Araujo de Aguiar, Carlos

• data ethnography
• algorithm ethnography
• interface ethnography
• audible design
• design with sound
• voice-based design
• voice-based AI
• voice-based emotion AI
• algorithmic hearing
• tone analysis
• wearable computing
• affective computing
• voice sentiment analysis

Statement dc:rights

• Copyright 2022 Monica Porteanu

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/115694

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F115694