A Mixed Emotions Framework And Its Applications In Affective Computing

Traditionally, computer vision has categorized displayed emotions into seven basic categories.: sadness, happiness, anger, fear, disgust, surprise, and neutral. This approach is restrictive because human emotions do not always fall neatly into these seven categories but often cross categorical borders, forming interesting mixtures. To address this problem, we developed a methodology based on two components: 1) CNN for simple emotions: A convolutional neural network (CNN) that identifies the seven emotions. We validated this CNN on RAVDESS, a well-known dataset of facial videos of people expressing distinct emotions while talking. 2) Co-occurrence matrix for mixed emotions: A post-hoc mixed emotion generation method, where the original seven-emotion probabilistic vector output by the CNN is used in an outer-product with itself to produce a co-occurrence matrix. The diagonal of this co-occurrence matrix holds the adjusted probabilities of the seven emotions. In contrast, the upper and lower triangles hold all their pair-wise combinations, which contain the corresponding probabilities of mixed emotions. The methodology takes probabilistic strength away from the original seven-emotion vector and allocates it to mixed emotions. We tested the novel mixed-emotion vs. the conventional seven-emotion methodologies in two naturalistic experiments: First, in an experiment focusing on the emotional effects of email interruptions during cognitive work, and second, in an experiment focusing on the emotional effects of online public speaking. In the email interruption experiment, the mixed-emotion methodology uniquely determined that, unlike knowledge workers who work uninterrupted, knowledge workers who are frequently interrupted by emails tend to display a mixture of sadness and fear on their faces. The latter presumably results from re-occurring negative stimuli in random email arrivals. The mixed-emotion methodology in the online public speaking experiment uniquely determined that more conscientious participants displayed a preponderance of mixed emotions concerning less conscientious participants. Mixed emotions represent a moderation of pure emotions, like angry-neutral vs. totally angry, and constitute a more acceptable visual communication between the speaker and the audience. Altogether, the mixed-emotion methodology significantly enhances analytical insight across different experimental settings.

Name thesis:degree_name

Doctor of Philosophy

Level thesis:degree_level

Doctoral

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Houston

Year dc:date.issued

2024

Author dc:creator

• Wesley, Amanveer

Advisor dc:contributor.advisor

• Pavlidis, Ioannis

Committee members dc:contributor.committeemember

• Vilalta, Ricardo
• Tsekos, Nikolaos
• Cirino, Paul

• Affective Computing
• Valence
• Mixed Emotions

Language dc:language.iso

en

Handle dc:identifier.uri

https://hdl.handle.net/10657/17765

OAI identifier oai:identifier

oai:uh-ir.tdl.org:10657/17765