Using affective computing to enhance the believability of virtual agents and the social interaction with users

The drive to create believable virtual agents capable of engaging in natural social interactions with humans has fuelled extensive research efforts in affective computing, the field dedicated to developing systems that recognize and adapt to human emotions. Despite advancements in recent decades, the field continues to face challenges related to (1) responding accurately to human emotions and fostering a continuous affective loop, (2) finding reliable stimuli to elicit a wide range of affective states, (3) processing affective cues appropriately and (4) generalizing affect recognition across contexts. In an attempt to address these issues, this work proposed identifying general affective states that include thematically similar emotions. Equipping agents with a model capable of detecting such sets would enable them to provide thematically close responses, enhancing their emotional coherence. <br/><br/>To assess the potential of the approach, the first phase of the project used the eight subspaces of the Pleasure-Arousal-Dominance (PAD) model to define the general affective states of the user and to design agent responses. An experiment was conducted and included interactions with two agents that dynamically adjusted facial expressions and text in response to the eight general states. The second phase of the project explored horror games as a medium to elicit anxiety-related emotions within the ”Anxious” PAD subspace. A new tool was designed to help collect affect annotation over time. These annotations, along with physiological measures generated a new dataset. Using this dataset, classifiers were built to identify subclasses within the P, A and D components of the selected subspace, while investigating how varying the size and offset of time windows for processing physiological signals affected the classifiers' performance. The final phase of the project explored routes for context generalization using the proposed adaptive windowing technique.<br/><br/>Results indicated a positive overall perception of the agent’s believability when responding to eight general affective states, with further improvement observed when changes in the user’s physiological arousal were included. Adjusting the window settings for processing physiological signals led to improved prediction performance. Moreover, the implementation of the adaptive windowing technique demonstrated the potential to improve the model's ability to better characterize affective states across diverse datasets.

Name dc:type.qualificationname

PhD

Level dc:type.qualificationlevel

Doctoral Thesis

Grantor dc:publisher.institution

Abertay University

Year dc:date.issued

2025

Author dc:creator

• Graja, Sarra

Advisors dc:contributor.advisor

• Lovell, Paul
• Fee, Kenneth

• Affective Computing
• Virtual agents
• HCI
• Human-computer interaction
• Emotions
• Multimodal dataset
• Affective feedback
• Annotation tools
• AffectRankTrace
• Pleasure Arousal Dominance
• PAD
• Anxiety
• Physiology
• Heart Rate Variability
• Respiration
• Galvanic Skin Response
• Electrocardiography
• Anxiety-related emotions
• General affective state
• Emotion subspaces
• Thematically related emotions
• Agent behaviour
• Affect aware agent
• Believability
• Emotion adaptation
• Affective loop
• Signal processing
• Classification
• Machine learning
• Time windows segmentation
• Emotion recognition
• Anxious

Language dc:language

eng

Identifier

oai:rke.abertay.ac.uk:studenttheses/8ce7da5a-d034-4d9d-800f-f0e4d0e13c0a

OAI identifier oai:identifier

oai:rke.abertay.ac.uk:studenttheses/8ce7da5a-d034-4d9d-800f-f0e4d0e13c0a