Beyond words: Understanding emotional shifts in maternal vocalizations through speech emotion recognition models

Automatic Speech Emotion Recognition (SER) has significant potential to provide insights into our understanding of dyadic communications. This study focuses on maternal vocalizations within mother-infant dyadic interactions, examining how mothers’ happy and neutral emotional tones shift in response to varying infant stress levels. To achieve this, we employ multiple models: our hybrid CNN-BiLSTM architecture, alongside pre-trained transformer-based models such as wav2vec 2.0 and HuBERT. Our evaluation demonstrates that the hybrid model outperforms these transformer-based approaches after fine-tuning, achieving a minimum improvement of 3.94 percentage points in the test accuracy and 11 percentage points in the weighted average of F1 scores in the IDP dataset. Using our fine-tuned model, we analyze maternal vocalizations in different age groups of infants (3, 6, and 9 months) and classify infants into low-, mid-, and high-stress categories based on the Root Mean Square (RMS) energy features of their vocalizations during stress-inducing events. Our findings reveal a moderate effect size (Cohen’s d) of associations between high stress levels and pronounced vocalization changes in mothers of 3-month-olds, more nuanced responses in mothers of 9-month-olds, and a balanced distribution of vocalization shifts in mothers of 6-month-olds. The novel application of SER in mother-infant studies underscores emotional adaptation in maternal vocalizations and its potential to expand analyses to bidirectional influences, providing deeper insights into emotional communication dynamics.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois Urbana-Champaign

Year dc:date

2025

Author dc:creator

• Tin, Alara

Contributors dc:contributor

• Hasegawa-Johnson, Mark A.

• Speech Emotion Recognition
• Cnn-bilstm
• Acoustic Features
• Domain Adaptation
• Mother-infant Interaction

Statement dc:rights

• Copyright 2025 Alara Tin

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/129636

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F129636