Development of an Emotion Recognition Classifier from Body Language Using Deep Learning for the Children with Autism to Help Identifying Human Emotions

This thesis focuses on training the machine learning model to recognize emotions from a person’s body language. This research aims to develop an emotion recognition classifier for children with autism spectrum disorder (ASD) to help recognize emotions in social interaction. It classifies six universal emotions- “anger,” “sad,” “happy,” “surprise,” “fear,” and “disgust.” Video data and image data have been used to train three different models. Three different datasets have been used- GEMEP, BEAST, and “7453 IRB” datasets. The BEAST dataset is an image dataset, and the GEMEP and “7453 IRB” are video audio datasets. Three deep learning algorithms were tested to classify emotions from body language: Convolution Neural Network (CNN), the combination of CNN and Recurrent Neural Network (CNN+RNN), and the combination of CNN and Long Short-Term Memory (CNN+LSTM). For this research, though the CNN model provides better accuracy than CNN+RNN and CNN+LSTM, CNN model ignores the temporal information from the sequenced video data. The CNN+RNN and CNN+LSTM model provides more realistic results than CNN because it considers both spatial and temporal information in the data, which is essential for analyzing video data. Same hyperparameters from CNN+RNN were used to train the CNN+LSTM model due to limited time and limited computation resources. For this experiment, CNN+RNN outperforms the CNN+LSTM with a 12.33% higher test accuracy for the combined video datasets. However, with better hyperparameter tuning, the performance of CNN+LSTM model can be improved.

Name thesis:degree_name

Master of Science

Level thesis:degree_level

Masters

Discipline thesis:degree_discipline

Electrical Engineering

Grantor

Texas State University

Year dc:date.issued

2021

Author dc:creator

• Rashid, Tamima

Advisor dc:contributor.advisor

• Valles, Damian

Committee members dc:contributor.committeemember

• Resendiz, Maria
• Stapleton, William A.

• emotion classification
• body language
• CNN
• CNN+RNN
• CNN+LSTM
• ResNet50
• deep learning

Language dc:language.iso

en

Handle dc:identifier.uri

https://hdl.handle.net/10877/17843

OAI identifier oai:identifier

oai:digital.library.txst.edu:10877/17843