How deep learning can help emotion recognition

As technological systems become more and more advanced, the need for including the human during the interaction process has become more apparent. One simple way is to have the computer system understand and respond to the human's emotions. Previous works in emotion recognition have focused on improving performance by incorporating domain knowledge into the underlying system either through pre-specified rules or hand-crafted features. However, in the last few years, learned feature representations have experienced a resurgence mainly due to the success of deep neural networks. In this dissertation, we highlight how deep neural networks, when applied to emotion recognition, can learn representations that not only achieve superior accuracy to hand-crafted techniques, but also align with previous domain knowledge. Moreover, we show how these learned representations can generalize to different definitions of emotions and to different input modalities. The first part of this dissertation considers the task of categorical emotion recognition on images. We show how a convolutional neural network (CNN) that achieves state-of-the-art performance can also learn features that strongly correspond to Facial Action Units (FAUs). In the second part, we focus our attention on emotion recognition in video. We take the image-based CNN model and combine it with a recurrent neural network (RNN) in order to do dimensional emotion recognition. We also visualize the portions of the faces that most strongly affect the output prediction by using the gradient as a saliency map. Lastly, we explore the merit of doing multimodal emotion recognition by combining our model with other models trained on audio and physiological data.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2017

Author dc:creator

• Khorrami, Pooya Rezvani

Contributors dc:contributor

• Huang, Thomas S.
• Hasegawa-Johnson, Mark
• Hoiem, Derek W.
• Liang, Zhi-Pei

• Emotion recognition
• Deep learning
• Machine learning
• Computer vision
• Facial expression recognition
• Affective computing
• Deep neural networks

Statement dc:rights

• Copyright 2017 Pooya Rezvani Khorrami

Language dc:language

en

Handle dc:identifier

http://hdl.handle.net/2142/97284

OAI identifier oai:identifier

oai:www.ideals.illinois.edu:2142/97284