A journey to photo-realistic facial animation synthesis

This dissertation presents preliminary work in facial animation generation with applications in educational psychology. In the first part of this dissertation, we describe two psychology studies as well as the computer vision techniques and platforms being used. Both studies investigate using conversational agents (CAs) as a way of delivering medical messages to patients. By incorporating CAs in the system, both semantic and emotional information can be delivered, which helps the patients, especially those with low heath and numerical literacy, to get a better understanding of their test results and medical instructions. Human studies were conducted to test the effectiveness of CA. In addition, the whole system was integrated with speech recognition as well as a natural language processing module to enable teach-back capability of the CA by providing a correct answer in case of a given wrong answer by the user, which will help them to get a better understanding of the medical messages being delivered. The second part of this dissertation documents the details of a proposed neural network based facial animation synthesis method. By unifying both appearance-based and warping-based methods in an end-to-end training process, the proposed system was able to generate vivid facial animation with highly preserved details. We show both qualitatively and quantitatively that the proposed system achieved a higher performance than baseline methods. In addition, visualization and ablation studies were conducted to further justify the effectiveness of the proposed system. In the third part, the previous facial animation synthesis system was integrated with another audio speech processing system. The final system was able to take speech signal and sample face images as input and generate the corresponding talking head animation as output. Comparison to the previous state-of-the-art method shows that the proposed system in this work achieves better performance.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2020

Author dc:creator

• Gu, Kuangxiao

Contributors dc:contributor

• Hasegawa-Johnson, Mark
• Huang, Thomas S.
• Morrow, Daniel G.
• Liang, Zhi-Pei
• Shi, Honghui

• facial animation synthesis
• talking head
• patient portal
• deep learning
• neural network
• audio-driven facial animation synthesis

Statement dc:rights

• Copyright 2020 Kuangxiao Gu

Language dc:language

en

Handle dc:identifier

http://hdl.handle.net/2142/108579

OAI identifier oai:identifier

oai:www.ideals.illinois.edu:2142/108579