A journey to photo-realistic facial animation synthesis
This dissertation presents preliminary work in facial animation generation with applications in educational psychology. In the ﬁrst part of this dissertation, we describe two psychology studies as well as the computer vision techniques and platforms being used. Both studies investigate using conversational agents (CAs) as a way of delivering medical messages to patients. By incorporating CAs in the system, both semantic and emotional information can be delivered, which helps the patients, especially those with low heath and numerical literacy, to get a better understanding of their test results and medical instructions. Human studies were conducted to test the eﬀectiveness of CA. In addition, the whole system was integrated with speech recognition as well as a natural language processing module to enable teach-back capability of the CA by providing a correct answer in case of a given wrong answer by the user, which will help them to get a better understanding of the medical messages being delivered. The second part of this dissertation documents the details of a proposed neural network based facial animation synthesis method. By unifying both appearance-based and warping-based methods in an end-to-end training process, the proposed system was able to generate vivid facial animation with highly preserved details. We show both qualitatively and quantitatively that the proposed system achieved a higher performance than baseline methods. In addition, visualization and ablation studies were conducted to further justify the eﬀectiveness of the proposed system. In the third part, the previous facial animation synthesis system was integrated with another audio speech processing system. The ﬁnal system was able to take speech signal and sample face images as input and generate the corresponding talking head animation as output. Comparison to the previous state-of-the-art method shows that the proposed system in this work achieves better performance.
facial animation synthesis, talking head, patient portal, deep learning, neural network, audio-driven facial animation synthesis
University of Illinois - Urbana-Champaign
Networked Digital Library of Theses and Dissertations
University of Illinois at Urbana-Champaign