High resolution neural frontal face synthesis from face encodings using adversarial loss

In this thesis, we present a novel neural network method to synthesize a person's face imagery with frontal face and neutral expression, given a single unconstrained face photograph. We achieve this by a data-driven approach to train neural networks with a large-scale in-the-wild dataset of face images. The most common way to tackle this is supervised learning, which requires many ground-truth input-output pairs. Moreover, in our problem context, finding clean frontal and neutral expression faces without occlusions leads to other challenging problems. To avoid this, we take a neural knowledge transfer approach, where we first train modular networks for each well-defined sub-task and exploit them to instill semantic senses to train the face decoder, i.e., neutral face synthesizer. For sub-tasks, we utilize face landmark detection and recognition modules, where curated datasets exist. In particular, the face recognition sub-task learns features strongly invariant to lighting, pose, and facial expression variations. Given the recognition feature, we leverage this invariance to train our face decoder to produce consistent frontal and neutral expression faces, while constraining each generated face: 1) to be a forward facing pose using the network trained for the landmark detection, and 2) to preserve the same identity as the input face using the network trained for face recognition. Furthermore, we attempt to boost the realism of the output faces using adversarial loss, in which a discriminator competes with the generator network and guides the generation of higher quality faces. In test time, only the face recognition network and face decoder are used to synthesize neutral faces. Our approach does not require supervised data and further minimizes sensitive data pre-processing pipelines. Compared to competing fully-supervised methods, our method produces comparable or often even favorable face appearances.

Name thesis:degree_name

Master

Department dc:contributor.department

Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science

Grantor dc:publisher

Massachusetts Institute of Technology

Year dc:date.issued

2019

Author dc:creator

• Wang, Andy(Andy L.),M. Eng.Massachusetts Institute of Technology.

Advisor dc:contributor.advisor

• Wojciech Matusik.

• Electrical Engineering and Computer Science.

Statement dc:rights

• MIT theses are protected by copyright. They may be viewed, downloaded, or printed from this source but further reproduction or distribution in any format is prohibited without written permission.

Licence dc:rights.uri

• http://dspace.mit.edu/handle/1721.1/7582

Language dc:language.iso

eng

Handle dc:identifier.uri

https://hdl.handle.net/1721.1/123120

OAI identifier oai:identifier

oai:dspace.mit.edu:1721.1/123120