Towards Accurate and Fair Deepfake Detection

In this thesis, we tackle two problems in the field of deepfake detection. First, we present a novel approach for the detection of deepfake videos using a pair of vision transformers pre-trained by a self-supervised masked autoencoding setup. Our method consists of two distinct components, one of which focuses on learning spatial information from individual RGB frames of the video, while the other learns temporal consistency information from optical flow fields generated from consecutive frames. Unlike most approaches where pre-training is performed on a generic large corpus of images, we show that by pre-training on smaller face-related datasets, strong results can be obtained. We perform various experiments to evaluate the performance of our method on commonly used datasets. Our experiments show that our method sets a new state-of-the-art in various setups including cross-dataset generalization. Subsequently, we introduce FairAlign, a new method to reduce bias and improve fairness in deepfake detection by aligning conditional embedding distributions in a high-dimensional kernel space. Our approach reduces information related to sensitive attributes in the embedding space that could potentially bias the detection process, thus promoting fairness. FairAlign is a versatile plug-and-play loss term compatible with various deepfake detection networks and is capable of enhancing fairness without compromising detection performance. In addition to applying FairAlign for reducing gender bias, we implement a systematic pipeline for the annotation of skin tones and promotion of fairness in deepfake detection related to this sensitive attribute. Finally, we perform the first comprehensive study toward quantifying and understanding the trade-off between fairness and accuracy in the context of deepfake detection. We use public deepfake datasets to evaluate our method. Through various experiments, we observe that FairAlign outperforms other bias-mitigating methods across various deepfake detection backbones for both gender and skin tone, setting a new stateof-the-art. Moreover, our fairness-accuracy trade-off analysis demonstrates that our approach demonstrates the best overall performance when considering effectiveness in both deepfake detection and reducing bias.

Department dc:contributor.department

Electrical and Computer Engineering

Year dc:date.issued

2024

Author dc:creator

• Das, Sayantan

Advisor dc:contributor.supervisor

• Etemad, Ali

• Deepfake Detection
• Facial Forgery Detection
• Deep Learning
• Machine Learning
• Generative AI

Statement dc:rights

• Attribution 4.0 International

Licence dc:rights.uri

• http://creativecommons.org/licenses/by/4.0/

Language dc:language.iso

eng

Handle dc:identifier.uri

https://hdl.handle.net/1974/32812

OAI identifier oai:identifier

oai:queensu.scholaris.ca:1974/32812