Adaptation of Generative Models for Image Manipulation and Recontextualization

In recent works, generative models such as generative adversarial networks (GANs) and diffusion models have demonstrated a remarkable ability to mimic the image data distributions and to synthesize highly photo-realistic images representing diverse sets of concepts. Such models capture rich semantic information of the image data, and can potentially be used as a prior in solving image manipulation and recontextualization problems. In this work, we aim to study various challenges in employing generative models as priors in solving image attribute editing, intrinsic image decomposition, and one-shot image stylization. First, we discuss the challenge of inverting a pre-trained GAN, a crucial step in exploiting the rich GAN image priors, and how we can achieve a near-perfect GAN Inversion for accurate image reconstruction and attribute editing. We extend the framework of GAN Inversion to multiple GANs that allow for jointly leveraging multiple GAN priors for the successful decomposition of an image into its intrinsic components such as albedo, shading, and specular. Further, we propose a GAN-based one-shot stylization method that can stylize an input image into multiple styles at once while using only one example of each reference style. Since the GAN-based stylization approaches are typically limited to specific subject domains, we also propose a diffusion model-based one-shot stylization approach, ZipLoRA, that allows for generating any subject in any style along with text-driven recontextualization capabilities.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2024

Author dc:creator

• Shah, Viraj

Contributors dc:contributor

• Lazebnik, Svetlana
• Forsyth, David
• Schwing, Alexander
• Hasegawa-Johnson, Mark

• Generative Models
• Generative Adversarial Networks
• Diffusion Models
• Image Editing
• Image Stylization
• Intrinsic Image Decomposition
• Recontextualization

Statement dc:rights

• Copyright 2024 Viraj Shah

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/125619

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F125619