Harnessing data priors to mitigate 3D data scarcity

Recently, we have witnessed remarkable advances in various fields related to machine learning and artificial intelligence, e.g., the rise of Large Language Models (LLMs) for natural language processing and diffusion models for visual content generation. Alongside improvements in algorithm design, the ability to train models on massive data has undoubtedly been a cornerstone of such progress. However, not all fields or tasks have the privilege of obtaining data of a similar scale. This naturally raises the question: how can we address the challenge of data scarcity in areas where large datasets are unavailable? In this dissertation, we study this challenge in the context of 3D computer vision and demonstrate how harnessing data priors from various domains can help mitigate 3D data scarcity. We begin by focusing on the task of dynamic view synthesis. By incorporating various data priors, e.g., those from a pre-trained static view synthesis model, we develop a system that is capable of producing high-quality free-viewpoint and free-time rendering from a monocular video without access to large-scale real-world multiview 4D (3D + time) data. In the domain of category-specific 3D content generation, we propose to leverage data priors from a pre-trained 2D Generative Adversarial Network (GAN), enabling us to obtain a 3D-aware GAN model very efficiently without relying on real-world multiview 3D data. Additionally, we utilize data priors from a pre-trained text-to-image diffusion model to tackle the task of 3D lighting, a new paradigm that outperforms many state-of-the-art inverse rendering approaches, without the need for extensive real-world multiview 3D relighting data. Finally, we conclude this dissertation with promising future directions.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2024

Author dc:creator

• Zhao, Xiaoming

Contributors dc:contributor

• Schwing, Alexander Gerhard
• Hoiem, Derek W
• Wang, Shenlong
• Colburn, Alex

• 3d Computer Vision
• Novel View Synthesis
• Dynamic Novel View Synthesis
• Generative Adversarial Network
• Diffusion Model
• 3d Relighting

Statement dc:rights

• Copyright 2024 Xiaoming Zhao

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/127176

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F127176