Towards controllable and consistent 3D scene editing

Recent advancements in diffusion-based generative models have unlocked the potential for transformative 3D content creation. However, existing approaches for 3D scene editing face challenges in efficiency, consistency, and controllability. In this thesis, we introduce two novel methods that address these limitations and provide more effective means for editing 3D scenes. Our first exploration focuses on global style editing with text instructions. We present ViCA-NeRF, the first view-consistency-aware method for 3D editing with text instructions. By leveraging depth information derived from neural radiance fields (NeRF) and aligning latent codes in the 2D diffusion model, ViCA-NeRF ensures multi-view consistency through geometric and learned regularization. Our two-stage approach, involving edit blending and refinement, results in more flexible, efficient, and detailed editing compared to state-of-the-art methods. Secondly, we introduce DragGaussian, an interactive framework for fine-grained 3D scene editing using intuitive drag manipulation. Our method ensures 3D consistency through Gaussian deformation-based geometric guidance and an efficient inverse-free drag editing approach. By fine-tuning 2D diffusion models with a history-aware tuning strategy, DragGaussian enables high-quality drag-based editing of geometry and appearance, supporting various manipulations such as object movement, pose adjustment, and shape modification. Experimental results demonstrate the effectiveness and versatility of our proposed methods. ViCA-NeRF achieves a 3x speedup in editing time while maintaining higher levels of consistency and detail. DragGaussian enables diverse edits within a 10-20 minute timeframe on a single GPU, showcasing its efficiency. By combining text-based instructions and interactive point-based manipulation, our methods significantly advance the field of 3D scene editing, providing more efficient, consistent, and controllable tools for content creation.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2024

Author dc:creator

• Dong, Jiahua

Contributors dc:contributor

• Wang, Yuxiong

• 3d Editing
• Diffusion Model
• Computer Vision

Statement dc:rights

• Copyright 2024 Jiahua Dong

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/124714

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F124714