Generative 3D scene modeling: algorithm and applications

The ability to generate realistic and functional 3D scenes at scale is becoming increasingly important across domains such as agriculture, autonomous driving, immersive gaming, and robot learning. However, manually constructing such environments is prohibitively expensive and often introduces domain gaps from real-world distributions. Generative modeling of 3D scenes offers a scalable alternative, with the promise of producing vast quantities of coherent, diverse, and controllable environments directly from data. This thesis aims to advance generative 3D scene modeling with a particular emphasis on large, outdoor, and multi-modal environments. We define two desiderata for an ideal generative scene model: the ability to imagine worlds beyond the observed extent (scene generation), and the ability to reinterpret observed environments with alternative variations (scene editing). Realizing these goals is far from trivial, raising three key research challenges: (1) how to enable large-scale scene generation with limited 3D data; (2) how to move beyond single-modality generation to handle diverse and integrated scene modalities; and (3) how to ensure that generative scene models bring tangible benefits to downstream applications. Our approach to these challenges is developed along two complementary dimensions: algorithms and applications. Algorithmically, the guiding philosophy is to leverage powerful 2D generative priors to overcome the scarcity of high-quality 3D data. This leads to two chapters: (1) SuperGaussian, which adapts pre-trained video models for 3D scene editing, showing that video priors can be repurposed for high-fidelity 3D super-resolution; and (2) SGAM, a progressive framework that combines generative sensor modeling with reconstruction to create globally consistent, large-scale virtual worlds from RGB-D sequences. On the application side, we demonstrate how generative scene modeling can directly support real-world use cases. In particular, Sim-on-Wheels provides a safe and realistic framework for testing autonomous driving by augmenting real-world driving scenes with virtual tra"c scenarios, while MMCityGen showcases how multi-modal generation can power urban digital twins for planning and analysis. Together, these contributions establish a principled approach to repurposing 2D priors for 3D scene modeling, extend generation into the multi-modal domain, and demonstrate concrete uses in autonomy and urban planning.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois Urbana-Champaign

Year dc:date

2025

Author dc:creator

• Shen, Yuan

Contributors dc:contributor

• Wang, Shenlong
• Hoiem, Derek
• Forsyth, David
• Ceylan, Duygu

• 3d scene generation
• 3d scene editing

Statement dc:rights

• Copyright 2025 Yuan Shen

Language dc:language

en

Handle dc:identifier

https://hdl.handle.net/2142/132482

OAI identifier oai:identifier

oai:www.ideals.illinois.edu:2142/132482