From objects to worlds: scalable learning of 3D assets

Learning to reconstruct and generate the 3D world is a fundamental research problem in computer vision, with critical applications across diverse domains. However, the development of robust 3D generation and reconstruction systems is hindered by the scarcity of high-quality 3D data. This thesis aims to address this scaling challenge along several dimensions. First, we introduce ShapeClipper, which leverages semantic consistency from unlabeled 2D images to learn 3D shape reconstruction models. This enables scalable 3D learning from only single-view images, without any 3D annotations. Second, we present PointInfinity, a resolution-invariant point diffusion model for learning continuous 3D surfaces from point clouds. PointInfinity facilitates 3D learning using noisy point clouds derived from object-centric videos. Third, we introduce ZeroShape and re-examine the classical regression-based 3D reconstruction approach. We show it outperforms diffusion methods in accuracy, as well as computational and data efficiency. Finally, we explore the feasibility of learning 3D from in-the-wild videos without any 3D prior or data. As an initial yet solid step, we evaluate the 3D awareness of recent video foundation models, and find that state-of-the-art video generative models already possess strong 3D understanding. Together, this thesis makes significant advancements in scalable learning of 3D, providing practical solutions for reconstruction and generating 3D objects and worlds under limited high-quality 3D data.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois Urbana-Champaign

Year dc:date

2025

Author dc:creator

• Huang, Zixuan

Contributors dc:contributor

• Rehg, James M.
• Schwing, Alexander
• Wang, Shenlong
• Wu, Jiajun
• Vedaldi, Andrea

• 3D Generation
• 3D Reconstruction
• Video Generation
• World Models

Statement dc:rights

• Copyright 2025 Zixuan Huang

Language dc:language

en

Handle dc:identifier

https://hdl.handle.net/2142/132639

OAI identifier oai:identifier

oai:www.ideals.illinois.edu:2142/132639