Modeling and editing 4D scenes by leveraging structural priors

4D scene understanding powers applications ranging from AR/VR and robotics to controllable video generation. The goal is to build representations that can faithfully model and manipulate real-world environments as they evolve over time—capturing both how scenes deform (modeling) and how they respond to high-level user instructions (editing). A popular approach is to represent scenes using compact geometric primitives such as 3D Gaussians, which enable efficient rendering and temporal consistency across frames. While recent advances in 3D reconstruction and video editing have shown promising results, many existing methods still overlook a key aspect: structure. Whether in the form of spatial rigidity or semantic hierarchy, structural priors are both abundant and underexplored. This thesis investigates how incorporating such priors—geometric and semantic—into 4D scene modeling and editing can enhance efficiency, controllability, and generalization. The first part of this thesis focuses on geometric structural priors in dynamic 3D modeling. Many dynamic scenes exhibit coherent change patterns: objects often deform in groups, move rigidly or semi-rigidly, or follow interpretable part-wise trajectories. Instead of modeling motion independently for each element, we propose a structural cascaded optimization framework that organizes 3D Gaussians into a coarse-to-fine hierarchy. This structure allows us to parameterize deformation using simple transformations—rotation, translation, and scaling—substantially accelerating optimization. It also enables dense point tracking and motion-based segmentation without requiring semantic labels. These results demonstrate the potential of structured representations for fast and interpretable 4D scene modeling. The second part explores semantic structural priors in video editing. User instructions often involve multiple entangled goals that are difficult to fulfill through a single transformation. To address this, we employ large language models (LLMs) to decompose complex prompts into interpretable semantic subgoals. Each subgoal defines an editing stage, executed within a training-free diffusion-based video editing framework. To accommodate varying subgoal complexity, we further prompt the LLM to estimate editing difficulty and adapt the interpolation schedule accordingly. This results in smoother transitions and robust edits, transforming the process into a semantically grounded and interpretable sequence. Together, these contributions highlight the value of structural reasoning in 4D scene understanding. By bridging geometric modeling and semantic editing, this thesis offers unified insights into building efficient, robust, and controllable 4D systems guided by structural priors.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois Urbana-Champaign

Year dc:date

2025

Author dc:creator

• Lyu, Jipeng

Contributors dc:contributor

• Wang, Yuxiong

• 4D Scene Understanding
• Geometric Structural Priors
• Semantic Video Editing
• 3D Gaussian Splatting
• Diffusion Models

Statement dc:rights

• Copyright 2025 Jipeng Lyu

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/129184

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F129184