Algorithms & Systems for Differentiable Graphics Programming

Differentiable graphics representations are now a centerpiece for learning-based approaches in inverse rendering, novel-view synthesis, data & compute efficient rendering, and even 3D generative models. We see great advances in performance & fidelity when mixing classical wisdom from existing primitives like meshes and textures, and novel primitives like tiny neural networks. This cross-pollination of ML & graphics is key to these advances, but is held back due to complications: existing frameworks like PyTorch are ill-suited to graphics programming both due to algorithmic problems, like discontinuities, and system-design problems that lead to poor performance & expressive power. This thesis discusses several original approaches that were developed with generalizability in mind, to allow these approaches to apply broadly to different domains that struggle with the same problems. The first half of this thesis tackles discontinuities in a wide variety of applications, both (i) through a compiler that takes the problem-specific boundary sampling idea and automates it through compiler passes, and (ii) the warped-area reparameterization method that can be used to handle discontinuities by entirely removing the requirement of such problem-specific boundary samplers. We show how this enables light-weight integration with existing renderers by reparameterizing a mesh-based path tracer and a neural SDF renderer to make them fully differentiable. The second half will present the SLANG.D compiler, an industry collaboration that resulted in a high-performance compiler for next-generation differentiable & neural graphics systems. We discuss how the user-centric focus of SLANG.D’s automatic differentiation system enables users to write large-scale differentiable graphics pipelines and re-use 1000s of lines of existing rendering infrastructure without sacrificing its performance.

Name thesis:degree_name

Doctoral

Department dc:contributor.department

Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science

Grantor dc:publisher

Massachusetts Institute of Technology

Year dc:date.issued

2024

Author dc:creator

• Bangaru, Sai Praveen

Advisor dc:contributor.advisor

• Durand, Frédo

Statement dc:rights

• Attribution 4.0 International (CC BY 4.0)
• Copyright retained by author(s)

Licence dc:rights.uri

• https://creativecommons.org/licenses/by/4.0/

Handle dc:identifier.uri

https://hdl.handle.net/1721.1/156329

OAI identifier oai:identifier

oai:dspace.mit.edu:1721.1/156329