Hardware acceleration of neural graphics

Neural graphics is a rapidly evolving domain which uses neural networks to replace entire or parts of rendering pipeline. There are two broad classes of neural graphics: 1) Non-generative neural graphics where conventional DNNs are used for rendering. 2) Generative neural graphics where generative models such as latent diffusion models are employed. This dissertation studies the performance characteristic of both non-generative as well as generative neural graphics pipelines on modern hardware platforms, and asks the question: Does neural graphics need hardware support? The research is segmented into three primary areas: non-generative neural graphics via Neural Representations (NRs), generative neural graphics via Latent Diffusion Models (LDMs), and performance enhancements through integrating neural graphics with relatively more mature neural super-resolution techniques. Firstly, we study non-generative neural graphics by focusing on Neural Representations (NRs) which form the backbone of various applications like Neural Radiance and Density Fields (NeRF), Neural Signed Distance Functions (NSDF), Neural Volume Rendering (NVR), and Gigapixel Image Approximation (GIA). We identify significant performance gaps when rendering 4K resolution frames at 60 FPS on current GPUs, with a shortfall ranging from 1.51X to 55.50X. To bridge this gap, we propose the Neural Graphics Processing Cluster (NGPC), an architecture that accelerates input encoding and multi-layer perceptron kernels, achieving up to 58.36× improvement in application-level performance. Secondly, in the realm of generative neural graphics, we explore Latent Diffusion Models (LDMs) which have surpassed all the other generative processes in high-fidelity image generation domain. The computational demands of LDMs are immense, particularly at high resolutions. We performed in-depth breakdown analysis of the primitive compute kernels of the latent diffusion models and used our kernel-level analysis to design a scalable architecture – Latent Diffusion Model Processing Unit (LDPU). Our estimates show that LDPU increasing the energy efficiency by up to 572× compared to traditional GPU baselines. Lastly, the dissertation combines neural graphics with neural super-resolution techniques to further enhance the output quality and computational efficiency of neural rendering processes. By employing these techniques, we can render at lower resolutions and upscale the outputs to achieve desired higher resolutions without the computational cost typically associated with high-resolution rendering. This comprehensive study not only underscores the necessity of specialized hardware for neural graphics but also sets a precedent for future innovations in this domain, potentially revolutionizing applications in virtual reality (VR), augmented reality (AR), mixed reality (XR), and beyond.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2024

Author dc:creator

• Mubarik, Muhammad Husnain

Contributors dc:contributor

• Kumar, Rakesh
• Chen, Deming
• Gupta, Saurabh
• Iyer, Ravi
• Lin, Yingyan

• Neural Radiance Fields
• Latent Diffusion Models
• Neural Graphics
• Neural Representations
• Diffusion Models
• Virtual Reality
• Augmented Reality
• Mixed Reality

Statement dc:rights

• Copyright 2024 Muhammad Husnain Mubarik

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/125559

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F125559