Studies of lighting for dense prediction and generation in computer vision

Modern dense prediction models do exceedingly well on benchmarks for the standard computer vision tasks of depth and normal estimation, object detection, and semantic segmentation. However, we show that the lighting in a scene has a significant effect on predictions, often producing inconsistent results for relit versions of the exact same scene. For surface normal prediction, we demonstrate that fine-tuning to enforce consistency under various lightings can mitigate this problem without sacrificing base accuracy of the pretrained model. Yet, such fine-tuning requires a dataset of relit scenes, which exist in limited quantity and are burdensome to produce. We therefore explore existing generative methods to create a synthetic relighting dataset, and propose our new method StyLitGAN. Based on StyleGAN architecture and the use of latent stylecode directions, StyLitGAN can realistically relight complex scenes without the need for labeled data. Fine-tuning a dense predictor with StyLitGAN images results in improvements comparable with that obtained by fine-tuning with true multi-illuminant images. We continue to investigate the effect of stylecode directions across different scenes, and show their use in producing desired results from reference images. We explore stylecode applications in explicitly-controllable scene lighting and uncover hints at their internal representation.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2024

Author dc:creator

• Soole, James

Contributors dc:contributor

• Forsyth, David A

• Computer Vision
• Artificial Intelligence
• Deep Learning
• Generative Ai
• Intrinsics
• Dense Prediction

Statement dc:rights

• Copyright 2024 James Soole

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/124287

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F124287