Deep feature manipulation for image synthesis and restoration

Deep learning has revolutionized image generation by capturing the underlying statistical distribution of training data, allowing for the creation of new, realistic images through latent representation manipulation. Deep learning has also significantly advanced image restoration techniques by improving the recovery of degraded images via removing noise, improving contrast, and enhancing details more effectively than traditional techniques, leading to state-of-the-art performance. Nevertheless, deep learning's success in image synthesis and restoration is often hampered by its neglect of data properties, model architecture, and task specifics. For instance, ignoring spatial invariances in tasks like image classification and segmentation can lead to models vulnerable to minor spatial transformations. Latent representations that are not interpretable can limit the model's ability to control salient image attributes, while the lack of feature invertibility can hinder high-quality image generation. To address these issues, this work introduces novel techniques that enhance performance by considering spatial invariances, capturing meaningful features, and improving feature invertibility. We propose (i) a novel down and upsampling method in CNN models to impose perfect shift equivariance towards improved performance on image classification and segmentation, (ii) a data-adaptive framework to build provably shift-equivariant ViT models, improving performance on discriminative tasks across hierarchical architectures, (iii) a neural style transfer-based data-augmentation technique to extract strong image representations through self-supervised learning, (iv) a computationally efficient photorealistic style transfer algorithm that matches multiscale geometric representations, eliminating the need for learned features to produce high-quality results by preserving fine-grained visual details, and (v) the use of adversarially robust features as a perceptual prior to invert contracted representations back into high-quality images, outperforming standard models and unlocking their use in various image synthesis and restoration tasks.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2024

Author dc:creator

• Rojas Gomez, Renan Alfredo

Contributors dc:contributor

• Do, Minh N
• Boppart, Stephen A
• Gupta, Saurabh
• Nguyen, Anh M
• Zhao, Zhizhen

• Image Synthesis
• Image Restoration
• Computer Vision
• Deep Learning
• Signal Processing
• Convolutional Neural Networks
• Vision Transformers
• Multiscale Image Representation

Statement dc:rights

• Copyright 2024 Renan Rojas Gomez

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/127378

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F127378