Inference-driven perceptual optimization

Perception is fundamental to modern computer vision. While most research has focused on improving datasets, model architectures, or training objectives, this thesis explores a complementary yet under-explored perspective: optimizing perception models through inference guidance. The core insight is that models often exhibit informative behaviors during inference, such as deviations from training assumptions or signals that reveal opportunities for further optimization. Although typically overlooked, these behaviors can be leveraged to improve model effectiveness and efficiency. To demonstrate the idea, this thesis presents two case studies, each targeting one of these dimensions. The first part investigates Inference Guidance for Diffusion Models, aiming to improve model effectiveness by aligning training objectives with inference behavior. Diffusion models, increasingly adopted for perception tasks, generate samples through an iterative denoising process. However, this process often misaligns with the objectives of discriminative tasks. Inference-time analysis reveals two key issues: (1) denoising timesteps contribute unevenly to perception quality, and (2) a distribution shift between training and inference leads to performance degradation. To address these challenges, we propose ADDP (Aligning Diffusion Denoising with Perception), a framework that reweights the training objective based on timestep importance and introduces data augmentations to simulate inference-time denoising errors. Experiments on depth estimation and referring image segmentation tasks demonstrate that ADDP improves both perceptual alignment and overall performance. The second part focuses on Inference Acceleration for Visual Encoders, aiming to enhance model efficiency by leveraging inference properties. Analysis of pre-trained vision transformers (ViTs) shows that sparsity naturally emerges in the attention patterns of deeper layers, enabling selective computation without sacrificing performance. Building on this observation, we introduce SVE (Sparse Vision Encoder), a framework that identifies target layers, restores performance through distillation, and reduces inference latency via sparsity prediction. Experiments across multiple vision encoders show that SVE achieves up to a 23% speedup while maintaining accuracy on classification and segmentation benchmarks. Together, these studies advocate for inference-aware optimization as a promising paradigm for advancing perception models. They demonstrate that careful analysis of inference-time behavior can uncover new opportunities for performance gains beyond conventional training-time interventions.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois Urbana-Champaign

Year dc:date

2025

Author dc:creator

• Xu, Xin

Contributors dc:contributor

• Wang, Yuxiong

• Perception
• Inference
• Diffusion Models
• Sparsity

Statement dc:rights

• Copyright 2025 Xin Xu

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/129555

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F129555