Beyond Gradients: Using Curvature Information for Deep Learning

This thesis investigates optimization and interpretability techniques for deep learning, extending beyond gradient-based methods to incorporate curvature information. We begin by addressing the limitations of first-order optimization methods like gradient descent, which can exhibit slow convergence in directions of low curvature within the loss landscape. To address these challenges, we introduce a novel framework that dynamically learns and integrates curvature information throughout optimization, significantly accelerating convergence. Our investigation extends to bilevel problems, focusing on hyperparameter optimization and influence functions. Bilevel optimization involves nested optimization problems where the outer-level objective depends on the optimal solution to an inner-level optimization. In gradient-based bilevel optimization, a key challenge is accurately capturing how outer-level variable updates affect the inner-level optimization problem, where inner-level curvature information provides essential sensitivity measures. For hyperparameter optimization, we improve existing hypernetwork-based approaches for learning curvature information by addressing subtle training pathologies. Examining the bilevel formulation of influence functions reveals that their application to neural networks approximates a different quantity than previously believed. We identify key limitations, including their inability to account for optimizers' implicit bias or multi-stage training processes. We present an unrolled differentiation formulation to address these shortcomings, providing a more comprehensive framework for analyzing a training data point's impact on model predictions. The thesis concludes by discussing limitations and future research directions for the proposed techniques.

Department dc:contributor.department

Computer Science

Year dc:date.issued

2025

Author dc:creator

• Bae, Juhan

Advisor dc:contributor.advisor

• Grosse, Roger

Handle dc:identifier.uri

https://hdl.handle.net/1807/142699

OAI identifier oai:identifier

oai:utoronto.scholaris.ca:1807/142699