Nonlinear and geometric control methods in deep learning theory

The practical success and popularity of deep neural networks for solving modern machine learning problems motivates us to develop a rigorous theoretical understanding of how depth affects model expressivity and generalizability. Interpreting deep learning architectures as control systems unlocks versatile tools from nonlinear systems theory to study these models and associated statistical learning problems. In this dissertation, we present three main technical works taking advantage of this perspective, which are summarized as follows: The first work describes an encoder-decoder architecture for learning immersed submanifolds from data inspired by the structure of the group action in Sussmann’s orbit theorem, which is built from composing forward- and backward-in-time flow maps. We proceed to develop generalization bounds for this model class and apply these results to a handful of illustrative examples. The second work investigates a technique for proving generalization bounds for neural ordinary differential equations based on transforming the model into an equivalent infinite-dimensional kernel machine through the use of the Chen–Fliess expansion, which expresses the model output as an infinite series in terms of signature integrals of the control and iterated Lie derivatives of the output map. This technique differs from strategies based on bounding the covering number of the model class by propagating a parameter perturbation through the flow map, and instead takes advantage of standard tools applicable to kernel machines. The third work focuses on deriving quantitative approximation error bounds for neural ordinary differential equations having at most quadratic nonlinearities in the dynamics. The simple dynamics of this model form demonstrates how expressivity can be derived primarily from iteratively composing many basic elementary operations, versus from the complexity of those elementary operations themselves. These results contribute to our understanding of what depth imparts to the capabilities of deep learning architectures.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2024

Author dc:creator

• Hanson, Joshua McKinley

Contributors dc:contributor

• Raginsky, Maxim
• Baryshnikov, Yuliy
• Belabbas, Mohamed Ali
• Liberzon, Daniel

• Neural Networks
• Statistical Learning Theory
• Rademacher Complexity
• Deep Learning
• Nonlinear Control
• Geometric Control

Statement dc:rights

• Copyright 2024 Joshua Hanson

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/127381

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F127381