Optimal Approximation of High-dimensional Functions on Smooth Manifolds Using Deep ReLU Neural Networks

The expressive power of deep neural networks is manifested by their remarkable ability to approximate multivariate functions in a way that appears to overcome the curse of dimensionality. This ability is exemplified by their success in solving high-dimensional problems where traditional numerical solvers fail due to their limitations in accurately representing high-dimensional structures. In this dissertation, we focus on the approximation theory of neural networks to explain this phenomenon. First, we construct a version of the Johnson-Lindenstrass Lemma on the smooth manifold to analyze the approximation of β-smooth H\"older functions defined on a $d$-dimensional smooth manifold $\mathcal{M}$ embedded in \mathbb{R}^D, with $d \ll D$, using deep neural networks. We prove that the uniform convergence estimates of the approximation and generalization errors by deep neural networks with ReLU activation functions do not depend on the ambient dimension $D$ of the function but only on its lower manifold dimension $d$, in a precise sense. Our result improves existing results from the literature where approximation and generalization errors were shown to depend weakly on $D$. Since the aforementioned method is not constructive and does not provide explicit knowledge on the structure of the deep neural networks involved, we next introduce a constructive approach to achieve a similar approximation error. For this analysis, we consider the approximation of $s$ times continuously differentiable functions defined on a $d$-dimensional smooth manifold $\mathcal{M}$ embedded in \mathbb{R}^D, with $d \ll D$. We prove that a deep ReLU neural network with fixed width depending polynomially on $D$ and depth L₀ can approximate an $s$ times continuously differentiable function with approximation rate C(s,d) L₀^-2s/d which is completely independent of the ambient dimension $D$. This result improves previous approximation estimates where the constant of the approximation rate depends on $D$. This dissertation also derives the theoretical optimal approximation rate of deep ReLU neural networks for β-smooth Hölder functions with β \in (0,1] and for $s$ times continuously differentiable functions with $s\in\mathbb N$ defined on [0,1]^D.

Name thesis:degree_name

Doctor of Philosophy

Discipline thesis:degree_discipline

Mathematics

Grantor

University of Houston

Year dc:date.issued

2024

Author dc:creator

• Shi, Ji

Advisor dc:contributor.advisor

• Labate, Demetrio

Committee members dc:contributor.committeemember

• Kakadiaris, Ioannis
• Azencott, Robert
• Mang, Andreas

• Mathematics

Language dc:language.iso

en

Handle dc:identifier.uri

https://hdl.handle.net/10657/19915

OAI identifier oai:identifier

oai:uh-ir.tdl.org:10657/19915