Improving algorithmic performance using stochastic learning rates: In-expectation and almost-surely stochastic approximation, and online learning, results and applications

In this work, multiplicative stochasticity is applied to the learning rate of stochastic optimization algorithms, giving rise to stochastic learning-rate schemes. In-expectation theoretical convergence results of Stochastic Gradient Descent (SGD) equipped with this novel stochastic learning rate scheme under the stochastic setting, as well as convergence results under the online optimization settings are provided. Empirical results consider the case of an adaptively uniformly distributed multiplicative stochasticity and include not only Stochastic Gradient Descent, but also other popular algorithms equipped with a stochastic learning rate. They demonstrate noticeable optimization performance gains, with respect to their deterministic-learning-rate versions. Under this stochastic learning rate framework, the theoretical almost sure (a.s.) convergence rates of the Stochastic Heavy Ball (SHB) algorithm in the convex and smooth, and the SGD algorithms in the nonconvex and smooth settings are investigated. In specific, it is shown that the a.s. convergence rates for both of these algorithms are accelerated when a stochastic-learning rate scheme satisfying certain criteria is used as opposed to a traditional deterministic-learning rate scheme. The reason for this acceleration is the multiplicative stochasticity which, mainly through its first moment but also its variance, beneficially affects the a.s. convergence rates.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2024

Author dc:creator

• Mamalis, Theodoros

Contributors dc:contributor

• Voulgaris, Petros
• Stipanovic, Dusan
• Liberzon, Daniel
• Subhonmesh, Bose

• Machine Learning
• Artificial Intelligence
• Optimization Algorithms
• Minimization
• Stochastic Systems
• Training Data
• Testing Data
• Upper Bound
• Empirical Loss Measurement
• Learning Rate
• Loss Function
• Convergence

Statement dc:rights

• Copyright 2024 Theodoros Mamalis

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/124240

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F124240