{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/124240"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/124240","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Improving algorithmic performance using stochastic learning rates: In-expectation and almost-surely stochastic approximation, and online learning, results and applications","abstract":"Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2024-09-16 without embargo terms","abstract_html":"Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2024-09-16 without embargo terms","abstract_has_math":false,"creators":["Mamalis, Theodoros"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"Ph.D.","degree_level":"Dissertation","degree_discipline":"Electrical & Computer Engr","degree_department":null,"school":null,"contributors":["Voulgaris, Petros","Stipanovic, Dusan","Liberzon, Daniel","Subhonmesh, Bose"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2024,"date_issued":"2024-05","date_published":"2024-05","updated_at":"2026-07-22T22:25:00Z","subjects":["Machine Learning","Artificial Intelligence","Optimization Algorithms","Minimization","Stochastic Systems","Training Data","Testing Data","Upper Bound","Empirical Loss Measurement","Learning Rate","Loss Function","Convergence"],"languages":["en","eng"],"rights":["Copyright 2024 Theodoros Mamalis"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://hdl.handle.net/2142/124240","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Voulgaris, Petros","Stipanovic, Dusan","Liberzon, Daniel","Subhonmesh, Bose"]},{"key":"dc:creator","label":"Author","values":["Mamalis, Theodoros"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2024-05","2024-04-10"]},{"key":"dc:type","label":"Dc Type","values":["text"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Electrical & Computer Engr"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Dissertation"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Ph.D."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Machine Learning","Artificial Intelligence","Optimization Algorithms","Minimization","Stochastic Systems","Training Data","Testing Data","Upper Bound","Empirical Loss Measurement","Learning Rate","Loss Function","Convergence"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en","eng"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2024 Theodoros Mamalis"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://hdl.handle.net/2142/124240"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2024-09-16 without embargo terms","The student, Theodoros Mamalis, accepted the attached license on 2024-04-08 at 17:02.","The student, Theodoros Mamalis, submitted this Dissertation for approval on 2024-04-08 at 17:10.","This Dissertation was approved for publication on 2024-04-10 at 15:33.","DSpace SAF Submission Ingestion Package generated from Vireo submission #20333 on 2024-09-16 at 00:33:49","In this work, multiplicative stochasticity is applied to the learning rate of stochastic optimization algorithms, giving rise to stochastic learning-rate schemes. In-expectation theoretical convergence results of Stochastic Gradient Descent (SGD) equipped with this novel stochastic learning rate scheme under the stochastic setting, as well as convergence results under the online optimization settings are provided. Empirical results consider the case of an adaptively uniformly distributed multiplicative stochasticity and include not only Stochastic Gradient Descent, but also other popular algorithms equipped with a stochastic learning rate. They demonstrate noticeable optimization performance gains, with respect to their deterministic-learning-rate versions. Under this stochastic learning rate framework, the theoretical almost sure (a.s.) convergence rates of the Stochastic Heavy Ball (SHB) algorithm in the convex and smooth, and the SGD algorithms in the nonconvex and smooth settings are investigated. In specific, it is shown that the a.s. convergence rates for both of these algorithms are accelerated when a stochastic-learning rate scheme satisfying certain criteria is used as opposed to a traditional deterministic-learning rate scheme. The reason for this acceleration is the multiplicative stochasticity which, mainly through its first moment but also its variance, beneficially affects the a.s. convergence rates."]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Improving algorithmic performance using stochastic learning rates: In-expectation and almost-surely stochastic approximation, and online learning, results and applications"]}]}],"canonical_facts":{"dc:contributor":["Voulgaris, Petros","Stipanovic, Dusan","Liberzon, Daniel","Subhonmesh, Bose"],"dc:creator":["Mamalis, Theodoros"],"dc:date":["2024-05","2024-04-10"],"dc:description":["Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2024-09-16 without embargo terms","The student, Theodoros Mamalis, accepted the attached license on 2024-04-08 at 17:02.","The student, Theodoros Mamalis, submitted this Dissertation for approval on 2024-04-08 at 17:10.","This Dissertation was approved for publication on 2024-04-10 at 15:33.","DSpace SAF Submission Ingestion Package generated from Vireo submission #20333 on 2024-09-16 at 00:33:49","In this work, multiplicative stochasticity is applied to the learning rate of stochastic optimization algorithms, giving rise to stochastic learning-rate schemes. In-expectation theoretical convergence results of Stochastic Gradient Descent (SGD) equipped with this novel stochastic learning rate scheme under the stochastic setting, as well as convergence results under the online optimization settings are provided. Empirical results consider the case of an adaptively uniformly distributed multiplicative stochasticity and include not only Stochastic Gradient Descent, but also other popular algorithms equipped with a stochastic learning rate. They demonstrate noticeable optimization performance gains, with respect to their deterministic-learning-rate versions. Under this stochastic learning rate framework, the theoretical almost sure (a.s.) convergence rates of the Stochastic Heavy Ball (SHB) algorithm in the convex and smooth, and the SGD algorithms in the nonconvex and smooth settings are investigated. In specific, it is shown that the a.s. convergence rates for both of these algorithms are accelerated when a stochastic-learning rate scheme satisfying certain criteria is used as opposed to a traditional deterministic-learning rate scheme. The reason for this acceleration is the multiplicative stochasticity which, mainly through its first moment but also its variance, beneficially affects the a.s. convergence rates."],"dc:format":["application/pdf"],"dc:identifier":["https://hdl.handle.net/2142/124240"],"dc:language":["en","eng"],"dc:rights":["Copyright 2024 Theodoros Mamalis"],"dc:subject":["Machine Learning","Artificial Intelligence","Optimization Algorithms","Minimization","Stochastic Systems","Training Data","Testing Data","Upper Bound","Empirical Loss Measurement","Learning Rate","Loss Function","Convergence"],"dc:title":["Improving algorithmic performance using stochastic learning rates: In-expectation and almost-surely stochastic approximation, and online learning, results and applications"],"dc:type":["text"],"thesis:degree_discipline":["Electrical & Computer Engr"],"thesis:degree_level":["Dissertation"],"thesis:degree_name":["Ph.D."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:25:00Z"}