Advances in Optimisation of Model Parameters and Hyperparameters for Neural Networks
type: CreativeWork
description: Machine Learning has exploded in popularity in recent years, and now sees use in a huge variety of applications, from advertisement targeting and image generation to the recent proliferation of Large Language Models. At the heart of each setting is the problem of optimising model parameters to minimise some loss metric, so the chosen optimisation algorithm plays a fundamental role in the training process — both through the optimisation logic itself, and the auxiliary *hyperparameters* which configure the optimiser’s behaviour. Moreover, Machine Learning tasks often demand unique properties from optimisers, distinct from those explored in classical optimisation research. In this Thesis, we explore three novel techniques for optimising these parameters and hyperparameters, each sharing the theme of an awareness of the underlying curvature of the optimisation space. We begin with a study of Hyperparameter Optimisation. Many existing algorithms require complete training runs to evaluate each proposed hyperparameter configuration, which carry considerable computational cost. Methods based on hypergradients use only one training pass, but these either cannot be applied to arbitrary optimiser hyperparameters (such as learning rates and momenta) or suffer considerable additional training time. In an extension to these methods, we develop an approximate hypergradient-based hyperparameter optimiser, which is applicable to any continuous hyperparameter appearing in a differentiable model weight update. Our algorithm requires only one training episode (with no restarts), has a motivating argument for convergence to the true hypergradient, and scales to optimising independent learning rates for each model parameter, which we demonstrate in a wide-ranging empirical study. This contribution advances our ability to effectively configure a parameter optimiser for Neural Network training. Having developed a technique for selecting the hyperparameters controlling the behaviour of an o…
abstract: Machine Learning has exploded in popularity in recent years, and now sees use in a huge variety of applications, from advertisement targeting and image generation to the recent proliferation of Large Language Models. At the heart of each setting is the problem of optimising model parameters to minimise some loss metric, so the chosen optimisation algorithm plays a fundamental role in the training process — both through the optimisation logic itself, and the auxiliary *hyperparameters* which configure the optimiser’s behaviour. Moreover, Machine Learning tasks often demand unique properties from optimisers, distinct from those explored in classical optimisation research. In this Thesis, we explore three novel techniques for optimising these parameters and hyperparameters, each sharing the theme of an awareness of the underlying curvature of the optimisation space. We begin with a study of Hyperparameter Optimisation. Many existing algorithms require complete training runs to evaluate each proposed hyperparameter configuration, which carry considerable computational cost. Methods based on hypergradients use only one training pass, but these either cannot be applied to arbitrary optimiser hyperparameters (such as learning rates and momenta) or suffer considerable additional training time. In an extension to these methods, we develop an approximate hypergradient-based hyperparameter optimiser, which is applicable to any continuous hyperparameter appearing in a differentiable model weight update. Our algorithm requires only one training episode (with no restarts), has a motivating argument for convergence to the true hypergradient, and scales to optimising independent learning rates for each model parameter, which we demonstrate in a wide-ranging empirical study. This contribution advances our ability to effectively configure a parameter optimiser for Neural Network training. Having developed a technique for selecting the hyperparameters controlling the behaviour of an o…
author: Clarke, Ross MacKinnon
language: eng
keywords: Adam, Artificial Intelligence, Hyperparameter Optimisation, K-FAC, Machine Learning, Neural Networks, Optimisation
license: https://apollo8-f-pro.lib.cam.ac.uk/bitstreams/544c547a-8595-4819-8cd6-f305e8ff5509/download
license: https://www.rioxx.net/licenses/all-rights-reserved/
date published: 2024-01-03
date modified: 2026-07-22T22:24:14Z
provider: Cambridge University
publisher: Networked Digital Library of Theses and Dissertations
type: WebPage
source organization: University of Cambridge

PropertyValue
property ID: DOI
value: 10.17863/CAM.111541