Advances in Optimisation of Model Parameters and Hyperparameters for Neural Networks

Machine Learning has exploded in popularity in recent years, and now sees use in a huge variety of applications, from advertisement targeting and image generation to the recent proliferation of Large Language Models. At the heart of each setting is the problem of optimising model parameters to minimise some loss metric, so the chosen optimisation algorithm plays a fundamental role in the training process — both through the optimisation logic itself, and the auxiliary *hyperparameters* which configure the optimiser’s behaviour. Moreover, Machine Learning tasks often demand unique properties from optimisers, distinct from those explored in classical optimisation research. In this Thesis, we explore three novel techniques for optimising these parameters and hyperparameters, each sharing the theme of an awareness of the underlying curvature of the optimisation space. We begin with a study of Hyperparameter Optimisation. Many existing algorithms require complete training runs to evaluate each proposed hyperparameter configuration, which carry considerable computational cost. Methods based on hypergradients use only one training pass, but these either cannot be applied to arbitrary optimiser hyperparameters (such as learning rates and momenta) or suffer considerable additional training time. In an extension to these methods, we develop an approximate hypergradient-based hyperparameter optimiser, which is applicable to any continuous hyperparameter appearing in a differentiable model weight update. Our algorithm requires only one training episode (with no restarts), has a motivating argument for convergence to the true hypergradient, and scales to optimising independent learning rates for each model parameter, which we demonstrate in a wide-ranging empirical study. This contribution advances our ability to effectively configure a parameter optimiser for Neural Network training. Having developed a technique for selecting the hyperparameters controlling the behaviour of an optimisation algorithm, we turn to the model parameter optimisation task itself to explore which algorithmic properties might be useful in this application. The intractably-large Hessian matrices of many Machine Learning models make it challenging to apply the second-order quasi-Newton methods which are popular in the broader continuous optimisation literature. Attempts to address non-convexity in the problem, for instance by modifying eigenvalues as in Saddle-Free Newton methods, only exacerbate the issue of intractability. Addressing both these concerns, we propose the first (to our knowledge) efficiently-scalable optimisation algorithm to asymptotically use the exact, eigenvalue-modified inverse Hessian. Our method uses a power series to principally square-root and invert the squared Hessian, then precondition a gradient vector, all without explicitly computing or storing the Hessian. A truncation of this infinite series yields an optimisation algorithm which is competitive with common first- and second-order approaches, a claim which our experiments verify. However, while performing this work, we noted unexpected differences between the computational efficiency of first-order, gradient-based methods and the theoretical efficiency of second-order, curvature-based methods. In particular, we were intrigued by the relative strength and popularity of the Adam algorithm and the performant but fragile nature of the K-FAC algorithm. Inspired by this behaviour, we conclude this Thesis by seeking to unify the benefits of both approaches, combining the stabilising heuristics of second-order methods (such as Levenberg-Marquardt damping) with the efficient update direction selection of first-order methods. Our resulting algorithm recasts the Adam optimiser from a second-order optimisation perspective by adopting features from the K-FAC optimiser, and raises interesting questions about the dynamics of both popular methods; we explore these in an empirical validation over a range of settings. To conclude, we summarise our contributions and explore some possible future research directions they raise. We briefly consider the imminent challenges faced by the Machine Learning research community, and make a case for an ethical, human-centred emphasis on potential developments.

Name dc:type.qualificationname

Doctor of Philosophy (PhD)

Level dc:type.qualificationlevel

Doctoral

Grantor dc:publisher.institution

University of Cambridge

Year dc:date.issued

2024

Author dc:creator

• Clarke, Ross MacKinnon

Advisor dc:contributor.advisor

• Hernández-Lobato, José Miguel

• Adam
• Artificial Intelligence
• Hyperparameter Optimisation
• K-FAC
• Machine Learning
• Neural Networks
• Optimisation

Licence

• https://apollo8-f-pro.lib.cam.ac.uk/bitstreams/544c547a-8595-4819-8cd6-f305e8ff5509/download
• https://www.rioxx.net/licenses/all-rights-reserved/

Language dc:language

eng

DOI dc:identifier.doi

https://doi.org/10.17863/CAM.111541

OAI identifier oai:identifier

oai:www.repository.cam.ac.uk:1810/372888