Empirical Approaches to Challenges in Neural Network Training and Deployment

Training and deploying neural networks is a challenging endeavor. During the training phase, we need to decide on what optimizer to use, the exact data to train on, the training curriculum, and many other factors-- all of which affect the final performance. Once the model is trained, we need to make sure that it is ready to be used and works as intended.This thesis aims to address these challenges in two parts. The first part focuses on gaining insights into what changes in the training pipeline affect the final performance. We start by showing empirically that more popular adaptive optimizers do not perform worse than optimizers they can approximate with a practical amount of hyperparameter tuning, which contrasts with conventional wisdom that adaptive methods generalize worse than non-adaptive methods. We next focus on multi-task learning, and show that using a particular ordering of training data can improve performance on the data-scarce tasks that are prone to overfitting. The second part focuses on the goal of understanding models' capabilities and risks. One capability that we study in detail is large language models' ability to piece together latent knowledge scattered across training documents, and use this knowledge without explicitly reasoning about it. We argue that this behavior needs to be studied and monitored due to potential safety risks of models uncovering and using implicit harmful knowledge without it being noticed by us. Finally, we introduce a framework for verifying the training data of a model, which can provide important information for determining the capabilities and potential risks of the model.

Department dc:contributor.department

Computer Science

Year dc:date.issued

2025

Author dc:creator

• Choi, Dami

Advisors dc:contributor.advisor

• Duvenaud, David K
• Maddison, Chris J

• Machine Learning

Statement dc:rights

• Attribution 4.0 International

Licence dc:rights.uri

• http://creativecommons.org/licenses/by/4.0/

Handle dc:identifier.uri

https://hdl.handle.net/1807/150082

OAI identifier oai:identifier

oai:utoronto.scholaris.ca:1807/150082