Hyperparameter tuning and its effects on deep learning performance and generalization

Hyperparameter tuning is an integral part of deep learning research. Finding hyperparameter values that effectively leverage the strengths of network architectures and training procedures is crucial to maximizing performance. However, extensive hyperparameter searches raise concerns about overfitting to re-used evaluation datasets. In this thesis, we perform a case study of hyperparameter search methods on SqueezeNet v1.0 with refinements added to the training procedure. We show that random search allows for improvement over baseline performance in few trials, achieve around a 2% increase in SqueezeNet accuracy on ImageNet, and provide evidence that contrary to the common notion of adaptive overfitting, accuracy gains achieved on a validation set through hyperparameter tuning result in larger gains on a held-out test set.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2020

Author dc:creator

• Rabe, Benjamin

Contributors dc:contributor

• Kindratenko, Volodymyr

• machine learning
• hyperparameter search

Statement dc:rights

• Copyright 2020 Benjamin Rabe

Language dc:language

en

Handle dc:identifier

http://hdl.handle.net/2142/107979

OAI identifier oai:identifier

oai:www.ideals.illinois.edu:2142/107979