Rethinking methods to train deep neural networks : contributions of distinct regimes during training

Deep neural networks are known to be highly non-convex. Many of the methods used in deep learning which are informed by convex optimization work surprisingly well. The training dynamics of optimization methods such as momentum suggest that training occurs in distinct regimes, attributed to learning rate. In the low learning rate regime, many convex intuitions hold, and the recommended methods are able to reach a good solution. In the high learning rate regime, the training behavior is not convex-like, but training longer in this period achieves better generalization. This thesis focuses on rethinking deep network training from the perspective of these phases in training. Empirical results suggest that each training regime, although distinct, work together to produce high performance on deep learning tasks. Moreover, we re-examine popular learning rate schedules and find that the paradigm of high and low learning rate regimes helps to explain their advantages.

Name thesis:degree_name

Master

Department dc:contributor.department

Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science

Grantor dc:publisher

Massachusetts Institute of Technology

Year dc:date.issued

2019

Author dc:creator

• Wei, Wendy,M. Eng.Massachusetts Institute of Technology.

Advisor dc:contributor.advisor

• Aleksander Ma̧dry.

• Electrical Engineering and Computer Science.

Statement dc:rights

• MIT theses are protected by copyright. They may be viewed, downloaded, or printed from this source but further reproduction or distribution in any format is prohibited without written permission.

Licence dc:rights.uri

• http://dspace.mit.edu/handle/1721.1/7582

Language dc:language.iso

eng

Handle dc:identifier.uri

https://hdl.handle.net/1721.1/123028

OAI identifier oai:identifier

oai:dspace.mit.edu:1721.1/123028