Trainability and generalization of small-scale neural networks

As deep learning has become solution for various machine learning, artificial intelligence applications, their architectures have been developed accordingly. Modern deep learning applications often use overparameterized setting, which is opposite to what conventional learning theory suggests. While deep neural networks are considered to be less vulnerable to overfitting even with their overparameterized architecture, this project observed that properly trained small-scale networks indeed outperform its larger counterparts. The generalization ability of small-scale networks has been overlooked in many researches and practice, due to their extremely slow convergence speed. This project observed that imbalanced layer-wise gradient norm can hider overall convergence speed of neural networks, and narrow networks are vulnerable to this. This projects investigates possible reasons of convergence failure of small-scale neural networks, and suggests a strategy to alleviate the problem.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Industrial Engineering

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2019

Author dc:creator

• Song, Myung Hwan

Contributors dc:contributor

• Sun, Ruoyu

• Deep Learning
• Neural Networks
• Learning Theory

Statement dc:rights

• Copyright 2019 Myung Hwan Song

Language dc:language

en

Handle dc:identifier

http://hdl.handle.net/2142/104821

OAI identifier oai:identifier

oai:www.ideals.illinois.edu:2142/104821