Time Aware Sigmoid Optimization : a new learning rate scheduling method

The correct choice of hyperparameters for the training of a deep neural network is a critical step to achieve a good result. Good hyperparameters would give rise to faster training and a lower error rate, while bad choices could make the network not even converge, rendering the whole training process useless. Among all the existing hyperparameters, perhaps the one with the greatest importance is the learning rate, which controls how the weights of a neural network are going to change at each interaction. In that context, by analyzing some theoretical findings in the area of information theory and topology of the loss function in deep learning, the author was able to come up with a new training rate decay method called Training Aware Sigmoid Optimization (TASO), which proposes a dual-phase during training. The proposed method aims to improve training, achieving a better inference performance in a reduced amount of time. A series of tests were done to evaluate this hypothesis, comparing TASO with different training methods such as Adam, ADAGrad, RMSProp, and SGD. Results obtained on three datasets (MNIST, CIFAR10, and CIFAR100) and with three different architectures (Lenet, VGG, and RESNET) have shown that TASO presents, in fact, an overall better performance than the other evaluated methods.

Grantor dc:publisher

Universidade Federal de Pernambuco

Year dc:date.issued

2019

Author dc:creator

• LEUCHTENBERG, Pedro Henrique Dreyer

Advisor dc:contributor.advisor

• ZANCHETTIN, Cleber

• Inteligência computacional
• Aprendizagem de máquinas
• Redes neurais profundas
• Taxa de aprendizado

Statement dc:rights

• openAccess
• Attribution-NonCommercial-NoDerivs 3.0 Brazil

Licence dc:rights.uri

• http://creativecommons.org/licenses/by-nc-nd/3.0/br/

Language dc:language.iso

eng

Repository record dc:identifier.uri

https://repositorio.ufpe.br/handle/123456789/38541

OAI identifier oai:identifier

oai:repositorio.ufpe.br:123456789/38541