An adaptive pruning algorithm for DNN compression

In recent years, deep neural networks have achieved remarkable results in various artificial intelligence tasks such as image recognition and machine language translation. Although deep neural networks achieve state-of-the-art accuracy for various tasks, the high accuracy comes with high computational complexity and large network size which leads to over-parameterization. The computational complexity and large size of deep neural networks limit their applications for low-power and mobile platforms. Techniques such as pruning, quantization and binarization have been developed to reduce network parameter size. Pruning is the one of well-studied methods to solve network over-parameterization and reduce computational complexity. A typical pruning process includes three stages: train the network, remove redundant weights according to certain criteria, and fine-tune the reduced network. However, the three-stage method is time-consuming and frequently cannot fully compensate for the pruned neurons which are actually important. In this thesis, we propose a new algorithm which merges the removing stage and fine-tuning stage into the training stage. The new algorithm reduces time complexity of the pruning process and makes incorrectly pruned neurons recoverable without significant loss of accuracy. We also explore the effects of different pruning criteria calculation methods, weight updating methods and pruning rate changing methods on pruning algorithm performance.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2020

Author dc:creator

• Wu, Jiaying

Contributors dc:contributor

• Kindratenko, Volodymyr

• Pruning
• DNN

Statement dc:rights

• Copyright 2020 Jiaying Wu

Language dc:language

en

Handle dc:identifier

http://hdl.handle.net/2142/108183

OAI identifier oai:identifier

oai:www.ideals.illinois.edu:2142/108183