Structural Priors in Deep Neural Networks

Deep learning has in recent years come to dominate the previously separate fields of research in machine learning, computer vision, natural language understanding and speech recognition. Despite breakthroughs in training deep networks, there remains a lack of understanding of both the optimization and structure of deep networks. The approach advocated by many researchers in the field has been to train monolithic networks with excess complexity, and strong regularization --- an approach that leaves much to desire in efficiency. Instead we propose that carefully designing networks in consideration of our prior knowledge of the task and learned representation can improve the memory and compute efficiency of state-of-the art networks, and even improve generalization --- what we propose to denote as structural priors. We present two such novel structural priors for convolutional neural networks, and evaluate them in state-of-the-art image classification CNN architectures. The first of these methods proposes to exploit our knowledge of the low-rank nature of most filters learned for natural images by structuring a deep network to learn a collection of mostly small, low-rank, filters. The second addresses the filter/channel extents of convolutional filters, by learning filters with limited channel extents. The size of these channel-wise basis filters increases with the depth of the model, giving a novel sparse connection structure that resembles a tree root. Both methods are found to improve the generalization of these architectures while also decreasing the size and increasing the efficiency of their training and test-time computation. Finally, we present work towards conditional computation in deep neural networks, moving towards a method of automatically learning structural priors in deep networks. We propose a new discriminative learning model, conditional networks, that jointly exploit the accurate representation learning capabilities of deep neural networks with the efficient conditional computation of decision trees. Conditional networks yield smaller models, and offer test-time flexibility in the trade-off of computation vs. accuracy.

Name dc:type.qualificationname

Doctor of Philosophy (PhD)

Level dc:type.qualificationlevel

Doctoral

Grantor dc:publisher.institution

University of Cambridge

Year dc:date.issued

2018

Author dc:creator

• Ioannou, Yani Andrew

Advisors dc:contributor.advisor

• Cipolla, Roberto
• Criminisi, Antonio

• Deep Learning
• Neural Networks
• Machine Learning
• Computer Vision
• Structural Priors
• Filter Groups
• Low Rank
• Convolution
• Deep Neural Networks
• Convolutional Neural Networks
• CNN
• Efficient

Licence

• https://apollo8-f-pro.lib.cam.ac.uk/bitstreams/82cc1852-7ac6-4518-8c15-b01e8fc7d527/download
• https://www.rioxx.net/licenses/all-rights-reserved/

Language dc:language

en

Author Identifier

0000-0002-9797-5888
0000-0002-8999-2151
0000-0001-7976-3374

OAI identifier oai:identifier

oai:www.repository.cam.ac.uk:1810/278976