Beneficial Initializations in Over-Parameterized Machine Learning Problems

We theoretically and empirically analyze the phenomenon of transfer learning in overparameterized machine learning. We start by showing that in over-parameterized linear regression, transfer learning is equivalent to solving regression from a non-zero initialization. We use this finding to propose LLBoost, a theoretically grounded, computationally efficient method to boost the validation and test accuracy of pretrained, over-parameterized models without impacting the original training accuracy. We evaluate LLBoost on CIFAR10, ImageNet-32, and ImageNet and also prove that it reduces the generalization error of any interpolating solution with high probability. By extending our analysis of transfer learning in linear regression, we present an approach for transfer learning in kernel regression. Namely, we demonstrate that transfer learning corresponds to adding a function to the minimum norm solution that produces zero error on the training data. We use this approach to perform transfer learning on image classification tasks using the neural tangent kernel.

Name thesis:degree_name

Master

Department dc:contributor.department

Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science

Grantor dc:publisher

Massachusetts Institute of Technology

Year dc:date.issued

2021

Author dc:creator

• Prasad, Neha

Advisor dc:contributor.advisor

• Uhler, Caroline

Statement dc:rights

• In Copyright - Educational Use Permitted
• Copyright MIT

Licence dc:rights.uri

• http://rightsstatements.org/page/InC-EDU/1.0/

Handle dc:identifier.uri

https://hdl.handle.net/1721.1/139050

OAI identifier oai:identifier

oai:dspace.mit.edu:1721.1/139050