{"id":{"repo_id":"mit","oai_identifier":"oai:dspace.mit.edu:1721.1/151236"},"canonical_url":"https://search.dev.ndltd.org/etd/mit/oai:dspace.mit.edu:1721.1/151236","repository":{"repo_id":"mit","name":"MIT","base_url":"https://dspace.mit.edu/oai/request"},"display":{"title":"Foundations of Machine Learning: Over-parameterization and Feature Learning","abstract":"In this thesis, we establish and analyze two core principles driving the success of neural networks: over-parameterization and feature learning. We leverage these principles to design models with improved performance and interpretability on various computer vision and biomedical applications. We begin by discussing the benefits of over-parameterization, i.e., using increasingly large networks that can perfectly fit training data. While prior work characterized the benefits of over-parameterized networks for supervised learning tasks, we show that over-parameterization is also beneficial for unsupervised learning problems, such as autoencoding. The ubiquitous advantage of using increasingly larger networks suggests that infinitely large networks should yield best performance. Remarkably, under certain conditions, training infinitely wide networks simplifies to training classical models known as kernel machines using the Neural Tangent Kernel (NTK). We showcase the practical value of the NTK by deriving and using it for matrix completion problems such as image inpainting and virtual drug screening. Additionally, we use the NTK connection to provide theoretical guarantees for deep neural networks. Namely, we construct interpolating infinitely wide and deep networks that are Bayes optimal, or consistent, for classification. While the NTK has been a useful tool for understanding properties of deep networks, it lacks a key component that is critical to the success of neural networks: feature learning. In the second part of this thesis, we identify and mathematically characterize the mechanism through which deep neural networks automatically select features, or patterns in data. We show that neural feature learning occurs by re-weighting features based on how much they change predictions upon perturbation. Our result explains various deep learning phenomena such as spurious features, lottery tickets, and grokking. Moreover, the mechanism identified in our work provides a backpropagation-free method for feature learning with any machine learning model. To demonstrate the effectiveness of this feature learning mechanism, we use it to enable feature learning in kernel machines. We show that the resulting models, referred to as Recursive Feature Machines, achieve state-of-the-art performance on tabular data. Overall, this thesis advances the foundations of machine learning and provides tools for building new machine learning models that are computationally simple, interpretable, and effective.","abstract_html":"In this thesis, we establish and analyze two core principles driving the success of neural networks: over-parameterization and feature learning. We leverage these principles to design models with improved performance and interpretability on various computer vision and biomedical applications. We begin by discussing the benefits of over-parameterization, i.e., using increasingly large networks that can perfectly fit training data. While prior work characterized the benefits of over-parameterized networks for supervised learning tasks, we show that over-parameterization is also beneficial for unsupervised learning problems, such as autoencoding. The ubiquitous advantage of using increasingly larger networks suggests that infinitely large networks should yield best performance. Remarkably, under certain conditions, training infinitely wide networks simplifies to training classical models known as kernel machines using the Neural Tangent Kernel (NTK). We showcase the practical value of the NTK by deriving and using it for matrix completion problems such as image inpainting and virtual drug screening. Additionally, we use the NTK connection to provide theoretical guarantees for deep neural networks. Namely, we construct interpolating infinitely wide and deep networks that are Bayes optimal, or consistent, for classification. While the NTK has been a useful tool for understanding properties of deep networks, it lacks a key component that is critical to the success of neural networks: feature learning. In the second part of this thesis, we identify and mathematically characterize the mechanism through which deep neural networks automatically select features, or patterns in data. We show that neural feature learning occurs by re-weighting features based on how much they change predictions upon perturbation. Our result explains various deep learning phenomena such as spurious features, lottery tickets, and grokking. Moreover, the mechanism identified in our work provides a backpropagation-free method for feature learning with any machine learning model. To demonstrate the effectiveness of this feature learning mechanism, we use it to enable feature learning in kernel machines. We show that the resulting models, referred to as Recursive Feature Machines, achieve state-of-the-art performance on tabular data. Overall, this thesis advances the foundations of machine learning and provides tools for building new machine learning models that are computationally simple, interpretable, and effective.","abstract_has_math":false,"creators":["Radhakrishnan, Adityanarayanan"],"institution":"Massachusetts Institute of Technology","degree_name":"Doctoral","degree_level":null,"degree_discipline":null,"degree_department":"Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science","school":null,"contributors":[],"advisors":["Uhler, Caroline"],"committee_chairs":[],"committee_members":[],"year":2023,"date_issued":"2023-06","date_published":"2023-06","updated_at":"2026-07-22T22:20:54Z","subjects":[],"languages":[],"rights":["In Copyright - Educational Use Permitted","Copyright retained by author(s)"],"rights_urls":["https://rightsstatements.org/page/InC-EDU/1.0/"],"identifier_entries":[]},"links":{"outbound_url":"https://hdl.handle.net/1721.1/151236","outbound_label":"Handle","outbound_source":"dc:identifier.uri"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor.advisor","label":"Advisor","values":["Uhler, Caroline"]},{"key":"dc:contributor.department","label":"Department","values":["Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science"]},{"key":"dc:creator","label":"Author","values":["Radhakrishnan, Adityanarayanan"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.accessioned","label":"Dc Date Accessioned","values":["2023-07-31T19:24:54Z"]},{"key":"dc:date.available","label":"Dc Date Available","values":["2023-07-31T19:24:54Z"]},{"key":"dc:date.issued","label":"Date","values":["2023-06"]},{"key":"dc:publisher","label":"Institution","values":["Massachusetts Institute of Technology"]},{"key":"dc:type","label":"Dc Type","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Doctoral","Doctor of Philosophy"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:rights","label":"Dc Rights","values":["In Copyright - Educational Use Permitted","Copyright retained by author(s)"]},{"key":"dc:rights.uri","label":"Rights URI","values":["https://rightsstatements.org/page/InC-EDU/1.0/"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier.uri","label":"Identifier URI","values":["https://hdl.handle.net/1721.1/151236"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["In this thesis, we establish and analyze two core principles driving the success of neural networks: over-parameterization and feature learning. We leverage these principles to design models with improved performance and interpretability on various computer vision and biomedical applications. We begin by discussing the benefits of over-parameterization, i.e., using increasingly large networks that can perfectly fit training data. While prior work characterized the benefits of over-parameterized networks for supervised learning tasks, we show that over-parameterization is also beneficial for unsupervised learning problems, such as autoencoding. The ubiquitous advantage of using increasingly larger networks suggests that infinitely large networks should yield best performance. Remarkably, under certain conditions, training infinitely wide networks simplifies to training classical models known as kernel machines using the Neural Tangent Kernel (NTK). We showcase the practical value of the NTK by deriving and using it for matrix completion problems such as image inpainting and virtual drug screening. Additionally, we use the NTK connection to provide theoretical guarantees for deep neural networks. Namely, we construct interpolating infinitely wide and deep networks that are Bayes optimal, or consistent, for classification. While the NTK has been a useful tool for understanding properties of deep networks, it lacks a key component that is critical to the success of neural networks: feature learning. In the second part of this thesis, we identify and mathematically characterize the mechanism through which deep neural networks automatically select features, or patterns in data. We show that neural feature learning occurs by re-weighting features based on how much they change predictions upon perturbation. Our result explains various deep learning phenomena such as spurious features, lottery tickets, and grokking. Moreover, the mechanism identified in our work provides a backpropagation-free method for feature learning with any machine learning model. To demonstrate the effectiveness of this feature learning mechanism, we use it to enable feature learning in kernel machines. We show that the resulting models, referred to as Recursive Feature Machines, achieve state-of-the-art performance on tabular data. Overall, this thesis advances the foundations of machine learning and provides tools for building new machine learning models that are computationally simple, interpretable, and effective."]},{"key":"dc:description.degree","label":"Dc Description Degree","values":["Ph.D."]},{"key":"dc:title","label":"Title","values":["Foundations of Machine Learning: Over-parameterization and Feature Learning"]}]}],"canonical_facts":{"dc:contributor.advisor":["Uhler, Caroline"],"dc:contributor.department":["Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science"],"dc:creator":["Radhakrishnan, Adityanarayanan"],"dc:date.accessioned":["2023-07-31T19:24:54Z"],"dc:date.available":["2023-07-31T19:24:54Z"],"dc:date.issued":["2023-06"],"dc:description.abstract":["In this thesis, we establish and analyze two core principles driving the success of neural networks: over-parameterization and feature learning. We leverage these principles to design models with improved performance and interpretability on various computer vision and biomedical applications. We begin by discussing the benefits of over-parameterization, i.e., using increasingly large networks that can perfectly fit training data. While prior work characterized the benefits of over-parameterized networks for supervised learning tasks, we show that over-parameterization is also beneficial for unsupervised learning problems, such as autoencoding. The ubiquitous advantage of using increasingly larger networks suggests that infinitely large networks should yield best performance. Remarkably, under certain conditions, training infinitely wide networks simplifies to training classical models known as kernel machines using the Neural Tangent Kernel (NTK). We showcase the practical value of the NTK by deriving and using it for matrix completion problems such as image inpainting and virtual drug screening. Additionally, we use the NTK connection to provide theoretical guarantees for deep neural networks. Namely, we construct interpolating infinitely wide and deep networks that are Bayes optimal, or consistent, for classification. While the NTK has been a useful tool for understanding properties of deep networks, it lacks a key component that is critical to the success of neural networks: feature learning. In the second part of this thesis, we identify and mathematically characterize the mechanism through which deep neural networks automatically select features, or patterns in data. We show that neural feature learning occurs by re-weighting features based on how much they change predictions upon perturbation. Our result explains various deep learning phenomena such as spurious features, lottery tickets, and grokking. Moreover, the mechanism identified in our work provides a backpropagation-free method for feature learning with any machine learning model. To demonstrate the effectiveness of this feature learning mechanism, we use it to enable feature learning in kernel machines. We show that the resulting models, referred to as Recursive Feature Machines, achieve state-of-the-art performance on tabular data. Overall, this thesis advances the foundations of machine learning and provides tools for building new machine learning models that are computationally simple, interpretable, and effective."],"dc:description.degree":["Ph.D."],"dc:identifier.uri":["https://hdl.handle.net/1721.1/151236"],"dc:publisher":["Massachusetts Institute of Technology"],"dc:rights":["In Copyright - Educational Use Permitted","Copyright retained by author(s)"],"dc:rights.uri":["https://rightsstatements.org/page/InC-EDU/1.0/"],"dc:title":["Foundations of Machine Learning: Over-parameterization and Feature Learning"],"dc:type":["Thesis"],"thesis:degree_name":["Doctoral","Doctor of Philosophy"]},"updated_at":"2026-07-22T22:20:54Z"}