Algorithms above the noise floor

Many success stories in the data sciences share an intriguing computational phenomenon. While the core algorithmic problems might seem intractable at first, simple heuristics or approximation algorithms often perform surprisingly well in practice. Common examples include optimizing non-convex functions or optimizing over non-convex sets. In theory, such problems are usually NP-hard. But in practice, they are often solved sufficiently well for applications in machine learning and statistics. Even when a problem is convex, we often settle for sub-optimal solutions returned by inexact methods like stochastic gradient descent. And in nearest neighbor search, a variety of approximation algorithms works remarkably well despite the "curse of dimensionality". In this thesis, we study this phenomenon in the context of three fundamental algorithmic problems arising in the data sciences. * In constrained optimization, we show that it is possible to optimize over a wide range of non-convex sets up to the statistical noise floor. * In unconstrained optimization, we prove that important convex problems already require approximation if we want to find a solution quickly. * In nearest neighbor search, we show that approximation guarantees can explain much of the good performance observed in practice. The overarching theme is that the computational hardness of many problems emerges only below the inherent "noise floor" of real data. Hence computational hardness of these problems does not prevent us from finding answers that perform well from a statistical perspective. This offers an explanation for why algorithmic problems in the data sciences often turn out to be easier than expected.

Department dc:contributor.department

Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science.

Grantor dc:publisher

Massachusetts Institute of Technology

Year dc:date.issued

2018

Author dc:creator

• Schmidt. Ludwig, Ph. D. Massachusetts Institute of Technology

Advisor dc:contributor.advisor

• Piotr Indyk.

• Electrical Engineering and Computer Science.

Statement dc:rights

• MIT theses are protected by copyright. They may be viewed, downloaded, or printed from this source but further reproduction or distribution in any format is prohibited without written permission.

Licence dc:rights.uri

• http://dspace.mit.edu/handle/1721.1/7582

Language dc:language.iso

eng

Handle dc:identifier.uri

http://hdl.handle.net/1721.1/118098

OAI identifier oai:identifier

oai:dspace.mit.edu:1721.1/118098