Everything old is new again : a fresh look at historical approaches in machine learning

This thesis shows that several old, somewhat discredited machine learning techniques are still valuable in the solution of modern, large-scale machine learning problems. We begin by considering Tikhonov regularization, a broad framework of schemes for binary classification. Tikhonov regularization attempts to find a function which simultaneously has small empirical loss on a training set and small norm in a Reproducing Kernel Hilbert Space. The choice of loss function determines the learning scheme. Using the hinge loss gives rise to the now well-known Support Vector Machine algorithm. We present SvmFu, a state-of-the-art SVM solver developed as part of thesis. We discuss the design and implementation issues involved in SvmFu, present empirical results on its performance, and offer general guidance on the use of SVMs to solve machine learning problems. We also consider, and advocate in many cases, the use of the more classical square loss, giving rise to the Regularized Least Squares Classifiation algorithm. RLSC is "trained" by solving a single system of linear equations. While it is widely believed that the SVM will perform substantially better than RLSC, we note that the same generalization bounds that apply to SVMs apply to RLSC, and we demonstrate empirically on both toy and real-world examples that RLSC's performance is essentially equivalent to SVMs across a wide range of problems, implying that the choice between SVM and RLSC should be based on computational tractability considerations. We demonstrate the empirical advantages and properties of RLSC, discussing the tradeoffs between RLSC and SVMs.

Department dc:contributor.department

Sloan School of Management.

Grantor dc:publisher

Massachusetts Institute of Technology

Year dc:date.issued

2002

Author dc:creator

• Rifkin, Ryan Michael, 1972-

Advisor dc:contributor.advisor

• Tomaso Poggio.

• Sloan School of Management.

Statement dc:rights

• M.I.T. theses are protected by copyright. They may be viewed from this source for any purpose, but reproduction or distribution in any format is prohibited without written permission. See provided URL for inquiries about permission.

Licence dc:rights.uri

• http://dspace.mit.edu/handle/1721.1/7582

Language dc:language.iso

eng

Handle dc:identifier.uri

http://hdl.handle.net/1721.1/17549

OAI identifier oai:identifier

oai:dspace.mit.edu:1721.1/17549