{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/108189"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/108189","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Completion of hinge loss has an implicit bias","abstract":"A new loss function is proposed which learns the hinge loss function an infinite number of times pushing $f(x_i)y_i \\to \\infty$. It is proven that for a linear model on linearly separable data this modified hinge loss function converges in the direction of the $\\ell_2$ max-margin separator at a rate of $\\bigO\\left( \\sqrt{d/t} \\right)$ where $d$ is the dimension of the data. Then, an explicit formula for the underlying dynamical system of the gradient descent iterates for two-layer linear networks on the inner product loss function is derived. Using the derived dynamical system, a precise explicit algorithm is developed which when implemented reproduces the gradient descent iterates of two-layer ReLU nets on the inner product exactly. This result is studied further to extrapolate conclusions for neural network optimization.","abstract_html":"A new loss function is proposed which learns the hinge loss function an infinite number of times pushing <span class=\"etd-inline-math\">f(x<sub>i</sub>)y<sub>i</sub> \\to \\infty</span>. It is proven that for a linear model on linearly separable data this modified hinge loss function converges in the direction of the <span class=\"etd-inline-math\">\\ell<sub>2</sub></span> max-margin separator at a rate of $\\bigO\\left( \\sqrt{d/t} \\right)$ where $d$ is the dimension of the data. Then, an explicit formula for the underlying dynamical system of the gradient descent iterates for two-layer linear networks on the inner product loss function is derived. Using the derived dynamical system, a precise explicit algorithm is developed which when implemented reproduces the gradient descent iterates of two-layer ReLU nets on the inner product exactly. This result is studied further to extrapolate conclusions for neural network optimization.","abstract_has_math":true,"creators":["Lizama, Justin N"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"M.S.","degree_level":"Thesis","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":["Telgarsky, Matus J"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2020,"date_issued":"2020-08-26T23:58:48Z","date_published":"2020-08-26T23:58:48Z","updated_at":"2026-07-22T22:24:47Z","subjects":["implicit","regularization","hinge","loss"],"languages":["en"],"rights":["Copyright 2020 Justin Lizama"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"http://hdl.handle.net/2142/108189","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Telgarsky, Matus J"]},{"key":"dc:creator","label":"Author","values":["Lizama, Justin N"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2020-08-26T23:58:48Z","2022-08-26T23:58:55Z","2020-05-12","2020-05"]},{"key":"dc:type","label":"Dc Type","values":["text","Thesis"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["M.S."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["implicit","regularization","hinge","loss"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2020 Justin Lizama"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["http://hdl.handle.net/2142/108189"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["A new loss function is proposed which learns the hinge loss function an infinite number of times pushing $f(x_i)y_i \\to \\infty$. It is proven that for a linear model on linearly separable data this modified hinge loss function converges in the direction of the $\\ell_2$ max-margin separator at a rate of $\\bigO\\left( \\sqrt{d/t} \\right)$ where $d$ is the dimension of the data. Then, an explicit formula for the underlying dynamical system of the gradient descent iterates for two-layer linear networks on the inner product loss function is derived. Using the derived dynamical system, a precise explicit algorithm is developed which when implemented reproduces the gradient descent iterates of two-layer ReLU nets on the inner product exactly. This result is studied further to extrapolate conclusions for neural network optimization.","Submission published under a 24 month embargo labeled 'U of I Access', the embargo will last until 2022-05-01","The student, Justin Lizama, accepted the attached license on 2020-05-12 at 12:38.","The student, Justin Lizama, submitted this Thesis for approval on 2020-05-12 at 13:23.","This Thesis was approved for publication on 2020-05-12 at 15:00.","DSpace SAF Submission Ingestion Package generated from Vireo submission #15352 on 2020-08-25 at 17:31:13","Made available in DSpace on 2020-08-26T23:58:48Z (GMT). No. of bitstreams: 2 LIZAMA-THESIS-2020.pdf: 1342025 bytes, checksum: a56819a442a48f4562e0e97587b99fd7 (MD5) LICENSE.txt: 4210 bytes, checksum: 0761322496976fd61d67a6ec4cb5f931 (MD5) Previous issue date: 2020-05-12","Embargo set by: Seth Robbins for item 115802 Lift date: 2022-08-26T23:58:55Z Reason: Author requested U of Illinois access only (OA after 2yrs) in Vireo ETD system","Author requested U of Illinois access only (OA after 2yrs) in Vireo ETD system","U of I Only"]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Completion of hinge loss has an implicit bias"]}]}],"canonical_facts":{"dc:contributor":["Telgarsky, Matus J"],"dc:creator":["Lizama, Justin N"],"dc:date":["2020-08-26T23:58:48Z","2022-08-26T23:58:55Z","2020-05-12","2020-05"],"dc:description":["A new loss function is proposed which learns the hinge loss function an infinite number of times pushing $f(x_i)y_i \\to \\infty$. It is proven that for a linear model on linearly separable data this modified hinge loss function converges in the direction of the $\\ell_2$ max-margin separator at a rate of $\\bigO\\left( \\sqrt{d/t} \\right)$ where $d$ is the dimension of the data. Then, an explicit formula for the underlying dynamical system of the gradient descent iterates for two-layer linear networks on the inner product loss function is derived. Using the derived dynamical system, a precise explicit algorithm is developed which when implemented reproduces the gradient descent iterates of two-layer ReLU nets on the inner product exactly. This result is studied further to extrapolate conclusions for neural network optimization.","Submission published under a 24 month embargo labeled 'U of I Access', the embargo will last until 2022-05-01","The student, Justin Lizama, accepted the attached license on 2020-05-12 at 12:38.","The student, Justin Lizama, submitted this Thesis for approval on 2020-05-12 at 13:23.","This Thesis was approved for publication on 2020-05-12 at 15:00.","DSpace SAF Submission Ingestion Package generated from Vireo submission #15352 on 2020-08-25 at 17:31:13","Made available in DSpace on 2020-08-26T23:58:48Z (GMT). No. of bitstreams: 2 LIZAMA-THESIS-2020.pdf: 1342025 bytes, checksum: a56819a442a48f4562e0e97587b99fd7 (MD5) LICENSE.txt: 4210 bytes, checksum: 0761322496976fd61d67a6ec4cb5f931 (MD5) Previous issue date: 2020-05-12","Embargo set by: Seth Robbins for item 115802 Lift date: 2022-08-26T23:58:55Z Reason: Author requested U of Illinois access only (OA after 2yrs) in Vireo ETD system","Author requested U of Illinois access only (OA after 2yrs) in Vireo ETD system","U of I Only"],"dc:format":["application/pdf"],"dc:identifier":["http://hdl.handle.net/2142/108189"],"dc:language":["en"],"dc:rights":["Copyright 2020 Justin Lizama"],"dc:subject":["implicit","regularization","hinge","loss"],"dc:title":["Completion of hinge loss has an implicit bias"],"dc:type":["text","Thesis"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["Thesis"],"thesis:degree_name":["M.S."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:24:47Z"}