{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/99417"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/99417","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Why deep neural networks for function approximation","abstract":"Recently there has been much interest in understanding why deep neural networks are preferred to shallow networks. We show that, for a large class of piecewise smooth functions, the number of neurons needed by a shallow network to approximate a function is exponentially larger than the corresponding number of neurons needed by a deep network for a given degree of function approximation. First, we consider univariate functions on a bounded interval and require a neural network to achieve an approximation error of ε uniformly over the interval. We show that shallow networks (i.e., networks whose depth does not depend on ε) require Ω(poly(1/ε)) neurons while deep networks (i.e., networks whose depth grows with 1/ε) require O(polylog(1/ε)) neurons. We then extend these results to certain classes of important multivariate functions. Our results are derived for neural networks which use a combination of rectifier linear units (ReLUs) and binary step units, two of the most popular types of activation functions. Our analysis builds on a simple observation: the multiplication of two bits can be represented by a ReLU.","abstract_html":"Recently there has been much interest in understanding why deep neural networks are preferred to shallow networks. We show that, for a large class of piecewise smooth functions, the number of neurons needed by a shallow network to approximate a function is exponentially larger than the corresponding number of neurons needed by a deep network for a given degree of function approximation. First, we consider univariate functions on a bounded interval and require a neural network to achieve an approximation error of ε uniformly over the interval. We show that shallow networks (i.e., networks whose depth does not depend on ε) require Ω(poly(1/ε)) neurons while deep networks (i.e., networks whose depth grows with 1/ε) require O(polylog(1/ε)) neurons. We then extend these results to certain classes of important multivariate functions. Our results are derived for neural networks which use a combination of rectifier linear units (ReLUs) and binary step units, two of the most popular types of activation functions. Our analysis builds on a simple observation: the multiplication of two bits can be represented by a ReLU.","abstract_has_math":false,"creators":["Liang, Shiyu"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"M.S.","degree_level":"Thesis","degree_discipline":"Electrical & Computer Engr","degree_department":null,"school":null,"contributors":["Srikant, Rayadurgam"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2018,"date_issued":"2018-03-13T15:49:14Z","date_published":"2018-03-13T15:49:14Z","updated_at":"2026-07-22T22:24:37Z","subjects":["Neural networks","Deep learning"],"languages":["en"],"rights":["Copyright 2017 Shiyu Liang"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"http://hdl.handle.net/2142/99417","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Srikant, Rayadurgam"]},{"key":"dc:creator","label":"Author","values":["Liang, Shiyu"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2018-03-13T15:49:14Z","2017-12-12","2017-12"]},{"key":"dc:type","label":"Dc Type","values":["text"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Electrical & Computer Engr"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["M.S."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Neural networks","Deep learning"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2017 Shiyu Liang"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["http://hdl.handle.net/2142/99417"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Recently there has been much interest in understanding why deep neural networks are preferred to shallow networks. We show that, for a large class of piecewise smooth functions, the number of neurons needed by a shallow network to approximate a function is exponentially larger than the corresponding number of neurons needed by a deep network for a given degree of function approximation. First, we consider univariate functions on a bounded interval and require a neural network to achieve an approximation error of ε uniformly over the interval. We show that shallow networks (i.e., networks whose depth does not depend on ε) require Ω(poly(1/ε)) neurons while deep networks (i.e., networks whose depth grows with 1/ε) require O(polylog(1/ε)) neurons. We then extend these results to certain classes of important multivariate functions. Our results are derived for neural networks which use a combination of rectifier linear units (ReLUs) and binary step units, two of the most popular types of activation functions. Our analysis builds on a simple observation: the multiplication of two bits can be represented by a ReLU.","Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2018-03-13 without embargo terms","The student, Shiyu Liang, accepted the attached license on 2017-12-11 at 16:30.","The student, Shiyu Liang, submitted this Thesis for approval on 2017-12-11 at 16:34.","This Thesis was approved for publication on 2017-12-12 at 12:47.","DSpace SAF Submission Ingestion Package generated from Vireo submission #11944 on 2018-03-13 at 10:12:16","Made available in DSpace on 2018-03-13T15:49:14Z (GMT). No. of bitstreams: 2 LIANG-THESIS-2017.pdf: 917126 bytes, checksum: bf157411ee10c2b5f29a5c41a2c184d3 (MD5) LICENSE.txt: 4208 bytes, checksum: dc5e44094544c0516a88c813369573a5 (MD5) Previous issue date: 2017-12-12"]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Why deep neural networks for function approximation"]}]}],"canonical_facts":{"dc:contributor":["Srikant, Rayadurgam"],"dc:creator":["Liang, Shiyu"],"dc:date":["2018-03-13T15:49:14Z","2017-12-12","2017-12"],"dc:description":["Recently there has been much interest in understanding why deep neural networks are preferred to shallow networks. We show that, for a large class of piecewise smooth functions, the number of neurons needed by a shallow network to approximate a function is exponentially larger than the corresponding number of neurons needed by a deep network for a given degree of function approximation. First, we consider univariate functions on a bounded interval and require a neural network to achieve an approximation error of ε uniformly over the interval. We show that shallow networks (i.e., networks whose depth does not depend on ε) require Ω(poly(1/ε)) neurons while deep networks (i.e., networks whose depth grows with 1/ε) require O(polylog(1/ε)) neurons. We then extend these results to certain classes of important multivariate functions. Our results are derived for neural networks which use a combination of rectifier linear units (ReLUs) and binary step units, two of the most popular types of activation functions. Our analysis builds on a simple observation: the multiplication of two bits can be represented by a ReLU.","Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2018-03-13 without embargo terms","The student, Shiyu Liang, accepted the attached license on 2017-12-11 at 16:30.","The student, Shiyu Liang, submitted this Thesis for approval on 2017-12-11 at 16:34.","This Thesis was approved for publication on 2017-12-12 at 12:47.","DSpace SAF Submission Ingestion Package generated from Vireo submission #11944 on 2018-03-13 at 10:12:16","Made available in DSpace on 2018-03-13T15:49:14Z (GMT). No. of bitstreams: 2 LIANG-THESIS-2017.pdf: 917126 bytes, checksum: bf157411ee10c2b5f29a5c41a2c184d3 (MD5) LICENSE.txt: 4208 bytes, checksum: dc5e44094544c0516a88c813369573a5 (MD5) Previous issue date: 2017-12-12"],"dc:format":["application/pdf"],"dc:identifier":["http://hdl.handle.net/2142/99417"],"dc:language":["en"],"dc:rights":["Copyright 2017 Shiyu Liang"],"dc:subject":["Neural networks","Deep learning"],"dc:title":["Why deep neural networks for function approximation"],"dc:type":["text"],"thesis:degree_discipline":["Electrical & Computer Engr"],"thesis:degree_level":["Thesis"],"thesis:degree_name":["M.S."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:24:37Z"}