{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/101625"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/101625","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Exploring model parallelism in distributed scheduling of neural network frameworks","abstract":"The growth in size and computational requirements in training Neural Networks (NN) over the past few years has led to an increase in their sizes. In many cases, the networks can grow so large that can no longer fit on a single machine. A model parallel approach, backed by partitioning of Neural Networks and placement of operators on devices in a distributed system, provides a better distributed solution to this problem. In this thesis, we motivate the case for device placement in Neural Networks. We propose, analyze and evaluate mSCT, a polynomial time algorithmic solution to this end. Additionally, we formulate an exponential time optimal ILP solution that models the placement problem. We summarize our contributions as: 1. We propose a theoretical solution to the memory constrained placement problem with makespan and approximation ratio guarantees. 2. We compare and contrast m-SCT with other state of the art scheduling algorithms in a simulation environment and show that it consistently performs well on real world graphs across a variety of network bandwidths and memory constraints. 3. We lay the foundation for the experimental evaluation of the proposed solutions in existing Machine Learning frameworks.","abstract_html":"The growth in size and computational requirements in training Neural Networks (NN) over the past few years has led to an increase in their sizes. In many cases, the networks can grow so large that can no longer fit on a single machine. A model parallel approach, backed by partitioning of Neural Networks and placement of operators on devices in a distributed system, provides a better distributed solution to this problem. In this thesis, we motivate the case for device placement in Neural Networks. We propose, analyze and evaluate mSCT, a polynomial time algorithmic solution to this end. Additionally, we formulate an exponential time optimal ILP solution that models the placement problem. We summarize our contributions as: 1. We propose a theoretical solution to the memory constrained placement problem with makespan and approximation ratio guarantees. 2. We compare and contrast m-SCT with other state of the art scheduling algorithms in a simulation environment and show that it consistently performs well on real world graphs across a variety of network bandwidths and memory constraints. 3. We lay the foundation for the experimental evaluation of the proposed solutions in existing Machine Learning frameworks.","abstract_has_math":false,"creators":["Srivastava, Pallavi"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"M.S.","degree_level":"Thesis","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":["Gupta, Indranil"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2018,"date_issued":"2018-09-27T16:18:03Z","date_published":"2018-09-27T16:18:03Z","updated_at":"2026-07-22T22:24:40Z","subjects":["model parallelism"],"languages":["en"],"rights":["Copyright 2018 Pallavi Srivastava"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"http://hdl.handle.net/2142/101625","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Gupta, Indranil"]},{"key":"dc:creator","label":"Author","values":["Srivastava, Pallavi"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2018-09-27T16:18:03Z","2018-07-20","2018-08"]},{"key":"dc:type","label":"Dc Type","values":["text"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["M.S."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["model parallelism"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2018 Pallavi Srivastava"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["http://hdl.handle.net/2142/101625"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["The growth in size and computational requirements in training Neural Networks (NN) over the past few years has led to an increase in their sizes. In many cases, the networks can grow so large that can no longer fit on a single machine. A model parallel approach, backed by partitioning of Neural Networks and placement of operators on devices in a distributed system, provides a better distributed solution to this problem. In this thesis, we motivate the case for device placement in Neural Networks. We propose, analyze and evaluate mSCT, a polynomial time algorithmic solution to this end. Additionally, we formulate an exponential time optimal ILP solution that models the placement problem. We summarize our contributions as: 1. We propose a theoretical solution to the memory constrained placement problem with makespan and approximation ratio guarantees. 2. We compare and contrast m-SCT with other state of the art scheduling algorithms in a simulation environment and show that it consistently performs well on real world graphs across a variety of network bandwidths and memory constraints. 3. We lay the foundation for the experimental evaluation of the proposed solutions in existing Machine Learning frameworks.","Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2018-09-27 without embargo terms","The student, Pallavi Srivastava, accepted the attached license on 2018-07-19 at 12:45.","The student, Pallavi Srivastava, submitted this Thesis for approval on 2018-07-19 at 12:51.","This Thesis was approved for publication on 2018-07-20 at 10:52.","DSpace SAF Submission Ingestion Package generated from Vireo submission #12942 on 2018-09-27 at 10:50:21","Made available in DSpace on 2018-09-27T16:18:03Z (GMT). No. of bitstreams: 2 SRIVASTAVA-THESIS-2018.pdf: 1201565 bytes, checksum: c2691b3e6c18ec19acaba0fa936d7cac (MD5) LICENSE.txt: 4215 bytes, checksum: da4dcb63bf484c9aaa9d69fdae86c182 (MD5) Previous issue date: 2018-07-20"]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Exploring model parallelism in distributed scheduling of neural network frameworks"]}]}],"canonical_facts":{"dc:contributor":["Gupta, Indranil"],"dc:creator":["Srivastava, Pallavi"],"dc:date":["2018-09-27T16:18:03Z","2018-07-20","2018-08"],"dc:description":["The growth in size and computational requirements in training Neural Networks (NN) over the past few years has led to an increase in their sizes. In many cases, the networks can grow so large that can no longer fit on a single machine. A model parallel approach, backed by partitioning of Neural Networks and placement of operators on devices in a distributed system, provides a better distributed solution to this problem. In this thesis, we motivate the case for device placement in Neural Networks. We propose, analyze and evaluate mSCT, a polynomial time algorithmic solution to this end. Additionally, we formulate an exponential time optimal ILP solution that models the placement problem. We summarize our contributions as: 1. We propose a theoretical solution to the memory constrained placement problem with makespan and approximation ratio guarantees. 2. We compare and contrast m-SCT with other state of the art scheduling algorithms in a simulation environment and show that it consistently performs well on real world graphs across a variety of network bandwidths and memory constraints. 3. We lay the foundation for the experimental evaluation of the proposed solutions in existing Machine Learning frameworks.","Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2018-09-27 without embargo terms","The student, Pallavi Srivastava, accepted the attached license on 2018-07-19 at 12:45.","The student, Pallavi Srivastava, submitted this Thesis for approval on 2018-07-19 at 12:51.","This Thesis was approved for publication on 2018-07-20 at 10:52.","DSpace SAF Submission Ingestion Package generated from Vireo submission #12942 on 2018-09-27 at 10:50:21","Made available in DSpace on 2018-09-27T16:18:03Z (GMT). No. of bitstreams: 2 SRIVASTAVA-THESIS-2018.pdf: 1201565 bytes, checksum: c2691b3e6c18ec19acaba0fa936d7cac (MD5) LICENSE.txt: 4215 bytes, checksum: da4dcb63bf484c9aaa9d69fdae86c182 (MD5) Previous issue date: 2018-07-20"],"dc:format":["application/pdf"],"dc:identifier":["http://hdl.handle.net/2142/101625"],"dc:language":["en"],"dc:rights":["Copyright 2018 Pallavi Srivastava"],"dc:subject":["model parallelism"],"dc:title":["Exploring model parallelism in distributed scheduling of neural network frameworks"],"dc:type":["text"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["Thesis"],"thesis:degree_name":["M.S."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:24:40Z"}