{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/107856"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/107856","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Timed execution in distributed machine learning","abstract":"Deep learning powers many transformative core technologies including Autonomous Driving, Natural Language Translation, and Automatic Medical Diagnosis. Its exceptional ability to extract intricate structures from high-dimensional data takes the credit for major advances in machine learning. Essential ingredients that make Deep Learning possible include: the availability of a massive curated data, a well-designed model, and readily available high-performance computation. The computation used in training deep neural networks has doubled every 3.4 months since 2012, five times faster than Moore's law. Fulfilling this massive computational demand that has long outgrown the capability of a single high-end node is vital to keep extending the flow of innovations. For example, in 2018, the AlphaGoZero model trained with 1.89 ExaFlops/s times a day. The state-of-the-art GPU at the time, NVidia V-100, could only deliver 125 TeraFlops. In a meanwhile, Summit, the fastest supercomputer in the world, could sustain 1 ExaFlops/s using 27,360 NVidia V-100 GPUs through distributed computation. This dissertation studies the challenges of scaling out an ML job to keep up with the computational demand, specifically the problems stemmed from the complex interplay of various resources in the distributed ML. In the first stage of this research, we developed methods and systems to properly observe a distributed ML environment by tracing a distributed execution, visualizing the results, and expanding the observability to the production infrastructure. Later we developed three systems to address scalability challenges using these methods and systems based on a precise execution timing of the spectrum of resources: Network: TicTac reduces internal computation delays by enforcing a near-optimal order on network transfers, which results in up to 37.7% throughput increases. Computation: Caramel increases the network utilization and decreases network congestion by modifying the order of computation and choosing the most fitted collective primitive for the workload. This result in cutting the training time up by a factor of up to $3.8$. While computation and network scheduling suggest an order of execution, TimedRPC addresses the issue of correctly enforcing this order by implementing a priority-based scheduling through pre-emption where an on-going transfer can be paused when a transfer with higher priority is requested. I/O: Diot maximizes the I/O throughput by tuning knobs such as number of concurrent I/O requests and read size on I/O pipeline. Additionally it detects the I/O delivery unfairness which may causes a struggling worker due to slow I/O through. Thesis Statement: Heuristic timing of distributed machine learning execution leads to utilization optimizations for computation, network, and storage, which in turn improves the overall throughput. In general, any multi-resource optimization involving parallelism is at least NP-Hard.","abstract_html":"Deep learning powers many transformative core technologies including Autonomous Driving, Natural Language Translation, and Automatic Medical Diagnosis. Its exceptional ability to extract intricate structures from high-dimensional data takes the credit for major advances in machine learning. Essential ingredients that make Deep Learning possible include: the availability of a massive curated data, a well-designed model, and readily available high-performance computation. The computation used in training deep neural networks has doubled every 3.4 months since 2012, five times faster than Moore's law. Fulfilling this massive computational demand that has long outgrown the capability of a single high-end node is vital to keep extending the flow of innovations. For example, in 2018, the AlphaGoZero model trained with 1.89 ExaFlops/s times a day. The state-of-the-art GPU at the time, NVidia V-100, could only deliver 125 TeraFlops. In a meanwhile, Summit, the fastest supercomputer in the world, could sustain 1 ExaFlops/s using 27,360 NVidia V-100 GPUs through distributed computation. This dissertation studies the challenges of scaling out an ML job to keep up with the computational demand, specifically the problems stemmed from the complex interplay of various resources in the distributed ML. In the first stage of this research, we developed methods and systems to properly observe a distributed ML environment by tracing a distributed execution, visualizing the results, and expanding the observability to the production infrastructure. Later we developed three systems to address scalability challenges using these methods and systems based on a precise execution timing of the spectrum of resources: Network: TicTac reduces internal computation delays by enforcing a near-optimal order on network transfers, which results in up to 37.7% throughput increases. Computation: Caramel increases the network utilization and decreases network congestion by modifying the order of computation and choosing the most fitted collective primitive for the workload. This result in cutting the training time up by a factor of up to $3.8$. While computation and network scheduling suggest an order of execution, TimedRPC addresses the issue of correctly enforcing this order by implementing a priority-based scheduling through pre-emption where an on-going transfer can be paused when a transfer with higher priority is requested. I/O: Diot maximizes the I/O throughput by tuning knobs such as number of concurrent I/O requests and read size on I/O pipeline. Additionally it detects the I/O delivery unfairness which may causes a struggling worker due to slow I/O through. Thesis Statement: Heuristic timing of distributed machine learning execution leads to utilization optimizations for computation, network, and storage, which in turn improves the overall throughput. In general, any multi-resource optimization involving parallelism is at least NP-Hard.","abstract_has_math":true,"creators":["Hashemi, Sayed Hadi"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"Ph.D.","degree_level":"Dissertation","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":["Campbell, Roy H","Kindratenko, Volodymyr","Gropp, William D","Godfrey, Philip B","Diamos, Gregory F"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2020,"date_issued":"2020-08-26T21:54:05Z","date_published":"2020-08-26T21:54:05Z","updated_at":"2026-07-22T22:24:47Z","subjects":["Machine Learning Systems, Distributed Systems, Deep Learning, Deep Neural Networks, TensorFlow, Scheduling"],"languages":["en"],"rights":["Copyright 2020 Sayed Hadi Hashemi"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"http://hdl.handle.net/2142/107856","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Campbell, Roy H","Kindratenko, Volodymyr","Gropp, William D","Godfrey, Philip B","Diamos, Gregory F"]},{"key":"dc:creator","label":"Author","values":["Hashemi, Sayed Hadi"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2020-08-26T21:54:05Z","2020-05-08","2020-05"]},{"key":"dc:type","label":"Dc Type","values":["text","Thesis"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Dissertation"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Ph.D."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Machine Learning Systems, Distributed Systems, Deep Learning, Deep Neural Networks, TensorFlow, Scheduling"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2020 Sayed Hadi Hashemi"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["http://hdl.handle.net/2142/107856"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Deep learning powers many transformative core technologies including Autonomous Driving, Natural Language Translation, and Automatic Medical Diagnosis. Its exceptional ability to extract intricate structures from high-dimensional data takes the credit for major advances in machine learning. Essential ingredients that make Deep Learning possible include: the availability of a massive curated data, a well-designed model, and readily available high-performance computation. The computation used in training deep neural networks has doubled every 3.4 months since 2012, five times faster than Moore's law. Fulfilling this massive computational demand that has long outgrown the capability of a single high-end node is vital to keep extending the flow of innovations. For example, in 2018, the AlphaGoZero model trained with 1.89 ExaFlops/s times a day. The state-of-the-art GPU at the time, NVidia V-100, could only deliver 125 TeraFlops. In a meanwhile, Summit, the fastest supercomputer in the world, could sustain 1 ExaFlops/s using 27,360 NVidia V-100 GPUs through distributed computation. This dissertation studies the challenges of scaling out an ML job to keep up with the computational demand, specifically the problems stemmed from the complex interplay of various resources in the distributed ML. In the first stage of this research, we developed methods and systems to properly observe a distributed ML environment by tracing a distributed execution, visualizing the results, and expanding the observability to the production infrastructure. Later we developed three systems to address scalability challenges using these methods and systems based on a precise execution timing of the spectrum of resources: Network: TicTac reduces internal computation delays by enforcing a near-optimal order on network transfers, which results in up to 37.7% throughput increases. Computation: Caramel increases the network utilization and decreases network congestion by modifying the order of computation and choosing the most fitted collective primitive for the workload. This result in cutting the training time up by a factor of up to $3.8$. While computation and network scheduling suggest an order of execution, TimedRPC addresses the issue of correctly enforcing this order by implementing a priority-based scheduling through pre-emption where an on-going transfer can be paused when a transfer with higher priority is requested. I/O: Diot maximizes the I/O throughput by tuning knobs such as number of concurrent I/O requests and read size on I/O pipeline. Additionally it detects the I/O delivery unfairness which may causes a struggling worker due to slow I/O through. Thesis Statement: Heuristic timing of distributed machine learning execution leads to utilization optimizations for computation, network, and storage, which in turn improves the overall throughput. In general, any multi-resource optimization involving parallelism is at least NP-Hard.","Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2020-08-25 without embargo terms","The student, Hadi Hashemi, accepted the attached license on 2020-05-02 at 16:29.","The student, Hadi Hashemi, submitted this Dissertation for approval on 2020-05-02 at 23:01.","This Dissertation was approved for publication on 2020-05-08 at 15:05.","DSpace SAF Submission Ingestion Package generated from Vireo submission #14892 on 2020-08-25 at 17:05:26","Made available in DSpace on 2020-08-26T21:54:05Z (GMT). No. of bitstreams: 3 HASHEMI-DISSERTATION-2020.pdf: 4299408 bytes, checksum: 635ce6153800295931d389b6e9284cda (MD5) LICENSE.txt: 4209 bytes, checksum: 5da3e5ae6dfd8662e7472ddf0648276b (MD5) PROQUEST_LICENSE.txt: 4555 bytes, checksum: 7e7e987d2ac8601f95dde1c49242a11a (MD5) Previous issue date: 2020-05-08","\"Updated information in dc.creator field from \"\"Hashemi, Hadi\"\" to \"\"Hashemi, Sayed Hadi\"\" by request of the author.\""]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Timed execution in distributed machine learning"]}]}],"canonical_facts":{"dc:contributor":["Campbell, Roy H","Kindratenko, Volodymyr","Gropp, William D","Godfrey, Philip B","Diamos, Gregory F"],"dc:creator":["Hashemi, Sayed Hadi"],"dc:date":["2020-08-26T21:54:05Z","2020-05-08","2020-05"],"dc:description":["Deep learning powers many transformative core technologies including Autonomous Driving, Natural Language Translation, and Automatic Medical Diagnosis. Its exceptional ability to extract intricate structures from high-dimensional data takes the credit for major advances in machine learning. Essential ingredients that make Deep Learning possible include: the availability of a massive curated data, a well-designed model, and readily available high-performance computation. The computation used in training deep neural networks has doubled every 3.4 months since 2012, five times faster than Moore's law. Fulfilling this massive computational demand that has long outgrown the capability of a single high-end node is vital to keep extending the flow of innovations. For example, in 2018, the AlphaGoZero model trained with 1.89 ExaFlops/s times a day. The state-of-the-art GPU at the time, NVidia V-100, could only deliver 125 TeraFlops. In a meanwhile, Summit, the fastest supercomputer in the world, could sustain 1 ExaFlops/s using 27,360 NVidia V-100 GPUs through distributed computation. This dissertation studies the challenges of scaling out an ML job to keep up with the computational demand, specifically the problems stemmed from the complex interplay of various resources in the distributed ML. In the first stage of this research, we developed methods and systems to properly observe a distributed ML environment by tracing a distributed execution, visualizing the results, and expanding the observability to the production infrastructure. Later we developed three systems to address scalability challenges using these methods and systems based on a precise execution timing of the spectrum of resources: Network: TicTac reduces internal computation delays by enforcing a near-optimal order on network transfers, which results in up to 37.7% throughput increases. Computation: Caramel increases the network utilization and decreases network congestion by modifying the order of computation and choosing the most fitted collective primitive for the workload. This result in cutting the training time up by a factor of up to $3.8$. While computation and network scheduling suggest an order of execution, TimedRPC addresses the issue of correctly enforcing this order by implementing a priority-based scheduling through pre-emption where an on-going transfer can be paused when a transfer with higher priority is requested. I/O: Diot maximizes the I/O throughput by tuning knobs such as number of concurrent I/O requests and read size on I/O pipeline. Additionally it detects the I/O delivery unfairness which may causes a struggling worker due to slow I/O through. Thesis Statement: Heuristic timing of distributed machine learning execution leads to utilization optimizations for computation, network, and storage, which in turn improves the overall throughput. In general, any multi-resource optimization involving parallelism is at least NP-Hard.","Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2020-08-25 without embargo terms","The student, Hadi Hashemi, accepted the attached license on 2020-05-02 at 16:29.","The student, Hadi Hashemi, submitted this Dissertation for approval on 2020-05-02 at 23:01.","This Dissertation was approved for publication on 2020-05-08 at 15:05.","DSpace SAF Submission Ingestion Package generated from Vireo submission #14892 on 2020-08-25 at 17:05:26","Made available in DSpace on 2020-08-26T21:54:05Z (GMT). No. of bitstreams: 3 HASHEMI-DISSERTATION-2020.pdf: 4299408 bytes, checksum: 635ce6153800295931d389b6e9284cda (MD5) LICENSE.txt: 4209 bytes, checksum: 5da3e5ae6dfd8662e7472ddf0648276b (MD5) PROQUEST_LICENSE.txt: 4555 bytes, checksum: 7e7e987d2ac8601f95dde1c49242a11a (MD5) Previous issue date: 2020-05-08","\"Updated information in dc.creator field from \"\"Hashemi, Hadi\"\" to \"\"Hashemi, Sayed Hadi\"\" by request of the author.\""],"dc:format":["application/pdf"],"dc:identifier":["http://hdl.handle.net/2142/107856"],"dc:language":["en"],"dc:rights":["Copyright 2020 Sayed Hadi Hashemi"],"dc:subject":["Machine Learning Systems, Distributed Systems, Deep Learning, Deep Neural Networks, TensorFlow, Scheduling"],"dc:title":["Timed execution in distributed machine learning"],"dc:type":["text","Thesis"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["Dissertation"],"thesis:degree_name":["Ph.D."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:24:47Z"}