{"id":{"repo_id":"mit","oai_identifier":"oai:dspace.mit.edu:1721.1/143146"},"canonical_url":"https://search.dev.ndltd.org/etd/mit/oai:dspace.mit.edu:1721.1/143146","repository":{"repo_id":"mit","name":"MIT","base_url":"https://dspace.mit.edu/oai/request"},"display":{"title":"Network Requirements for Distributed Machine Learning Training in the Cloud","abstract":"In this thesis, I characterize the impact of network bandwidth on distributed machine learning training. I test four popular machine learning models (ResNet, DenseNet, VGG, and BERT) on an Nvidia A-100 cluster to determine the impact of bursty and non-bursty cross traffic (such as web-search traffic and long-lived flows) on the iteration time and throughput of distributed training. By varying the cross traffic load, I measure the impact of network congestion on training iteration times. I observe that with heavy web-search cross traffic (80% of link capacity), on average training iteration time is increased by up to 4 to 8×, for ResNet and BERT models, respectively. Further, I establish that the ring-all reduce communication collective is negatively impacted by network congestion even if the congestion is only affecting part of the ring. I also develop empirical models for the behavior of machine learning training in the presence of each type of cross traffic deployed. These results provide the motivation for developing novel congestion control protocols that are tailored for distributed training environments.","abstract_html":"In this thesis, I characterize the impact of network bandwidth on distributed machine learning training. I test four popular machine learning models (ResNet, DenseNet, VGG, and BERT) on an Nvidia A-100 cluster to determine the impact of bursty and non-bursty cross traffic (such as web-search traffic and long-lived flows) on the iteration time and throughput of distributed training. By varying the cross traffic load, I measure the impact of network congestion on training iteration times. I observe that with heavy web-search cross traffic (80% of link capacity), on average training iteration time is increased by up to 4 to 8×, for ResNet and BERT models, respectively. Further, I establish that the ring-all reduce communication collective is negatively impacted by network congestion even if the congestion is only affecting part of the ring. I also develop empirical models for the behavior of machine learning training in the presence of each type of cross traffic deployed. These results provide the motivation for developing novel congestion control protocols that are tailored for distributed training environments.","abstract_has_math":false,"creators":["Salamy, James"],"institution":"Massachusetts Institute of Technology","degree_name":"Master","degree_level":null,"degree_discipline":null,"degree_department":"Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science","school":null,"contributors":[],"advisors":["Ghobadi, Manya"],"committee_chairs":[],"committee_members":[],"year":2022,"date_issued":"2022-02","date_published":"2022-02","updated_at":"2026-07-22T22:21:36Z","subjects":[],"languages":[],"rights":["In Copyright - Educational Use Permitted","Copyright MIT"],"rights_urls":["http://rightsstatements.org/page/InC-EDU/1.0/"],"identifier_entries":[]},"links":{"outbound_url":"https://hdl.handle.net/1721.1/143146","outbound_label":"Handle","outbound_source":"dc:identifier.uri"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor.advisor","label":"Advisor","values":["Ghobadi, Manya"]},{"key":"dc:contributor.department","label":"Department","values":["Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science"]},{"key":"dc:creator","label":"Author","values":["Salamy, James"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.accessioned","label":"Dc Date Accessioned","values":["2022-06-15T12:59:27Z"]},{"key":"dc:date.available","label":"Dc Date Available","values":["2022-06-15T12:59:27Z"]},{"key":"dc:date.issued","label":"Date","values":["2022-02"]},{"key":"dc:publisher","label":"Institution","values":["Massachusetts Institute of Technology"]},{"key":"dc:type","label":"Dc Type","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Master","Master of Science in Electrical Engineering and Computer Science"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:rights","label":"Dc Rights","values":["In Copyright - Educational Use Permitted","Copyright MIT"]},{"key":"dc:rights.uri","label":"Rights URI","values":["http://rightsstatements.org/page/InC-EDU/1.0/"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier.uri","label":"Identifier URI","values":["https://hdl.handle.net/1721.1/143146"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["In this thesis, I characterize the impact of network bandwidth on distributed machine learning training. I test four popular machine learning models (ResNet, DenseNet, VGG, and BERT) on an Nvidia A-100 cluster to determine the impact of bursty and non-bursty cross traffic (such as web-search traffic and long-lived flows) on the iteration time and throughput of distributed training. By varying the cross traffic load, I measure the impact of network congestion on training iteration times. I observe that with heavy web-search cross traffic (80% of link capacity), on average training iteration time is increased by up to 4 to 8×, for ResNet and BERT models, respectively. Further, I establish that the ring-all reduce communication collective is negatively impacted by network congestion even if the congestion is only affecting part of the ring. I also develop empirical models for the behavior of machine learning training in the presence of each type of cross traffic deployed. These results provide the motivation for developing novel congestion control protocols that are tailored for distributed training environments."]},{"key":"dc:description.degree","label":"Dc Description Degree","values":["S.M."]},{"key":"dc:title","label":"Title","values":["Network Requirements for Distributed Machine Learning Training in the Cloud"]}]}],"canonical_facts":{"dc:contributor.advisor":["Ghobadi, Manya"],"dc:contributor.department":["Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science"],"dc:creator":["Salamy, James"],"dc:date.accessioned":["2022-06-15T12:59:27Z"],"dc:date.available":["2022-06-15T12:59:27Z"],"dc:date.issued":["2022-02"],"dc:description.abstract":["In this thesis, I characterize the impact of network bandwidth on distributed machine learning training. I test four popular machine learning models (ResNet, DenseNet, VGG, and BERT) on an Nvidia A-100 cluster to determine the impact of bursty and non-bursty cross traffic (such as web-search traffic and long-lived flows) on the iteration time and throughput of distributed training. By varying the cross traffic load, I measure the impact of network congestion on training iteration times. I observe that with heavy web-search cross traffic (80% of link capacity), on average training iteration time is increased by up to 4 to 8×, for ResNet and BERT models, respectively. Further, I establish that the ring-all reduce communication collective is negatively impacted by network congestion even if the congestion is only affecting part of the ring. I also develop empirical models for the behavior of machine learning training in the presence of each type of cross traffic deployed. These results provide the motivation for developing novel congestion control protocols that are tailored for distributed training environments."],"dc:description.degree":["S.M."],"dc:identifier.uri":["https://hdl.handle.net/1721.1/143146"],"dc:publisher":["Massachusetts Institute of Technology"],"dc:rights":["In Copyright - Educational Use Permitted","Copyright MIT"],"dc:rights.uri":["http://rightsstatements.org/page/InC-EDU/1.0/"],"dc:title":["Network Requirements for Distributed Machine Learning Training in the Cloud"],"dc:type":["Thesis"],"thesis:degree_name":["Master","Master of Science in Electrical Engineering and Computer Science"]},"updated_at":"2026-07-22T22:21:36Z"}