LightSpeed: A Framework to Profile and Evaluate Inference Accelerators at Scale

The massive growth of machine learning-based applications, and the end of Moore’s law, created a pressing need to build highly efficient computing platforms from the ground up. Consequently, researchers and practitioners have been developing highly innovative cutting-edge architectures to meet today’s exponentially increasing demands for machine learning services. However, evaluating the performance gains of newly developed machine learning systems at scale is extremely challenging. Existing evaluation platforms are often specialized to a specific hardware target, such as GPUs, making them less amenable to novel designs. Moreover, evaluating the performance of a newly designed system at scale requires careful consideration of workload and traffic patterns. To address the above challenges, I introduce LightSpeed, a framework to profile and evaluate inference accelerators at scale. LightSpeed is an event-based simulator that enables users to compare the performance of their system to best-in-class accelerators at scale. LightSpeed profiles the computation and communication requirements of real-world deep neural networks through accurate measurements on hardware. It then simulates the service time of inference requests under a variety of accelerators and scheduling algorithms.

Name thesis:degree_name

Master

Department dc:contributor.department

Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science

Grantor dc:publisher

Massachusetts Institute of Technology

Year dc:date.issued

2023

Author dc:creator

• Williams, Christian

Advisor dc:contributor.advisor

• Ghobadi, Manya

Statement dc:rights

• In Copyright - Educational Use Permitted
• Copyright retained by author(s)

Licence dc:rights.uri

• https://rightsstatements.org/page/InC-EDU/1.0/

Handle dc:identifier.uri

https://hdl.handle.net/1721.1/151351

OAI identifier oai:identifier

oai:dspace.mit.edu:1721.1/151351