Rethinking Serverless for Machine Learning Inference

In the era of artificial intelligence and machine learning, AI/ML inference tasks have become exceedingly popular. However, executing these workloads on dedicated hardware may not be feasible for many users due to high maintenance costs, varying load patterns, and time to production. Furthermore, ML inference workloads are stateless, and most of them are not extremely latency sensitive. For example, tasks such as fake review removal, abusive language detection, tweet classification, image tagging, and free-tier-chat-bots do not require real-time inference. All these characteristics make serverless platforms a good fit for deployment, and in this work, we identify the bottlenecks involved in hosting these inference jobs on serverless and optimize serverless for better performance and resource utilization. Specifically, we identify model loading and model memory duplication as major bottlenecks in Serverless Inference, and to address these problems, we propose a new approach that rethinks the way we serve FaaS requests. To support this design, we employ a hybrid scaling approach to implement the autoscale feature of serverless.

Name thesis:degree_name

Master of Science

Level thesis:degree_level

masters

Discipline thesis:degree_discipline

Computer Science & Applications

Department dc:contributor.department

Computer Science and Applications

Grantor dc:publisher

Virginia Tech

Year dc:date.issued

2023

Author dc:creator

• Ellore, Anish Reddy

Chair dc:contributor.committeechair

• Butt, Ali R.

Committee members dc:contributor.committeemember

• Hu, Liting
• Williams, Daniel John

• Serverles
• FaaS
• Machine Learning Inference
• Model Serving
• Container

Statement dc:rights

• In Copyright

Licence dc:rights.uri

• http://rightsstatements.org/vocab/InC/1.0/

Language dc:language.iso

en

Dc Identifier Other

vt_gsexam:38235

OAI identifier oai:identifier

oai:vtechworks.lib.vt.edu:10919/116068