TensorRT inference performance study in MLModelScope

As deep learning has been adopted in various domains, the inference process is of growing importance to ensure the deployment across multiple computing platforms. Within many deep learning frameworks that support freezing and deploying the well-trained models, NVIDIA TensorRT is the leading framework that is exclusively developed for inference. It allows the developer to optimize the model to facilitate high-performance inference. While it has been shown extensively that TensorRT can significantly boost the inference capability, quantitative study is lacking on how assorted optimization strategies can improve the inference compared to other well-known deep learning frameworks such as TensorFlow. This thesis presents such a study that consists of experiments using TensorRT on MLModelScope, a deep learning inference platform that enables standardized inference and multi-level profiling.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2020

Author dc:creator

• Tang, Jingning

Contributors dc:contributor

• Hwu, Wen-Mei

• deep learning
• inference
• TensorRT
• MLModelScope

Statement dc:rights

• Copyright 2020 Jingning Tang

Language dc:language

en

Handle dc:identifier

http://hdl.handle.net/2142/108566

OAI identifier oai:identifier

oai:www.ideals.illinois.edu:2142/108566