Performance modeling framework for SLO-driven MapReduce environments

Several companies are increasingly using MapReduce for efficient large scale data processing such as personalized advertising, spam detection, and data mining tasks. There is a growing need among MapReduce users to achieve different Service Level Objectives (SLOs). Often, applications need to complete data processing within a certain time deadline. Alternatively, users are interested in completing a set of jobs as fast as possible. Designing, prototyping, and evaluating new resource allocation and job scheduling algorithms to support these SLOs in MapReduce environments is challenging, labor-intensive, and time-consuming. Hence, accurate and efficient workload management and performance modeling tools are needed. Our hypothesis is that performance modeling of MapReduce environments through a combination of measurement, simulation, and analytical modeling for enabling different service level objectives is feasible, novel, and useful. To support this hypothesis, we propose an analytical performance model based on key performance characteristics measured from past job executions and build a simulator capable of replaying these job traces. We survey different attempts at performance modeling and its applications, and contrast our work. To demonstrate the usefulness of our techniques, we apply them to achieve service level objectives such as enabling deadline-driven scheduling, optimizing makespan of a set of MapReduce jobs and comparing hardware alternatives.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2013

Author dc:creator

• Verma, Abhishek

Contributors dc:contributor

• Campbell, Roy H.
• Gupta, Indranil
• Gropp, William D.
• Cherkasova, Ludmila

• MapReduce
• Performance Modeling
• Service Level Objectives
• Hadoop

Statement dc:rights

• Copyright 2012 Abhishek Verma

Language dc:language

en

Handle dc:identifier

http://hdl.handle.net/2142/42276

OAI identifier oai:identifier

oai:www.ideals.illinois.edu:2142/42276