Mitigating Spark straggler tasks for iterative applications by data re-partitioning

Many of the data science applications nowadays feature large datasets and short tasks that run many iterations. When running these applications on a parallel processing framework like Apache Spark, one problem that affects the running time is the straggler, where a disproportionate long-running task slows down the entire cluster. In this work we present a straggler mitigation technique tailored for applications that run small tasks for many iterations over a large dataset, and implemented the algorithm in Apache Spark. We monitor the resources available on each Spark node, and dynamically re partition the dataset proportional to the estimated resource available. We have shown that our algorithm has negligible overhead for resource monitoring, and can improve the performance of Spark cluster significantly when stragglers are present.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2017

Author dc:creator

• Teng, Bo

Contributors dc:contributor

• Campbell, Roy H.

• Straggler
• Machine learning
• Apache Spark
• Iterative application

Statement dc:rights

• Copyright 2017 Bo Teng

Language dc:language

en

Handle dc:identifier

http://hdl.handle.net/2142/97707

OAI identifier oai:identifier

oai:www.ideals.illinois.edu:2142/97707