Optimizing interactive analytics engines for heterogeneous clusters

This thesis targets the growing area of interactive data analytics engines. It builds upon a system called Getafix, an intelligent data replication and placement algorithm, and optimizes Getafix for running mixed queries over a heterogeneous cluster. The new algorithm is called Getafix-H, a cluster aware version of Getafix replication algorithm, with built-in optimizations for segment balancing and cluster auto-tiering. We integrated Getafix-H as an extension to Getafix inside Druid, a modern open-source interactive data analytics engine. We present experimental results using workloads from Yahoo!’s production Druid cluster. Compared to Getafix, Getafix-H improves the tail latency by 18% and reduces memory usage by up to 27% (2-3X improvement over Scarlett). In presence of stragglers, Getafix-H improves tail latency by 55% and reduces memory usage by upto 20% compared to Getafix. Getafix-H enables sysadmins to auto-tier a heterogeneous cluster with the tiering accuracy of up to 80%.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2018

Author dc:creator

• Raina, Ashwini

Contributors dc:contributor

• Gupta, Indranil

• Real-time analytics, data replication

Statement dc:rights

• Copyright 2018 Ashwini Raina

Language dc:language

en

Handle dc:identifier

http://hdl.handle.net/2142/101460

OAI identifier oai:identifier

oai:www.ideals.illinois.edu:2142/101460