Henge: An intent-driven scheduler for multi-tenant stream processing

This thesis presents Henge, a system that supports intent-based multi-tenancy in modern stream processing applications. Henge supports multi-tenancy as a first-class citizen: everyone inside an organization can now submit their stream processing jobs to a single, shared, consolidated cluster. Additionally, Henge allows each tenant (job) to specify its own intents (i.e., requirements) as a Service Level Objective (SLO) that captures latency and/or throughput. In a multi-tenant cluster, the Henge scheduler adapts continually to meet jobs’ SLOs in spite of limited cluster resources, and under dynamic input workloads. SLOs are soft and are based on utility functions. Henge continually tracks SLO satisfaction, and when jobs miss their SLOs, it wisely navigates the state space to perform resource allocations in real time, maximizing total system utility achieved by all jobs in the system. Henge is integrated in Apache Storm and the thesis presents experimental results, using both production topologies and real datasets.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2018

Author dc:creator

• Kalim, Faria

Contributors dc:contributor

• Gupta, Indranil

• Stream processing
• Service level objectives
• Multi-tenancy
• Distributed systems

Statement dc:rights

• Copyright 2017 Faria Kalim

Language dc:language

en

Handle dc:identifier

http://hdl.handle.net/2142/99503

OAI identifier oai:identifier

oai:www.ideals.illinois.edu:2142/99503