Adaptive batching of streams to enhance throughput and to support dynamic load balancing

As data permeates all disciplines, the role of big data becomes increasingly important. Sensors, IoT devices, social networks, and online transactions are all generating data that can be monitored constantly to enable a business to identify opportunity to enhance customer service and increase revenue. This need for real-time processing of big data has led to the development of frameworks for distributed stream processing in clusters. It is important for such frameworks to be resilient against variable operating conditions such as server load variation, changes in data ingestion rates, and workload characteristics. In this thesis, we explore the effects of the batch size on the performance of streaming workloads by developing an adaptive batching framework and building load-balancing algorithms on top of this framework. We explore the idea of using a combination of adaptive batching of tuples and dynamic tuple dispatching to improve the throughput and load-distribution of the workload. We show through experiments that the system is able to be resilient and robust under varying operating conditions.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2017

Author dc:creator

• Jayakumar, Anirudh

Contributors dc:contributor

• Abdelzaher, Tarek F.

• Load-balancing
• Apache storm
• Stream processing

Statement dc:rights

• Copyright 2016 Anirudh Jayakumar

Language dc:language

en

Handle dc:identifier

http://hdl.handle.net/2142/95402

OAI identifier oai:identifier

oai:www.ideals.illinois.edu:2142/95402