Distributed dynamic partially stateful dataflow

This thesis present a distributed implementation of Noria, a new streaming dataflow system that simplifies the infrastructure of read-heavy web applications by unifying the database, caching layer, and parts of application logic in a single system. Noria's partially-stateful dataflow allows it to evict and reconstruct state on demand, and avoid prior dataflow systems' restriction to windowed state. Unlike existing dataflow systems, Noria adapts on-line to schema and query changes, and shares state and computation across related queries to eliminate duplicate effort. Noria's distributed design enables it to leverage the compute power of an entire cluster while providing high availability thanks to its fault tolerant design. On a single machine, Noria already outperforms MySQL by up to 7 x, but when running across a cluster of machines, it can scale to tens of millions of reads and millions of writes per second.

Department dc:contributor.department

Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science.

Grantor dc:publisher

Massachusetts Institute of Technology

Year dc:date.issued

2018

Author dc:creator

• Behrens, Jonathan (Jonathan Kyle)

Advisor dc:contributor.advisor

• M. Frans Kaashoek and Malte Schwarzkopf.

• Electrical Engineering and Computer Science.

Statement dc:rights

• MIT theses are protected by copyright. They may be viewed, downloaded, or printed from this source but further reproduction or distribution in any format is prohibited without written permission.

Licence dc:rights.uri

• http://dspace.mit.edu/handle/1721.1/7582

Language dc:language.iso

eng

Handle dc:identifier.uri

http://hdl.handle.net/1721.1/118054

OAI identifier oai:identifier

oai:dspace.mit.edu:1721.1/118054