Optimizing end-to-end machine learning pipelines for model training

Modern data analysis programs often consist of complex operations. They combine multiple heterogeneous data sources, perform data cleaning and feature transformations, and apply machine learning algorithms to train models on the preprocessed data. Existing systems can execute such end-to-end training pipelines. However, they face unique challenges in their applicability to large scale data. In particular, current approaches either rely on in-memory execution (e.g., Python Pandas and scikit-learn) or they do not provide convenient programming abstractions for specifying data analysis programs (e.g., Apache Flink and Spark). Moreover, these systems do not support optimizations across operations in these pipelines, which also limits their efficient execution. In this thesis, we present our contributions towards a more efficient execution of end-to-end machine learning pipelines for model training. In particular, we discuss the following three contributions: In our first contribution, we propose a programming abstraction in the dynamic language R on top of the distributed dataflow engine Apache Flink. Our abstraction scales to large amounts of data but also hides system specifics of the dataflow engine. We integrate R and the system language Java in a shared runtime to alleviate the performance overhead of current solutions for language integration in case of complex user-defined functions. In our second contribution, we introduce an intermediate representation for data analytics programs, which allows us to optimize the complete training pipeline. We base our approach on a low-level intermediate representation that provides access to the whole program and augments two abstractions on top of it: a layer that unifies the application of user-defined functions enables operator fusion and pushdown; a high-level representation of the operations enables plan variant selection. In our third contribution, we propose a new context-aware operator for dataflow engines, which directly creates an efficient partitioning schema for matrices from normalized data sources. The operator decouples the evaluation of the join predicate from the materialization of the result. This minimizes data shuffling in distributed settings and enables the operator to choose the materialization strategy depending on the shape of the input relations. In summary, we conclude that we need a holistic system design that covers all tiers – programming abstraction, intermediate representation, and execution backend – to overcome the scalability challenges of large-scale data analysis programs.

Author dc:creator

• Kunft, Andreas

Advisor dc:contributor.advisor

• Markl, Volker

Licence dc:rights.uri

• https://creativecommons.org/licenses/by/4.0/

Language dc:language.iso

en

Identifier URI

http://dx.doi.org/10.14279/depositonce-8953

OAI identifier oai:identifier

oai:depositonce.tu-berlin.de:11303/9943