Fault tolerance in parallel data processing systems

These days data is collected any time and everywhere. The number of devices we are using every day is steadily growing. Most of those devices collect data about their usage and environment. That data is no longer gathered to answer a particular hypothesis. Instead, it is gathered to find patterns that could build a hypothesis. The collected data is often semi-structured, may stem from different sources, and is probably cluttered. The term BigData emerged for this kind of information. Parallel data processing systems are designed to handle BigData. They work on a large number of parallel working nodes. The high number of nodes and the long runtime of jobs lead to a high failure probability. Existing fault tolerance strategies for parallel data processing systems usually handle faults with full restarts or work in a blocking manner. Either the systems do not consider faults at all, and restart the entire job if a fault occurs, or they save all intermediate data before they start the next task. This thesis proposes better approach to fault tolerance in parallel data processing systems. The basis of the approach reduces restarts and works in a nonblocking manner. The introduced ephemeral materialization points hold intermediate data in memory while monitoring the running job. This monitoring enables the system to choose the sweet spots for materialization. At the same time, the materialization points allow the pipelining of data. Based on this method the thesis introduces several continuous fault tolerance techniques. On the one hand, it covers data and software faults, which are not included by the typical retry methods. On the other hand, it covers further optimizations mfor jobs with stateless tasks. Stateless tasks do not have to reprocess all their input to produce the same output, as they do not have to reach a certain state. The possibility to run a task at any point of the input stream offers the opportunity for further optimizations on the fault tolerance method. In this case it is possible to add additional nodes to the system during recovery or to skip parts of the input stream. The evaluations of the approaches show that they offer a fast recovery with small runtime and disc space overhead in a failure free case.

Author dc:creator

• Höger, Mareike Ruth

Advisor dc:contributor.advisor

• Kao, Odej

Licence dc:rights.uri

• http://rightsstatements.org/vocab/InC/1.0/

Language dc:language.iso

en

Identifier URI

http://dx.doi.org/10.14279/depositonce-9089

OAI identifier oai:identifier

oai:depositonce.tu-berlin.de:11303/10101