Parallel and Distributed Just-in-Time Shell Script Compilation

In the past several years, the shell has received renewed interest from the research community. This thesis describes the work I did to advance the performance and capabilities of the current state-of-the-art shell-script parallelization systems. In the first half of this thesis, I focus on my contributions to PaSh-JIT, a JIT compiler for parallelizing POSIX shell scripts. In the second half, I explore the design and implementation of Distributed-PaSh, a shell that can utilize distributed computing resources and easily interface with distributed storage systems to efficiently execute data-processing pipelines. Distributed-PaSh analyzes the dataflow graph of a given script to create highly parallel data pipelines and execute those pipelines in a distributed cluster while giving special attention to data locality and movement. Distributed-PaSh achieves higher performance than single machine sequential and parallel shells.

Name thesis:degree_name

Master

Department dc:contributor.department

Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science

Grantor dc:publisher

Massachusetts Institute of Technology

Year dc:date.issued

2022

Author dc:creator

• Mustafa, Tammam

Advisors dc:contributor.advisor

• Vasilakis, Nikos
• Rinard, Martin C.

Statement dc:rights

• In Copyright - Educational Use Permitted
• Copyright MIT

Licence dc:rights.uri

• http://rightsstatements.org/page/InC-EDU/1.0/

Handle dc:identifier.uri

https://hdl.handle.net/1721.1/147902

OAI identifier oai:identifier

oai:dspace.mit.edu:1721.1/147902