{"id":{"repo_id":"mit","oai_identifier":"oai:dspace.mit.edu:1721.1/113441"},"canonical_url":"https://search.dev.ndltd.org/etd/mit/oai:dspace.mit.edu:1721.1/113441","repository":{"repo_id":"mit","name":"MIT","base_url":"https://dspace.mit.edu/oai/request"},"display":{"title":"Work-sharing framework for Apache Spark","abstract":"Apache Spark is a popular framework for distributed data processing that generalizes the MapReduce model and significantly improves the performance of many use cases. People can use Spark to query enormous data sets faster than before to gain insights for a competitive edge in industry. Often these ad-hoc queries perform similar work, and there is an opportunity to share the work of different queries. This can reduce the total computation time even more. We have developed a Wrapper class which performs such optimizations. In particular, its strategy of lazy evaluation allows duplicate computation to be avoided and multiple related Spark jobs to be executed at the same time, reducing the scheduling overhead. Overall, the system demonstrates significant efficiency gains when compared to default Spark.","abstract_html":"Apache Spark is a popular framework for distributed data processing that generalizes the MapReduce model and significantly improves the performance of many use cases. People can use Spark to query enormous data sets faster than before to gain insights for a competitive edge in industry. Often these ad-hoc queries perform similar work, and there is an opportunity to share the work of different queries. This can reduce the total computation time even more. We have developed a Wrapper class which performs such optimizations. In particular, its strategy of lazy evaluation allows duplicate computation to be avoided and multiple related Spark jobs to be executed at the same time, reducing the scheduling overhead. Overall, the system demonstrates significant efficiency gains when compared to default Spark.","abstract_has_math":false,"creators":["Yu, Lucy, M. Eng. Massachusetts Institute of Technology"],"institution":"Massachusetts Institute of Technology","degree_name":null,"degree_level":null,"degree_discipline":null,"degree_department":"Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science.","school":null,"contributors":[],"advisors":["Matei Zaharia."],"committee_chairs":[],"committee_members":[],"year":2016,"date_issued":"2016","date_published":"2016","updated_at":"2026-07-22T22:21:56Z","subjects":["Electrical Engineering and Computer Science."],"languages":["eng"],"rights":["MIT theses are protected by copyright. They may be viewed, downloaded, or printed from this source but further reproduction or distribution in any format is prohibited without written permission."],"rights_urls":["http://dspace.mit.edu/handle/1721.1/7582"],"identifier_entries":[]},"links":{"outbound_url":"http://hdl.handle.net/1721.1/113441","outbound_label":"Handle","outbound_source":"dc:identifier.uri"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor.advisor","label":"Advisor","values":["Matei Zaharia."]},{"key":"dc:contributor.department","label":"Department","values":["Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science."]},{"key":"dc:contributor.other","label":"Dc Contributor Other","values":["Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science."]},{"key":"dc:creator","label":"Author","values":["Yu, Lucy, M. Eng. Massachusetts Institute of Technology"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.accessioned","label":"Dc Date Accessioned","values":["2018-02-08T15:57:51Z"]},{"key":"dc:date.available","label":"Dc Date Available","values":["2018-02-08T15:57:51Z"]},{"key":"dc:date.issued","label":"Date","values":["2016"]},{"key":"dc:publisher","label":"Institution","values":["Massachusetts Institute of Technology"]},{"key":"dc:type","label":"Dc Type","values":["Thesis"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Electrical Engineering and Computer Science."]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language.iso","label":"Language (ISO)","values":["eng"]},{"key":"dc:rights","label":"Dc Rights","values":["MIT theses are protected by copyright. They may be viewed, downloaded, or printed from this source but further reproduction or distribution in any format is prohibited without written permission."]},{"key":"dc:rights.uri","label":"Rights URI","values":["http://dspace.mit.edu/handle/1721.1/7582"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier.uri","label":"Identifier URI","values":["http://hdl.handle.net/1721.1/113441"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Thesis: M. Eng., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2016.","This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections.","Cataloged from student-submitted PDF version of thesis.","Includes bibliographical references (page 39)."]},{"key":"dc:description.abstract","label":"Abstract","values":["Apache Spark is a popular framework for distributed data processing that generalizes the MapReduce model and significantly improves the performance of many use cases. People can use Spark to query enormous data sets faster than before to gain insights for a competitive edge in industry. Often these ad-hoc queries perform similar work, and there is an opportunity to share the work of different queries. This can reduce the total computation time even more. We have developed a Wrapper class which performs such optimizations. In particular, its strategy of lazy evaluation allows duplicate computation to be avoided and multiple related Spark jobs to be executed at the same time, reducing the scheduling overhead. Overall, the system demonstrates significant efficiency gains when compared to default Spark."]},{"key":"dc:description.degree","label":"Dc Description Degree","values":["M. Eng."]},{"key":"dc:title","label":"Title","values":["Work-sharing framework for Apache Spark"]}]}],"canonical_facts":{"dc:contributor.advisor":["Matei Zaharia."],"dc:contributor.department":["Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science."],"dc:contributor.other":["Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science."],"dc:creator":["Yu, Lucy, M. Eng. Massachusetts Institute of Technology"],"dc:date.accessioned":["2018-02-08T15:57:51Z"],"dc:date.available":["2018-02-08T15:57:51Z"],"dc:date.issued":["2016"],"dc:description":["Thesis: M. Eng., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2016.","This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections.","Cataloged from student-submitted PDF version of thesis.","Includes bibliographical references (page 39)."],"dc:description.abstract":["Apache Spark is a popular framework for distributed data processing that generalizes the MapReduce model and significantly improves the performance of many use cases. People can use Spark to query enormous data sets faster than before to gain insights for a competitive edge in industry. Often these ad-hoc queries perform similar work, and there is an opportunity to share the work of different queries. This can reduce the total computation time even more. We have developed a Wrapper class which performs such optimizations. In particular, its strategy of lazy evaluation allows duplicate computation to be avoided and multiple related Spark jobs to be executed at the same time, reducing the scheduling overhead. Overall, the system demonstrates significant efficiency gains when compared to default Spark."],"dc:description.degree":["M. Eng."],"dc:identifier.uri":["http://hdl.handle.net/1721.1/113441"],"dc:language.iso":["eng"],"dc:publisher":["Massachusetts Institute of Technology"],"dc:rights":["MIT theses are protected by copyright. They may be viewed, downloaded, or printed from this source but further reproduction or distribution in any format is prohibited without written permission."],"dc:rights.uri":["http://dspace.mit.edu/handle/1721.1/7582"],"dc:subject":["Electrical Engineering and Computer Science."],"dc:title":["Work-sharing framework for Apache Spark"],"dc:type":["Thesis"]},"updated_at":"2026-07-22T22:21:56Z"}