{"id":{"repo_id":"mit","oai_identifier":"oai:dspace.mit.edu:1721.1/151471"},"canonical_url":"https://search.dev.ndltd.org/etd/mit/oai:dspace.mit.edu:1721.1/151471","repository":{"repo_id":"mit","name":"MIT","base_url":"https://dspace.mit.edu/oai/request"},"display":{"title":"SparkSim: A Counterfactual Approach for Spark Cluster Scheduling","abstract":"Simulating and testing scheduling policies can be immensely time- and resourceintensive. In this work, we explore a novel approach, SparkSim, to scheduling policy training that is faster and more efficient than traditional scheduling policy testing. Our approach is based on an extension of CausalSim’s existing trace-driven approach [3], which we apply to replace the current Spark Cluster scheduling policy testing in simulation. To simulate the runtime under a new scheduling policy, our method consists of training a neural model to learn about unseen and unbiased computation elements of the cluster, extracting them, and using them as latents in predicting the duration of a workload from an existing trace. We implement this using a counterfactual approach, which takes a trace that was executed to predict a new one as if it had taken place under the same cluster conditions. My thesis focuses on evaluating and investigating the performance of SparkSim. We evaluate SparkSim on two baselines that do not require training. Our results show that SparkSim underperforms against these baselines during easier prediction tasks (such as copying from source), but outperforms them when the prediction tasks get harder. Future work lends itself to greatly improve upon these results.","abstract_html":"Simulating and testing scheduling policies can be immensely time- and resourceintensive. In this work, we explore a novel approach, SparkSim, to scheduling policy training that is faster and more efficient than traditional scheduling policy testing. Our approach is based on an extension of CausalSim’s existing trace-driven approach [3], which we apply to replace the current Spark Cluster scheduling policy testing in simulation. To simulate the runtime under a new scheduling policy, our method consists of training a neural model to learn about unseen and unbiased computation elements of the cluster, extracting them, and using them as latents in predicting the duration of a workload from an existing trace. We implement this using a counterfactual approach, which takes a trace that was executed to predict a new one as if it had taken place under the same cluster conditions. My thesis focuses on evaluating and investigating the performance of SparkSim. We evaluate SparkSim on two baselines that do not require training. Our results show that SparkSim underperforms against these baselines during easier prediction tasks (such as copying from source), but outperforms them when the prediction tasks get harder. Future work lends itself to greatly improve upon these results.","abstract_has_math":false,"creators":["Rodríguez Garnica, Sol Estrella"],"institution":"Massachusetts Institute of Technology","degree_name":"Master","degree_level":null,"degree_discipline":null,"degree_department":"Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science","school":null,"contributors":[],"advisors":["Nasr-Esfahany, Arash","Madden, Samuel","Alizadeh, Mohammad"],"committee_chairs":[],"committee_members":[],"year":2023,"date_issued":"2023-06","date_published":"2023-06","updated_at":"2026-07-22T22:21:32Z","subjects":[],"languages":[],"rights":["In Copyright - Educational Use Permitted","Copyright retained by author(s)"],"rights_urls":["https://rightsstatements.org/page/InC-EDU/1.0/"],"identifier_entries":[]},"links":{"outbound_url":"https://hdl.handle.net/1721.1/151471","outbound_label":"Handle","outbound_source":"dc:identifier.uri"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor.advisor","label":"Advisor","values":["Nasr-Esfahany, Arash","Madden, Samuel","Alizadeh, Mohammad"]},{"key":"dc:contributor.department","label":"Department","values":["Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science"]},{"key":"dc:creator","label":"Author","values":["Rodríguez Garnica, Sol Estrella"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.accessioned","label":"Dc Date Accessioned","values":["2023-07-31T19:42:23Z"]},{"key":"dc:date.available","label":"Dc Date Available","values":["2023-07-31T19:42:23Z"]},{"key":"dc:date.issued","label":"Date","values":["2023-06"]},{"key":"dc:publisher","label":"Institution","values":["Massachusetts Institute of Technology"]},{"key":"dc:type","label":"Dc Type","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Master","Master of Engineering in Electrical Engineering and Computer Science"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:rights","label":"Dc Rights","values":["In Copyright - Educational Use Permitted","Copyright retained by author(s)"]},{"key":"dc:rights.uri","label":"Rights URI","values":["https://rightsstatements.org/page/InC-EDU/1.0/"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier.uri","label":"Identifier URI","values":["https://hdl.handle.net/1721.1/151471"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["Simulating and testing scheduling policies can be immensely time- and resourceintensive. In this work, we explore a novel approach, SparkSim, to scheduling policy training that is faster and more efficient than traditional scheduling policy testing. Our approach is based on an extension of CausalSim’s existing trace-driven approach [3], which we apply to replace the current Spark Cluster scheduling policy testing in simulation. To simulate the runtime under a new scheduling policy, our method consists of training a neural model to learn about unseen and unbiased computation elements of the cluster, extracting them, and using them as latents in predicting the duration of a workload from an existing trace. We implement this using a counterfactual approach, which takes a trace that was executed to predict a new one as if it had taken place under the same cluster conditions. My thesis focuses on evaluating and investigating the performance of SparkSim. We evaluate SparkSim on two baselines that do not require training. Our results show that SparkSim underperforms against these baselines during easier prediction tasks (such as copying from source), but outperforms them when the prediction tasks get harder. Future work lends itself to greatly improve upon these results."]},{"key":"dc:description.degree","label":"Dc Description Degree","values":["M.Eng."]},{"key":"dc:title","label":"Title","values":["SparkSim: A Counterfactual Approach for Spark Cluster Scheduling"]}]}],"canonical_facts":{"dc:contributor.advisor":["Nasr-Esfahany, Arash","Madden, Samuel","Alizadeh, Mohammad"],"dc:contributor.department":["Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science"],"dc:creator":["Rodríguez Garnica, Sol Estrella"],"dc:date.accessioned":["2023-07-31T19:42:23Z"],"dc:date.available":["2023-07-31T19:42:23Z"],"dc:date.issued":["2023-06"],"dc:description.abstract":["Simulating and testing scheduling policies can be immensely time- and resourceintensive. In this work, we explore a novel approach, SparkSim, to scheduling policy training that is faster and more efficient than traditional scheduling policy testing. Our approach is based on an extension of CausalSim’s existing trace-driven approach [3], which we apply to replace the current Spark Cluster scheduling policy testing in simulation. To simulate the runtime under a new scheduling policy, our method consists of training a neural model to learn about unseen and unbiased computation elements of the cluster, extracting them, and using them as latents in predicting the duration of a workload from an existing trace. We implement this using a counterfactual approach, which takes a trace that was executed to predict a new one as if it had taken place under the same cluster conditions. My thesis focuses on evaluating and investigating the performance of SparkSim. We evaluate SparkSim on two baselines that do not require training. Our results show that SparkSim underperforms against these baselines during easier prediction tasks (such as copying from source), but outperforms them when the prediction tasks get harder. Future work lends itself to greatly improve upon these results."],"dc:description.degree":["M.Eng."],"dc:identifier.uri":["https://hdl.handle.net/1721.1/151471"],"dc:publisher":["Massachusetts Institute of Technology"],"dc:rights":["In Copyright - Educational Use Permitted","Copyright retained by author(s)"],"dc:rights.uri":["https://rightsstatements.org/page/InC-EDU/1.0/"],"dc:title":["SparkSim: A Counterfactual Approach for Spark Cluster Scheduling"],"dc:type":["Thesis"],"thesis:degree_name":["Master","Master of Engineering in Electrical Engineering and Computer Science"]},"updated_at":"2026-07-22T22:21:32Z"}