{"id":{"repo_id":"carleton","oai_identifier":"oai:carleton.scholaris.ca:20.500.14718/41201"},"canonical_url":"https://search.dev.ndltd.org/etd/carleton/oai:carleton.scholaris.ca:20.500.14718/41201","repository":{"repo_id":"carleton","name":"Carleton University","base_url":"https://carleton.scholaris.ca/server/oai/request"},"display":{"title":"Automated Discovery of Big Data Workload Types","abstract":"Big data workload characterization is an inevitable part of big data workload prediction and auto-tuning big data applications. Due to many different ways of applying big data frameworks and applications, there are various categories that these workloads can belong. Clustering techniques are applied in this research to detect Apache Spark and Hadoop workloads independent of historical data. Clustering techniques are compared in terms of different evaluation metrics, and the ones with the highest performance are introduced. The DBSCAN algorithm has shown the best performance and adequacy with 71% and 80% for the Purity, and Windows Type Accuracy (Awt), respectively. Ultimately, the Incremental DBSCAN algorithm and Den-Stream (an online version of DBSCAN) are presented as the most practical methods for big data workload discovery automatization. A scheme is then provided to use these algorithms integrated with methods to self-discover their hyperparameters. Ultimately, the procedure is fully automated.","abstract_html":"Big data workload characterization is an inevitable part of big data workload prediction and auto-tuning big data applications. Due to many different ways of applying big data frameworks and applications, there are various categories that these workloads can belong. Clustering techniques are applied in this research to detect Apache Spark and Hadoop workloads independent of historical data. Clustering techniques are compared in terms of different evaluation metrics, and the ones with the highest performance are introduced. The DBSCAN algorithm has shown the best performance and adequacy with 71% and 80% for the Purity, and Windows Type Accuracy (Awt), respectively. Ultimately, the Incremental DBSCAN algorithm and Den-Stream (an online version of DBSCAN) are presented as the most practical methods for big data workload discovery automatization. A scheme is then provided to use these algorithms integrated with methods to self-discover their hyperparameters. Ultimately, the procedure is fully automated.","abstract_has_math":false,"creators":["Shahmirza, Anousheh"],"institution":"Carleton University","degree_name":"Master of Computer Science (M.C.S.)","degree_level":"Master's","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":[],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2021,"date_issued":"2021","date_published":"2021","updated_at":"2026-07-24T01:34:25Z","subjects":[],"languages":["en"],"rights":["Copyright © 2021 the author(s). Theses may be used for non-commercial research, educational, or related academic purposes only. Such uses include personal study, research, scholarship, and teaching. Theses may only be shared by linking to Carleton University Institutional Repository and no part may be used without proper attribution to the author. No part may be used for commercial purposes directly or indirectly via a for-profit platform; no adaptation or derivative works are permitted without consent from the copyright owner."],"rights_urls":[],"identifier_entries":[{"key":"dc:identifier.doi","label":"DOI","values":["10.22215/etd/2021-14757"],"render_values":[{"text":"10.22215/etd/2021-14757","href":"https://doi.org/10.22215/etd/2021-14757","code":true}]}]},"links":{"outbound_url":"https://hdl.handle.net/20.500.14718/41201","outbound_label":"Handle","outbound_source":"dc:identifier.uri"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:creator","label":"Author","values":["Shahmirza, Anousheh"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.accessioned","label":"Dc Date Accessioned","values":["2025-04-08T20:14:43Z"]},{"key":"dc:date.available","label":"Dc Date Available","values":["2025-04-08T20:14:43Z"]},{"key":"dc:date.issued","label":"Date","values":["2021"]},{"key":"dc:publisher","label":"Institution","values":["Carleton University"]},{"key":"dc:type","label":"Dc Type","values":["thesis"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Master's"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Master of Computer Science (M.C.S.)"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language.iso","label":"Language (ISO)","values":["en"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright © 2021 the author(s). Theses may be used for non-commercial research, educational, or related academic purposes only. Such uses include personal study, research, scholarship, and teaching. Theses may only be shared by linking to Carleton University Institutional Repository and no part may be used without proper attribution to the author. No part may be used for commercial purposes directly or indirectly via a for-profit platform; no adaptation or derivative works are permitted without consent from the copyright owner."]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier.doi","label":"DOI","values":["10.22215/etd/2021-14757"]},{"key":"dc:identifier.uri","label":"Identifier URI","values":["https://hdl.handle.net/20.500.14718/41201"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["Big data workload characterization is an inevitable part of big data workload prediction and auto-tuning big data applications. Due to many different ways of applying big data frameworks and applications, there are various categories that these workloads can belong. Clustering techniques are applied in this research to detect Apache Spark and Hadoop workloads independent of historical data. Clustering techniques are compared in terms of different evaluation metrics, and the ones with the highest performance are introduced. The DBSCAN algorithm has shown the best performance and adequacy with 71% and 80% for the Purity, and Windows Type Accuracy (Awt), respectively. Ultimately, the Incremental DBSCAN algorithm and Den-Stream (an online version of DBSCAN) are presented as the most practical methods for big data workload discovery automatization. A scheme is then provided to use these algorithms integrated with methods to self-discover their hyperparameters. Ultimately, the procedure is fully automated."]},{"key":"dc:title","label":"Title","values":["Automated Discovery of Big Data Workload Types"]}]}],"canonical_facts":{"dc:creator":["Shahmirza, Anousheh"],"dc:date.accessioned":["2025-04-08T20:14:43Z"],"dc:date.available":["2025-04-08T20:14:43Z"],"dc:date.issued":["2021"],"dc:description.abstract":["Big data workload characterization is an inevitable part of big data workload prediction and auto-tuning big data applications. Due to many different ways of applying big data frameworks and applications, there are various categories that these workloads can belong. Clustering techniques are applied in this research to detect Apache Spark and Hadoop workloads independent of historical data. Clustering techniques are compared in terms of different evaluation metrics, and the ones with the highest performance are introduced. The DBSCAN algorithm has shown the best performance and adequacy with 71% and 80% for the Purity, and Windows Type Accuracy (Awt), respectively. Ultimately, the Incremental DBSCAN algorithm and Den-Stream (an online version of DBSCAN) are presented as the most practical methods for big data workload discovery automatization. A scheme is then provided to use these algorithms integrated with methods to self-discover their hyperparameters. Ultimately, the procedure is fully automated."],"dc:identifier.doi":["10.22215/etd/2021-14757"],"dc:identifier.uri":["https://hdl.handle.net/20.500.14718/41201"],"dc:language.iso":["en"],"dc:publisher":["Carleton University"],"dc:rights":["Copyright © 2021 the author(s). Theses may be used for non-commercial research, educational, or related academic purposes only. Such uses include personal study, research, scholarship, and teaching. Theses may only be shared by linking to Carleton University Institutional Repository and no part may be used without proper attribution to the author. No part may be used for commercial purposes directly or indirectly via a for-profit platform; no adaptation or derivative works are permitted without consent from the copyright owner."],"dc:title":["Automated Discovery of Big Data Workload Types"],"dc:type":["thesis"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["Master's"],"thesis:degree_name":["Master of Computer Science (M.C.S.)"]},"updated_at":"2026-07-24T01:34:25Z"}