{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/108007"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/108007","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Demystifying a dark art: Understanding real-world machine learning model development","abstract":"It is well-known that the process of developing machine learning (ML) workflows is a dark-art; even experts struggle to find an optimal workflow leading to a high accuracy model. Users currently rely on empirical trial-and-error to obtain their own set of battle-tested guidelines to inform their modeling decisions. In this study, we aim to demystify this dark art by understanding how people iterate on ML workflows in practice. We analyze over 475k user-generated workflows on OpenML, an open-source platform for tracking and sharing ML workflows. We find that users often adopt a manual, automated, or mixed approach when iterating on their workflows. We observe that manual approaches result in fewer wasted iterations compared to automated approaches. Yet, automated approaches often involve more preprocessing and hyperparameter options explored, resulting in higher performance overall---suggesting potential benefits for a human-in-the-loop ML system that appropriately recommends a clever combination of the two strategies.","abstract_html":"It is well-known that the process of developing machine learning (ML) workflows is a dark-art; even experts struggle to find an optimal workflow leading to a high accuracy model. Users currently rely on empirical trial-and-error to obtain their own set of battle-tested guidelines to inform their modeling decisions. In this study, we aim to demystify this dark art by understanding how people iterate on ML workflows in practice. We analyze over 475k user-generated workflows on OpenML, an open-source platform for tracking and sharing ML workflows. We find that users often adopt a manual, automated, or mixed approach when iterating on their workflows. We observe that manual approaches result in fewer wasted iterations compared to automated approaches. Yet, automated approaches often involve more preprocessing and hyperparameter options explored, resulting in higher performance overall---suggesting potential benefits for a human-in-the-loop ML system that appropriately recommends a clever combination of the two strategies.","abstract_has_math":false,"creators":["Lee, Angela"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"M.S.","degree_level":"Thesis","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":["Parameswaran, Aditya"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2020,"date_issued":"2020-08-26T21:54:57Z","date_published":"2020-08-26T21:54:57Z","updated_at":"2026-07-22T22:24:47Z","subjects":["machine learning","data analysis","empirical studies","user behavior"],"languages":["en"],"rights":["Copyright 2020 Angela Lee"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"http://hdl.handle.net/2142/108007","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Parameswaran, Aditya"]},{"key":"dc:creator","label":"Author","values":["Lee, Angela"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2020-08-26T21:54:57Z","2020-05-11","2020-05"]},{"key":"dc:type","label":"Dc Type","values":["text","Thesis"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["M.S."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["machine learning","data analysis","empirical studies","user behavior"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2020 Angela Lee"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["http://hdl.handle.net/2142/108007"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["It is well-known that the process of developing machine learning (ML) workflows is a dark-art; even experts struggle to find an optimal workflow leading to a high accuracy model. Users currently rely on empirical trial-and-error to obtain their own set of battle-tested guidelines to inform their modeling decisions. In this study, we aim to demystify this dark art by understanding how people iterate on ML workflows in practice. We analyze over 475k user-generated workflows on OpenML, an open-source platform for tracking and sharing ML workflows. We find that users often adopt a manual, automated, or mixed approach when iterating on their workflows. We observe that manual approaches result in fewer wasted iterations compared to automated approaches. Yet, automated approaches often involve more preprocessing and hyperparameter options explored, resulting in higher performance overall---suggesting potential benefits for a human-in-the-loop ML system that appropriately recommends a clever combination of the two strategies.","Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2020-08-25 without embargo terms","The student, Angela Lee, accepted the attached license on 2020-05-07 at 12:17.","The student, Angela Lee, submitted this Thesis for approval on 2020-05-07 at 12:34.","This Thesis was approved for publication on 2020-05-11 at 11:02.","DSpace SAF Submission Ingestion Package generated from Vireo submission #15274 on 2020-08-25 at 17:13:04","Made available in DSpace on 2020-08-26T21:54:57Z (GMT). No. of bitstreams: 2 LEE-THESIS-2020.pdf: 1919422 bytes, checksum: dd5eec0227826dc90420503326ab18f4 (MD5) LICENSE.txt: 4207 bytes, checksum: 000ee453a82396072bc74eb2752b0dcf (MD5) Previous issue date: 2020-05-11"]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Demystifying a dark art: Understanding real-world machine learning model development"]}]}],"canonical_facts":{"dc:contributor":["Parameswaran, Aditya"],"dc:creator":["Lee, Angela"],"dc:date":["2020-08-26T21:54:57Z","2020-05-11","2020-05"],"dc:description":["It is well-known that the process of developing machine learning (ML) workflows is a dark-art; even experts struggle to find an optimal workflow leading to a high accuracy model. Users currently rely on empirical trial-and-error to obtain their own set of battle-tested guidelines to inform their modeling decisions. In this study, we aim to demystify this dark art by understanding how people iterate on ML workflows in practice. We analyze over 475k user-generated workflows on OpenML, an open-source platform for tracking and sharing ML workflows. We find that users often adopt a manual, automated, or mixed approach when iterating on their workflows. We observe that manual approaches result in fewer wasted iterations compared to automated approaches. Yet, automated approaches often involve more preprocessing and hyperparameter options explored, resulting in higher performance overall---suggesting potential benefits for a human-in-the-loop ML system that appropriately recommends a clever combination of the two strategies.","Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2020-08-25 without embargo terms","The student, Angela Lee, accepted the attached license on 2020-05-07 at 12:17.","The student, Angela Lee, submitted this Thesis for approval on 2020-05-07 at 12:34.","This Thesis was approved for publication on 2020-05-11 at 11:02.","DSpace SAF Submission Ingestion Package generated from Vireo submission #15274 on 2020-08-25 at 17:13:04","Made available in DSpace on 2020-08-26T21:54:57Z (GMT). No. of bitstreams: 2 LEE-THESIS-2020.pdf: 1919422 bytes, checksum: dd5eec0227826dc90420503326ab18f4 (MD5) LICENSE.txt: 4207 bytes, checksum: 000ee453a82396072bc74eb2752b0dcf (MD5) Previous issue date: 2020-05-11"],"dc:format":["application/pdf"],"dc:identifier":["http://hdl.handle.net/2142/108007"],"dc:language":["en"],"dc:rights":["Copyright 2020 Angela Lee"],"dc:subject":["machine learning","data analysis","empirical studies","user behavior"],"dc:title":["Demystifying a dark art: Understanding real-world machine learning model development"],"dc:type":["text","Thesis"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["Thesis"],"thesis:degree_name":["M.S."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:24:47Z"}