Demystifying a dark art: Understanding real-world machine learning model development

It is well-known that the process of developing machine learning (ML) workflows is a dark-art; even experts struggle to find an optimal workflow leading to a high accuracy model. Users currently rely on empirical trial-and-error to obtain their own set of battle-tested guidelines to inform their modeling decisions. In this study, we aim to demystify this dark art by understanding how people iterate on ML workflows in practice. We analyze over 475k user-generated workflows on OpenML, an open-source platform for tracking and sharing ML workflows. We find that users often adopt a manual, automated, or mixed approach when iterating on their workflows. We observe that manual approaches result in fewer wasted iterations compared to automated approaches. Yet, automated approaches often involve more preprocessing and hyperparameter options explored, resulting in higher performance overall---suggesting potential benefits for a human-in-the-loop ML system that appropriately recommends a clever combination of the two strategies.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2020

Author dc:creator

• Lee, Angela

Contributors dc:contributor

• Parameswaran, Aditya

• machine learning
• data analysis
• empirical studies
• user behavior

Statement dc:rights

• Copyright 2020 Angela Lee

Language dc:language

en

Handle dc:identifier

http://hdl.handle.net/2142/108007

OAI identifier oai:identifier

oai:www.ideals.illinois.edu:2142/108007