A conceptual model for transparent, reusable, and collaborative data cleaning

Data cleaning is an essential component of data preparation in machine learning and other data science workflows. It is a time-consuming and error-prone task that can greatly affect the reliability of subsequent analyses. Tools must capture provenance information to ensure transparent and auditable data-cleaning processes. However, existing provenance models have limitations in tracing and querying changes at different levels of granularity. To address this, we proposed a new conceptual model that captures fine-grained retrospective provenance and extends it with prospective provenance to represent operations or workflows that change the datasets. This hybrid model allows powerful queries and supports advanced use cases like auditing data cleaning workflows. Additionally, we extended the model to present a conceptual model focusing on reusability and collaboration in data cleaning. It addresses scenarios where multiple users contribute to dataset changes and enables tracking of curator actions, identifying dependencies between cleaning operations, and facilitating collaboration. Through an experimental case study, we demonstrated the reusability of data-cleaning workflows, different users' contributions, and collaboration's effectiveness in improving data quality.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Information Sciences

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2023

Author dc:creator

• Parulian, Nikolaus Nova

Contributors dc:contributor

• Ludäscher, Bertram
• Downie, John Stephen
• Diesner, Jana
• Bosch, Nigel

• Data Cleaning
• Data Quality
• Provenance
• Data Preparation
• Machine Learning
• Artificial Intelligence
• Workflows
• Metadata
• Visualization

Statement dc:rights

• Copyright 2023 Nikolaus Parulian

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/121511

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F121511