{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/121445"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/121445","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Randomness-aware testing of machine learning-based systems","abstract":"Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2023-12-04 without embargo terms","abstract_html":"Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2023-12-04 without embargo terms","abstract_has_math":false,"creators":["Dutta, Saikat"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"Ph.D.","degree_level":"Dissertation","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":["Misailovic, Sasa","Marinov, Darko","Adve, Vikram","Gligoric, Milos","Lahiri, Shuvendu"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2023,"date_issued":"2023-08","date_published":"2023-08","updated_at":"2026-07-22T22:24:57Z","subjects":["Machine Learning","Software Testing"],"languages":["en","eng"],"rights":["Copyright 2023 Saikat Dutta"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://hdl.handle.net/2142/121445","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Misailovic, Sasa","Marinov, Darko","Adve, Vikram","Gligoric, Milos","Lahiri, Shuvendu"]},{"key":"dc:creator","label":"Author","values":["Dutta, Saikat"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2023-08","2023-07-05"]},{"key":"dc:type","label":"Dc Type","values":["text"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Dissertation"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Ph.D."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Machine Learning","Software Testing"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en","eng"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2023 Saikat Dutta"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://hdl.handle.net/2142/121445"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2023-12-04 without embargo terms","The student, Saikat Dutta, accepted the attached license on 2023-06-30 at 13:12.","The student, Saikat Dutta, submitted this Dissertation for approval on 2023-06-30 at 13:21.","This Dissertation was approved for publication on 2023-07-05 at 13:56.","DSpace SAF Submission Ingestion Package generated from Vireo submission #19489 on 2023-12-04 at 17:00:22","Machine Learning (ML) is rapidly revolutionizing the way modern-day systems are developed. However, testing ML-based systems is challenging due to 1) the presence of non-determinism, both internal (e.g., stochastic algorithms) and external (e.g., execution environment), and 2) the absence of well-defined accuracy specifications. Most traditional software testing techniques widely used today cannot tackle these challenges because they often assume determinism and require a precise test oracle. This dissertation presents work on automated testing and debugging of ML-based systems and on improving developer-written tests in such systems. To achieve these goals, this dissertation presents principled techniques that build on mathematical foundations from probability theory and statistics to reason about the underlying non-determinism and accuracy. The presented techniques help developers to detect more bugs and to efficiently navigate trade-offs between test quality and efficiency. This dissertation presents contributions along two key directions. First, a key challenge in testing ML-based systems is generating tests that can systematically and effectively explore the input space of the algorithms under test. However, because the inputs to an ML algorithm are complex objects, like a deep neural network model and data, generating inputs that are both syntactically and semantically correct is challenging. Additionally, ML algorithms do not come with well-defined test oracles, which makes it difficult to reason about correctness. This dissertation presents systematic test generation and debugging techniques that tackle these challenges by combining techniques from programming languages, differential testing, and probabilistic reasoning. These techniques have helped detect more than 50 previously unknown bugs in ML libraries and enabled faster debugging of failures. Second, this dissertation presents techniques that improve the quality of regression tests in ML libraries. When writing such tests, developers often fail to adequately account for the randomness of the algorithms under test and rely on guesswork in selecting various test configurations. Consequently, such regression tests often end up being either flaky, i.e., they pass or fail non-deterministically for same code, become expensive to run, or have lower fault-detection effectiveness. This dissertation presents novel test repair techniques that combine principled statistical methods, mathematical optimization, and domain knowledge to systematically tackle these challenges. These techniques have already improved the quality of over 200 tests in over 60 open-source ML libraries, many of which are used at companies like Google, Meta, Microsoft, and Uber as well as in many academic and scientific communities."]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Randomness-aware testing of machine learning-based systems"]}]}],"canonical_facts":{"dc:contributor":["Misailovic, Sasa","Marinov, Darko","Adve, Vikram","Gligoric, Milos","Lahiri, Shuvendu"],"dc:creator":["Dutta, Saikat"],"dc:date":["2023-08","2023-07-05"],"dc:description":["Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2023-12-04 without embargo terms","The student, Saikat Dutta, accepted the attached license on 2023-06-30 at 13:12.","The student, Saikat Dutta, submitted this Dissertation for approval on 2023-06-30 at 13:21.","This Dissertation was approved for publication on 2023-07-05 at 13:56.","DSpace SAF Submission Ingestion Package generated from Vireo submission #19489 on 2023-12-04 at 17:00:22","Machine Learning (ML) is rapidly revolutionizing the way modern-day systems are developed. However, testing ML-based systems is challenging due to 1) the presence of non-determinism, both internal (e.g., stochastic algorithms) and external (e.g., execution environment), and 2) the absence of well-defined accuracy specifications. Most traditional software testing techniques widely used today cannot tackle these challenges because they often assume determinism and require a precise test oracle. This dissertation presents work on automated testing and debugging of ML-based systems and on improving developer-written tests in such systems. To achieve these goals, this dissertation presents principled techniques that build on mathematical foundations from probability theory and statistics to reason about the underlying non-determinism and accuracy. The presented techniques help developers to detect more bugs and to efficiently navigate trade-offs between test quality and efficiency. This dissertation presents contributions along two key directions. First, a key challenge in testing ML-based systems is generating tests that can systematically and effectively explore the input space of the algorithms under test. However, because the inputs to an ML algorithm are complex objects, like a deep neural network model and data, generating inputs that are both syntactically and semantically correct is challenging. Additionally, ML algorithms do not come with well-defined test oracles, which makes it difficult to reason about correctness. This dissertation presents systematic test generation and debugging techniques that tackle these challenges by combining techniques from programming languages, differential testing, and probabilistic reasoning. These techniques have helped detect more than 50 previously unknown bugs in ML libraries and enabled faster debugging of failures. Second, this dissertation presents techniques that improve the quality of regression tests in ML libraries. When writing such tests, developers often fail to adequately account for the randomness of the algorithms under test and rely on guesswork in selecting various test configurations. Consequently, such regression tests often end up being either flaky, i.e., they pass or fail non-deterministically for same code, become expensive to run, or have lower fault-detection effectiveness. This dissertation presents novel test repair techniques that combine principled statistical methods, mathematical optimization, and domain knowledge to systematically tackle these challenges. These techniques have already improved the quality of over 200 tests in over 60 open-source ML libraries, many of which are used at companies like Google, Meta, Microsoft, and Uber as well as in many academic and scientific communities."],"dc:format":["application/pdf"],"dc:identifier":["https://hdl.handle.net/2142/121445"],"dc:language":["en","eng"],"dc:rights":["Copyright 2023 Saikat Dutta"],"dc:subject":["Machine Learning","Software Testing"],"dc:title":["Randomness-aware testing of machine learning-based systems"],"dc:type":["text"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["Dissertation"],"thesis:degree_name":["Ph.D."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:24:57Z"}