{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/97479"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/97479","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Multiple-implementation testing of supervised learning software","abstract":"Machine Learning (ML) software, used to implement an ML algorithm, is widely used in many application domains such as financial, business, and engineering domains. Faults in ML software can cause substantial losses in these application domains. Thus, it is very critical to conduct effective testing of ML software to detect and eliminate its faults. However, testing ML software is difficult, especially on producing test oracles used for checking behavior correctness (such as using expected properties or expected test outputs). To tackle the test-oracle issue, this thesis presents a novel black-box approach of multiple-implementation testing for supervised learning software. The insight underlying the approach is that there can be multiple implementations (independently written) for a supervised learning algorithm, and majority of them may produce the expected output for a test input (even if none of these implementations are fault-free). In particular, the proposed approach derives a pseudo oracle for a test input by running the test input on n implementations of the supervised learning algorithm, and then using the common test output produced by a majority (determined by a percentage threshold) of these n implementations. The proposed approach includes techniques to address challenges in multiple-implementation testing (or generally testing) of supervised learning software: the definition of test cases in testing supervised learning software, along with resolution of inconsistent algorithm configurations across implementations. In addition, to improve dependability of supervised learning software during in-field usage while incurring low runtime overhead, The approach includes a multiple-implementation monitoring technique. The evaluations on the proposed approach show that multiple-implementation testing is effective in detecting real faults in real-world ML software (even popularly used ones), including 5 faults from 10 NaiveBayes implementations and 4 faults from 20 k-nearest neighbor implementations, and the proposed technique of multiple-implementation monitoring substantially reduces the need of running multiple implementations with high prediction accuracy.","abstract_html":"Machine Learning (ML) software, used to implement an ML algorithm, is widely used in many application domains such as financial, business, and engineering domains. Faults in ML software can cause substantial losses in these application domains. Thus, it is very critical to conduct effective testing of ML software to detect and eliminate its faults. However, testing ML software is difficult, especially on producing test oracles used for checking behavior correctness (such as using expected properties or expected test outputs). To tackle the test-oracle issue, this thesis presents a novel black-box approach of multiple-implementation testing for supervised learning software. The insight underlying the approach is that there can be multiple implementations (independently written) for a supervised learning algorithm, and majority of them may produce the expected output for a test input (even if none of these implementations are fault-free). In particular, the proposed approach derives a pseudo oracle for a test input by running the test input on n implementations of the supervised learning algorithm, and then using the common test output produced by a majority (determined by a percentage threshold) of these n implementations. The proposed approach includes techniques to address challenges in multiple-implementation testing (or generally testing) of supervised learning software: the definition of test cases in testing supervised learning software, along with resolution of inconsistent algorithm configurations across implementations. In addition, to improve dependability of supervised learning software during in-field usage while incurring low runtime overhead, The approach includes a multiple-implementation monitoring technique. The evaluations on the proposed approach show that multiple-implementation testing is effective in detecting real faults in real-world ML software (even popularly used ones), including 5 faults from 10 NaiveBayes implementations and 4 faults from 20 k-nearest neighbor implementations, and the proposed technique of multiple-implementation monitoring substantially reduces the need of running multiple implementations with high prediction accuracy.","abstract_has_math":false,"creators":["Alebiosu, Oreoluwa"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"M.S.","degree_level":"Thesis","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":["Xie, Tao"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2017,"date_issued":"2017-08-10T19:16:11Z","date_published":"2017-08-10T19:16:11Z","updated_at":"2026-07-22T22:24:34Z","subjects":["Machine learning","Multiple-implementation testing","Differential testing","Supervised learning","Multiple-implementation monitoring","Software monitoring","k-Nearest neighbor (kNN)","Naive Bayes","Software testing","Pseudo oracle","Algorithm configurations","Percentage threshold","Black box","Test oracle","Multiple implementation","NaiveBayes"],"languages":["en"],"rights":["Copyright 2017 Oreoluwa Alebiosu"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"http://hdl.handle.net/2142/97479","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Xie, Tao"]},{"key":"dc:creator","label":"Author","values":["Alebiosu, Oreoluwa"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2017-08-10T19:16:11Z","2017-04-26","2017-05"]},{"key":"dc:type","label":"Dc Type","values":["text"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["M.S."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Machine learning","Multiple-implementation testing","Differential testing","Supervised learning","Multiple-implementation monitoring","Software monitoring","k-Nearest neighbor (kNN)","Naive Bayes","Software testing","Pseudo oracle","Algorithm configurations","Percentage threshold","Black box","Test oracle","Multiple implementation","NaiveBayes"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2017 Oreoluwa Alebiosu"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["http://hdl.handle.net/2142/97479"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Machine Learning (ML) software, used to implement an ML algorithm, is widely used in many application domains such as financial, business, and engineering domains. Faults in ML software can cause substantial losses in these application domains. Thus, it is very critical to conduct effective testing of ML software to detect and eliminate its faults. However, testing ML software is difficult, especially on producing test oracles used for checking behavior correctness (such as using expected properties or expected test outputs). To tackle the test-oracle issue, this thesis presents a novel black-box approach of multiple-implementation testing for supervised learning software. The insight underlying the approach is that there can be multiple implementations (independently written) for a supervised learning algorithm, and majority of them may produce the expected output for a test input (even if none of these implementations are fault-free). In particular, the proposed approach derives a pseudo oracle for a test input by running the test input on n implementations of the supervised learning algorithm, and then using the common test output produced by a majority (determined by a percentage threshold) of these n implementations. The proposed approach includes techniques to address challenges in multiple-implementation testing (or generally testing) of supervised learning software: the definition of test cases in testing supervised learning software, along with resolution of inconsistent algorithm configurations across implementations. In addition, to improve dependability of supervised learning software during in-field usage while incurring low runtime overhead, The approach includes a multiple-implementation monitoring technique. The evaluations on the proposed approach show that multiple-implementation testing is effective in detecting real faults in real-world ML software (even popularly used ones), including 5 faults from 10 NaiveBayes implementations and 4 faults from 20 k-nearest neighbor implementations, and the proposed technique of multiple-implementation monitoring substantially reduces the need of running multiple implementations with high prediction accuracy.","Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2017-08-10 without embargo terms","The student, Oreoluwa Alebiosu, accepted the attached license on 2017-04-25 at 15:33.","The student, Oreoluwa Alebiosu, submitted this Thesis for approval on 2017-04-25 at 15:50.","This Thesis was approved for publication on 2017-04-26 at 12:12.","DSpace SAF Submission Ingestion Package generated from Vireo submission #11055 on 2017-08-10 at 13:46:30","Made available in DSpace on 2017-08-10T19:16:11Z (GMT). No. of bitstreams: 2 ALEBIOSU-THESIS-2017.pdf: 451619 bytes, checksum: f7841641a230f20d6af46d579d53f6bd (MD5) LICENSE.txt: 4214 bytes, checksum: 367b9354493fa10e164197e6568fb813 (MD5) Previous issue date: 2017-04-26"]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Multiple-implementation testing of supervised learning software"]}]}],"canonical_facts":{"dc:contributor":["Xie, Tao"],"dc:creator":["Alebiosu, Oreoluwa"],"dc:date":["2017-08-10T19:16:11Z","2017-04-26","2017-05"],"dc:description":["Machine Learning (ML) software, used to implement an ML algorithm, is widely used in many application domains such as financial, business, and engineering domains. Faults in ML software can cause substantial losses in these application domains. Thus, it is very critical to conduct effective testing of ML software to detect and eliminate its faults. However, testing ML software is difficult, especially on producing test oracles used for checking behavior correctness (such as using expected properties or expected test outputs). To tackle the test-oracle issue, this thesis presents a novel black-box approach of multiple-implementation testing for supervised learning software. The insight underlying the approach is that there can be multiple implementations (independently written) for a supervised learning algorithm, and majority of them may produce the expected output for a test input (even if none of these implementations are fault-free). In particular, the proposed approach derives a pseudo oracle for a test input by running the test input on n implementations of the supervised learning algorithm, and then using the common test output produced by a majority (determined by a percentage threshold) of these n implementations. The proposed approach includes techniques to address challenges in multiple-implementation testing (or generally testing) of supervised learning software: the definition of test cases in testing supervised learning software, along with resolution of inconsistent algorithm configurations across implementations. In addition, to improve dependability of supervised learning software during in-field usage while incurring low runtime overhead, The approach includes a multiple-implementation monitoring technique. The evaluations on the proposed approach show that multiple-implementation testing is effective in detecting real faults in real-world ML software (even popularly used ones), including 5 faults from 10 NaiveBayes implementations and 4 faults from 20 k-nearest neighbor implementations, and the proposed technique of multiple-implementation monitoring substantially reduces the need of running multiple implementations with high prediction accuracy.","Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2017-08-10 without embargo terms","The student, Oreoluwa Alebiosu, accepted the attached license on 2017-04-25 at 15:33.","The student, Oreoluwa Alebiosu, submitted this Thesis for approval on 2017-04-25 at 15:50.","This Thesis was approved for publication on 2017-04-26 at 12:12.","DSpace SAF Submission Ingestion Package generated from Vireo submission #11055 on 2017-08-10 at 13:46:30","Made available in DSpace on 2017-08-10T19:16:11Z (GMT). No. of bitstreams: 2 ALEBIOSU-THESIS-2017.pdf: 451619 bytes, checksum: f7841641a230f20d6af46d579d53f6bd (MD5) LICENSE.txt: 4214 bytes, checksum: 367b9354493fa10e164197e6568fb813 (MD5) Previous issue date: 2017-04-26"],"dc:format":["application/pdf"],"dc:identifier":["http://hdl.handle.net/2142/97479"],"dc:language":["en"],"dc:rights":["Copyright 2017 Oreoluwa Alebiosu"],"dc:subject":["Machine learning","Multiple-implementation testing","Differential testing","Supervised learning","Multiple-implementation monitoring","Software monitoring","k-Nearest neighbor (kNN)","Naive Bayes","Software testing","Pseudo oracle","Algorithm configurations","Percentage threshold","Black box","Test oracle","Multiple implementation","NaiveBayes"],"dc:title":["Multiple-implementation testing of supervised learning software"],"dc:type":["text"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["Thesis"],"thesis:degree_name":["M.S."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:24:34Z"}