This dissertation proposes researching an approach to incorporate and align Software black-box testing methods into Machine Learning (ML) applications, specifically in the context of computer vision models. Typically, testing methods within Software Engineering (SE) encompass a range of test types that assess levels of a software system, such as Unit, Integration, Functional, and System testing [1]. The testing spectrum offers two perspectives on the system: black-box, where the system’s code is hidden, and white-box, where the system's code is exposed for testing. Software Quality pairs testing with requirements, in a many-to-one relationship, to ensure proper validation of the software system. Within ML, models are often tested in a black-box manner, with the focus on the input-output classification. However, the results are quantified by metrics such as, F-1 score, accuracy, precision, and recall, which do not capture the nuanced behavior of the ML model [2]. Interestingly, ML Non-Functional Requirements (NFR) are a growing research area with ML Functional Requirements (FR) remaining relatively unexplored [3]. This dissertation proposes using the growing ML FR domain to invigorate the testing of ML. The goal of this research is to leverage the expanding functional requirements research to introduce black-box testing methods of equivalence partition testing, decision table testing, and boundary value analysis, to uncover the behavioral qualities of an ML model. This dissertation accomplishes the exploration of ML testing through four case studies, which iteratively examine software engineering-based black-box testing methods and the adjustments required to accommodate the stochastic and non-deterministic nature of ML. The new SE-inspired ML testing strategies were evaluated on their abilities to fulfill requirements, which are a staple for traditional SE-based verification. The results showed that the strategies, by fulfilling requirements linked to specific behaviors, are effective in testing and identifying ML model behaviors in both industry-grade and academic object detection models.
","abstract_html":"<p>This dissertation proposes researching an approach to incorporate and align Software black-box testing methods into Machine Learning (ML) applications, specifically in the context of computer vision models. Typically, testing methods within Software Engineering (SE) encompass a range of test types that assess levels of a software system, such as Unit, Integration, Functional, and System testing [1]. The testing spectrum offers two perspectives on the system: black-box, where the system’s code is hidden, and white-box, where the system's code is exposed for testing. Software Quality pairs testing with requirements, in a many-to-one relationship, to ensure proper validation of the software system. Within ML, models are often tested in a black-box manner, with the focus on the input-output classification. However, the results are quantified by metrics such as, F-1 score, accuracy, precision, and recall, which do not capture the nuanced behavior of the ML model [2]. Interestingly, ML Non-Functional Requirements (NFR) are a growing research area with ML Functional Requirements (FR) remaining relatively unexplored [3]. This dissertation proposes using the growing ML FR domain to invigorate the testing of ML. The goal of this research is to leverage the expanding functional requirements research to introduce black-box testing methods of equivalence partition testing, decision table testing, and boundary value analysis, to uncover the behavioral qualities of an ML model. This dissertation accomplishes the exploration of ML testing through four case studies, which iteratively examine software engineering-based black-box testing methods and the adjustments required to accommodate the stochastic and non-deterministic nature of ML. The new SE-inspired ML testing strategies were evaluated on their abilities to fulfill requirements, which are a staple for traditional SE-based verification. The results showed that the strategies, by fulfilling requirements linked to specific behaviors, are effective in testing and identifying ML model behaviors in both industry-grade and academic object detection models.</p>","abstract_has_math":false,"creators":["Elvira, Timothy"],"institution":null,"degree_name":"Doctor of Philosophy in Electrical Engineering & Computer Science","degree_level":"Dissertation - Open Access","degree_discipline":"Electrical Engineering and Computer Science","degree_department":null,"school":null,"contributors":[],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2025,"date_issued":"2025-04-01T07:00:00Z","date_published":"2025-04-01T07:00:00Z","updated_at":"2026-07-27T19:26:16Z","subjects":["Software Engineering","Machine Learning","Software Testing","Verification","ML Engineering","Black Box testing","ML Testing","Deep Learning","Object detection","Computer Vision","Artificial Intelligence and Robotics","Data Science"],"languages":[],"rights":[],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://commons.erau.edu/edt/886","outbound_label":"Repository record","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:creator","label":"Author","values":["Elvira, Timothy"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"thesis:degree_discipline","label":"Discipline","values":["Electrical Engineering and Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Dissertation - Open Access"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Doctor of Philosophy in Electrical Engineering & Computer Science"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Software Engineering","Machine Learning","Software Testing","Verification","ML Engineering","Black Box testing","ML Testing","Deep Learning","Object detection","Computer Vision","Artificial Intelligence and Robotics","Data Science"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://commons.erau.edu/edt/886"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["This dissertation proposes researching an approach to incorporate and align Software black-box testing methods into Machine Learning (ML) applications, specifically in the context of computer vision models. Typically, testing methods within Software Engineering (SE) encompass a range of test types that assess levels of a software system, such as Unit, Integration, Functional, and System testing [1]. The testing spectrum offers two perspectives on the system: black-box, where the system’s code is hidden, and white-box, where the system's code is exposed for testing. Software Quality pairs testing with requirements, in a many-to-one relationship, to ensure proper validation of the software system. Within ML, models are often tested in a black-box manner, with the focus on the input-output classification. However, the results are quantified by metrics such as, F-1 score, accuracy, precision, and recall, which do not capture the nuanced behavior of the ML model [2]. Interestingly, ML Non-Functional Requirements (NFR) are a growing research area with ML Functional Requirements (FR) remaining relatively unexplored [3]. This dissertation proposes using the growing ML FR domain to invigorate the testing of ML. The goal of this research is to leverage the expanding functional requirements research to introduce black-box testing methods of equivalence partition testing, decision table testing, and boundary value analysis, to uncover the behavioral qualities of an ML model. This dissertation accomplishes the exploration of ML testing through four case studies, which iteratively examine software engineering-based black-box testing methods and the adjustments required to accommodate the stochastic and non-deterministic nature of ML. The new SE-inspired ML testing strategies were evaluated on their abilities to fulfill requirements, which are a staple for traditional SE-based verification. The results showed that the strategies, by fulfilling requirements linked to specific behaviors, are effective in testing and identifying ML model behaviors in both industry-grade and academic object detection models.
"]},{"key":"dc:title","label":"Title","values":["Applying Software Engineering Black-Box Methods for Testing Machine Learning Models"]}]}],"canonical_facts":{"dc:creator":["Elvira, Timothy"],"dc:description.abstract":["This dissertation proposes researching an approach to incorporate and align Software black-box testing methods into Machine Learning (ML) applications, specifically in the context of computer vision models. Typically, testing methods within Software Engineering (SE) encompass a range of test types that assess levels of a software system, such as Unit, Integration, Functional, and System testing [1]. The testing spectrum offers two perspectives on the system: black-box, where the system’s code is hidden, and white-box, where the system's code is exposed for testing. Software Quality pairs testing with requirements, in a many-to-one relationship, to ensure proper validation of the software system. Within ML, models are often tested in a black-box manner, with the focus on the input-output classification. However, the results are quantified by metrics such as, F-1 score, accuracy, precision, and recall, which do not capture the nuanced behavior of the ML model [2]. Interestingly, ML Non-Functional Requirements (NFR) are a growing research area with ML Functional Requirements (FR) remaining relatively unexplored [3]. This dissertation proposes using the growing ML FR domain to invigorate the testing of ML. The goal of this research is to leverage the expanding functional requirements research to introduce black-box testing methods of equivalence partition testing, decision table testing, and boundary value analysis, to uncover the behavioral qualities of an ML model. This dissertation accomplishes the exploration of ML testing through four case studies, which iteratively examine software engineering-based black-box testing methods and the adjustments required to accommodate the stochastic and non-deterministic nature of ML. The new SE-inspired ML testing strategies were evaluated on their abilities to fulfill requirements, which are a staple for traditional SE-based verification. The results showed that the strategies, by fulfilling requirements linked to specific behaviors, are effective in testing and identifying ML model behaviors in both industry-grade and academic object detection models.
"],"dc:identifier":["https://commons.erau.edu/edt/886"],"dc:subject":["Software Engineering","Machine Learning","Software Testing","Verification","ML Engineering","Black Box testing","ML Testing","Deep Learning","Object detection","Computer Vision","Artificial Intelligence and Robotics","Data Science"],"dc:title":["Applying Software Engineering Black-Box Methods for Testing Machine Learning Models"],"thesis:degree_discipline":["Electrical Engineering and Computer Science"],"thesis:degree_level":["Dissertation - Open Access"],"thesis:degree_name":["Doctor of Philosophy in Electrical Engineering & Computer Science"]},"updated_at":"2026-07-27T19:26:16Z"}