{"id":{"repo_id":"texas-state","oai_identifier":"oai:digital.library.txst.edu:10877/24744"},"canonical_url":"https://search.dev.ndltd.org/etd/texas-state/oai:digital.library.txst.edu:10877/24744","repository":{"repo_id":"texas-state","name":"Texas State University","base_url":"https://digital.library.txst.edu/server/oai/request"},"display":{"title":"Quantifying Uncertainty in Model Evaluation","abstract":"As Machine Learning models have quickly evolved in the past decade, ways of measuring their potential haven’t. This proposal will pose that simple point estimate performance indicators are ill suited to describe models that exhibit inherent variability. Addressing classifier performance variability is an uncomfortable truth that often complicates model evaluation and comparison, which makes it a convenient step to skip if the opportunity is presented. While industry cutting-edge implementations have internal ways to account for model assessment and uncertainty estimation, the academic community hasn’t reached final consensus for a widely used standard. Through bridging the gap between post-hoc and resampling-based frameworks, this proposal seeks to offer validated approach to estimating uncertainty with a particular emphasis on capturing population-level behavior or in other words, the model’s true potential. Alongside the validated theoretical framework, a software implementation of tools to reproduce the uncertainty estimation routine will be published to further incentivize the adoption and use of the work.","abstract_html":"As Machine Learning models have quickly evolved in the past decade, ways of measuring their potential haven’t. This proposal will pose that simple point estimate performance indicators are ill suited to describe models that exhibit inherent variability. Addressing classifier performance variability is an uncomfortable truth that often complicates model evaluation and comparison, which makes it a convenient step to skip if the opportunity is presented. While industry cutting-edge implementations have internal ways to account for model assessment and uncertainty estimation, the academic community hasn’t reached final consensus for a widely used standard. Through bridging the gap between post-hoc and resampling-based frameworks, this proposal seeks to offer validated approach to estimating uncertainty with a particular emphasis on capturing population-level behavior or in other words, the model’s true potential. Alongside the validated theoretical framework, a software implementation of tools to reproduce the uncertainty estimation routine will be published to further incentivize the adoption and use of the work.","abstract_has_math":false,"creators":["Liron, Amir"],"institution":"Texas State University","degree_name":"Master of Science","degree_level":"Masters","degree_discipline":"Engineering","degree_department":null,"school":null,"contributors":[],"advisors":["Jimenez, Jesus A."],"committee_chairs":[],"committee_members":["Mendez, Francis A.","Dutta, Anandi K."],"year":2026,"date_issued":"2026-05","date_published":"2026-05","updated_at":"2026-07-27T21:22:43Z","subjects":["uncertainty quantification","classifier evaluation","Bayesian inference","confusion matrix"],"languages":["en"],"rights":[],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://hdl.handle.net/10877/24744","outbound_label":"Handle","outbound_source":"dc:identifier.uri"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor.advisor","label":"Advisor","values":["Jimenez, Jesus A."]},{"key":"dc:contributor.committeemember","label":"Committee Member","values":["Mendez, Francis A.","Dutta, Anandi K."]},{"key":"dc:creator","label":"Author","values":["Liron, Amir"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.accessioned","label":"Dc Date Accessioned","values":["2026-05-07T19:03:45Z"]},{"key":"dc:date.issued","label":"Date","values":["2026-05"]},{"key":"dc:type","label":"Dc Type","values":["Thesis"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Engineering"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Masters"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Master of Science"]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["Texas State University"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["uncertainty quantification","classifier evaluation","Bayesian inference","confusion matrix"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language.iso","label":"Language (ISO)","values":["en"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier.uri","label":"Identifier URI","values":["https://hdl.handle.net/10877/24744"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["As Machine Learning models have quickly evolved in the past decade, ways of measuring their potential haven’t. This proposal will pose that simple point estimate performance indicators are ill suited to describe models that exhibit inherent variability. Addressing classifier performance variability is an uncomfortable truth that often complicates model evaluation and comparison, which makes it a convenient step to skip if the opportunity is presented. While industry cutting-edge implementations have internal ways to account for model assessment and uncertainty estimation, the academic community hasn’t reached final consensus for a widely used standard. Through bridging the gap between post-hoc and resampling-based frameworks, this proposal seeks to offer validated approach to estimating uncertainty with a particular emphasis on capturing population-level behavior or in other words, the model’s true potential. Alongside the validated theoretical framework, a software implementation of tools to reproduce the uncertainty estimation routine will be published to further incentivize the adoption and use of the work."]},{"key":"dc:format","label":"Dc Format","values":["Text"]},{"key":"dc:format.medium","label":"Dc Format Medium","values":["1 file (.pdf)"]},{"key":"dc:title","label":"Title","values":["Quantifying Uncertainty in Model Evaluation"]}]}],"canonical_facts":{"dc:contributor.advisor":["Jimenez, Jesus A."],"dc:contributor.committeemember":["Mendez, Francis A.","Dutta, Anandi K."],"dc:creator":["Liron, Amir"],"dc:date.accessioned":["2026-05-07T19:03:45Z"],"dc:date.issued":["2026-05"],"dc:description.abstract":["As Machine Learning models have quickly evolved in the past decade, ways of measuring their potential haven’t. This proposal will pose that simple point estimate performance indicators are ill suited to describe models that exhibit inherent variability. Addressing classifier performance variability is an uncomfortable truth that often complicates model evaluation and comparison, which makes it a convenient step to skip if the opportunity is presented. While industry cutting-edge implementations have internal ways to account for model assessment and uncertainty estimation, the academic community hasn’t reached final consensus for a widely used standard. Through bridging the gap between post-hoc and resampling-based frameworks, this proposal seeks to offer validated approach to estimating uncertainty with a particular emphasis on capturing population-level behavior or in other words, the model’s true potential. Alongside the validated theoretical framework, a software implementation of tools to reproduce the uncertainty estimation routine will be published to further incentivize the adoption and use of the work."],"dc:format":["Text"],"dc:format.medium":["1 file (.pdf)"],"dc:identifier.uri":["https://hdl.handle.net/10877/24744"],"dc:language.iso":["en"],"dc:subject":["uncertainty quantification","classifier evaluation","Bayesian inference","confusion matrix"],"dc:title":["Quantifying Uncertainty in Model Evaluation"],"dc:type":["Thesis"],"thesis:degree_discipline":["Engineering"],"thesis:degree_level":["Masters"],"thesis:degree_name":["Master of Science"],"thesis:institution_name":["Texas State University"]},"updated_at":"2026-07-27T21:22:43Z"}