Quantifying Uncertainty in Model Evaluation

As Machine Learning models have quickly evolved in the past decade, ways of measuring their potential haven’t. This proposal will pose that simple point estimate performance indicators are ill suited to describe models that exhibit inherent variability. Addressing classifier performance variability is an uncomfortable truth that often complicates model evaluation and comparison, which makes it a convenient step to skip if the opportunity is presented. While industry cutting-edge implementations have internal ways to account for model assessment and uncertainty estimation, the academic community hasn’t reached final consensus for a widely used standard. Through bridging the gap between post-hoc and resampling-based frameworks, this proposal seeks to offer validated approach to estimating uncertainty with a particular emphasis on capturing population-level behavior or in other words, the model’s true potential. Alongside the validated theoretical framework, a software implementation of tools to reproduce the uncertainty estimation routine will be published to further incentivize the adoption and use of the work.

Name thesis:degree_name

Master of Science

Level thesis:degree_level

Masters

Discipline thesis:degree_discipline

Engineering

Grantor

Texas State University

Year dc:date.issued

2026

Author dc:creator

• Liron, Amir

Advisor dc:contributor.advisor

• Jimenez, Jesus A.

Committee members dc:contributor.committeemember

• Mendez, Francis A.
• Dutta, Anandi K.

• uncertainty quantification
• classifier evaluation
• Bayesian inference
• confusion matrix

Language dc:language.iso

en

Handle dc:identifier.uri

https://hdl.handle.net/10877/24744

OAI identifier oai:identifier

oai:digital.library.txst.edu:10877/24744