Probabilistic performance metric elicitation

Metric elicitation is a type of inverse decision problem where the goal is to learn a loss function for classification using expert comparisons between candidate classifiers. However, for many practical tasks, such an expert can be noisy. Here we present a unified approach for learning metrics robust to constant and location-dependent noise models. Our approach takes advantage of the problem's similarity to probabilistic bisection search and uses pairwise comparisons to update a pseudo-belief distribution for the performance metric. Our theoretical results guarantee convergence in practical settings and extend beyond previous results to include multi-expert elicitation. Quantitative comparisons against existing methods for performance metric elicitation and inverse decision theory demonstrate the advantage of our approach.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2022

Author dc:creator

• Robertson, Zachary

Contributors dc:contributor

• Koyejo, Oluwasanmi

• Machine Learning
• Active Learning
• Metric Selection

Statement dc:rights

• Copyright 2022 Zachary Robertson

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/115425

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F115425