{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/113902"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/113902","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Classification performance metric elicitation and its applications","abstract":"Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2022-04-06 without embargo terms","abstract_html":"Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2022-04-06 without embargo terms","abstract_has_math":false,"creators":["Hiranandani, Gaurush"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"Ph.D.","degree_level":"Dissertation","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":["Koyejo, Oluwasanmi","Rayadurgam, Srikant","Smaragdis, Paris","Agarwal, Shivani"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2022,"date_issued":"2022-04-29T21:34:49Z","date_published":"2022-04-29T21:34:49Z","updated_at":"2026-07-22T22:24:53Z","subjects":["Computer science"],"languages":["en","eng"],"rights":["Copyright 2021 Gaurush Hiranandani"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"http://hdl.handle.net/2142/113902","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Koyejo, Oluwasanmi","Rayadurgam, Srikant","Smaragdis, Paris","Agarwal, Shivani"]},{"key":"dc:creator","label":"Author","values":["Hiranandani, Gaurush"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2022-04-29T21:34:49Z","2021-12","2021-12-02"]},{"key":"dc:type","label":"Dc Type","values":["text","Thesis"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Dissertation"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Ph.D."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Computer science"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en","eng"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2021 Gaurush Hiranandani"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["http://hdl.handle.net/2142/113902"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2022-04-06 without embargo terms","The student, Gaurush Hiranandani, accepted the attached license on 2021-12-02 at 09:55.","The student, Gaurush Hiranandani, submitted this Dissertation for approval on 2021-12-02 at 10:08.","This Dissertation was approved for publication on 2021-12-02 at 12:36.","DSpace SAF Submission Ingestion Package generated from Vireo submission #17357 on 2022-04-06 at 17:10:46","Made available in DSpace on 2022-04-29T21:34:49Z (GMT). No. of bitstreams: 3 HIRANANDANI-DISSERTATION-2021.pdf: 5645203 bytes, checksum: 2f60a3250fa788b5a275d2deafde036d (MD5) LICENSE.txt: 4216 bytes, checksum: 0480f8fc93914c2216c8bb082da40e42 (MD5) PROQUEST_LICENSE.txt: 4562 bytes, checksum: d895b2e4abda84d1888ea0cd9fd2ec0d (MD5) Previous issue date: 2021-12-02","Given a learning problem with real-world tradeoffs, which cost function should the model be trained to optimize? This is the metric selection problem in machine learning. Despite its practical interest, there is limited formal guidance on how to select metrics for machine learning applications. This thesis outlines metric elicitation as a principled framework for selecting the performance metric that best reflects implicit user preferences. Once specified, the evaluation metric can be used to compare and train models. In this manuscript, we formalize the problem of Metric Elicitation and devise novel strategies for eliciting classification performance metrics using pairwise preference feedback over classifiers. Specifically, we provide novel strategies for eliciting linear and linear-fractional metrics for binary and multiclass classification problems, which are then extended to a framework that elicits group-fair performance metrics in the presence of multiple sensitive groups. All the elicitation strategies that we discuss are robust to both finite sample and feedback noise, thus are useful in practice for real-world applications. Using the tools and the geometric characterizations of the feasible confusion statistics space from the binary, multiclass, and multiclass-multigroup classification setups, we further provide strategies to elicit from a wider range of complex, modern multiclass metrics defined by quadratic functions of predictive rates by exploiting their local linear structure. This strategy can then be easily extended to eliciting metrics of higher order polynomials. From application perspective, we also propose to use the metric elicitation framework in optimizing complex black box metrics that is amenable to deep network training. In particular, the linear elicitation strategies can be used to elicit local-linear approximation of the black-box metrics, which are then exploited by existing iterative optimization routines. Lastly, to bring theory closer to practice, we conduct a preliminary real-user study that shows the efficacy of the metric elicitation framework in recovering the users' preferred performance metric in a binary classification setup."]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Classification performance metric elicitation and its applications"]}]}],"canonical_facts":{"dc:contributor":["Koyejo, Oluwasanmi","Rayadurgam, Srikant","Smaragdis, Paris","Agarwal, Shivani"],"dc:creator":["Hiranandani, Gaurush"],"dc:date":["2022-04-29T21:34:49Z","2021-12","2021-12-02"],"dc:description":["Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2022-04-06 without embargo terms","The student, Gaurush Hiranandani, accepted the attached license on 2021-12-02 at 09:55.","The student, Gaurush Hiranandani, submitted this Dissertation for approval on 2021-12-02 at 10:08.","This Dissertation was approved for publication on 2021-12-02 at 12:36.","DSpace SAF Submission Ingestion Package generated from Vireo submission #17357 on 2022-04-06 at 17:10:46","Made available in DSpace on 2022-04-29T21:34:49Z (GMT). No. of bitstreams: 3 HIRANANDANI-DISSERTATION-2021.pdf: 5645203 bytes, checksum: 2f60a3250fa788b5a275d2deafde036d (MD5) LICENSE.txt: 4216 bytes, checksum: 0480f8fc93914c2216c8bb082da40e42 (MD5) PROQUEST_LICENSE.txt: 4562 bytes, checksum: d895b2e4abda84d1888ea0cd9fd2ec0d (MD5) Previous issue date: 2021-12-02","Given a learning problem with real-world tradeoffs, which cost function should the model be trained to optimize? This is the metric selection problem in machine learning. Despite its practical interest, there is limited formal guidance on how to select metrics for machine learning applications. This thesis outlines metric elicitation as a principled framework for selecting the performance metric that best reflects implicit user preferences. Once specified, the evaluation metric can be used to compare and train models. In this manuscript, we formalize the problem of Metric Elicitation and devise novel strategies for eliciting classification performance metrics using pairwise preference feedback over classifiers. Specifically, we provide novel strategies for eliciting linear and linear-fractional metrics for binary and multiclass classification problems, which are then extended to a framework that elicits group-fair performance metrics in the presence of multiple sensitive groups. All the elicitation strategies that we discuss are robust to both finite sample and feedback noise, thus are useful in practice for real-world applications. Using the tools and the geometric characterizations of the feasible confusion statistics space from the binary, multiclass, and multiclass-multigroup classification setups, we further provide strategies to elicit from a wider range of complex, modern multiclass metrics defined by quadratic functions of predictive rates by exploiting their local linear structure. This strategy can then be easily extended to eliciting metrics of higher order polynomials. From application perspective, we also propose to use the metric elicitation framework in optimizing complex black box metrics that is amenable to deep network training. In particular, the linear elicitation strategies can be used to elicit local-linear approximation of the black-box metrics, which are then exploited by existing iterative optimization routines. Lastly, to bring theory closer to practice, we conduct a preliminary real-user study that shows the efficacy of the metric elicitation framework in recovering the users' preferred performance metric in a binary classification setup."],"dc:format":["application/pdf"],"dc:identifier":["http://hdl.handle.net/2142/113902"],"dc:language":["en","eng"],"dc:rights":["Copyright 2021 Gaurush Hiranandani"],"dc:subject":["Computer science"],"dc:title":["Classification performance metric elicitation and its applications"],"dc:type":["text","Thesis"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["Dissertation"],"thesis:degree_name":["Ph.D."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:24:53Z"}