Learning structured representations with hyperbolic embeddings

Most real-world data consists of a natural hierarchy or an inherent label structure that is either already available or can be constructed/inferred cheaply. However, majority of the existing models for representation learning either completely ignore this hierarchy, treating the labels as permutation invariant, or attempt to utilize this information using less desirable distance metrics with bounded dimensionality. This leads to distortion of the semantic context in the label hierarchy, and also adversely affect it's performance on the in-distribution (ID) classification task. In addition, in the context of real-world machine learning systems, Out-of-distribution (OOD) detection is an even more challenging and critical task to ensure reliability of the deployed models. However, the current distance-based approaches do not consider any structured knowledge and rely on a distance measurement from the ID cluster-centroids, learnt in a label invariant fashion. To approach these challenges, in this thesis, we propose using hyperbolic geometry for incorporating this rich structured hierarchy about the label space into the representation learning. We demonstrate that accurately embedding the label information can lead to more fine-grained learning of structure-informed features, which are discriminative and helpful for a variety of tasks. For this purpose, we propose a novel method HypCPCC: Hyperbolic Cophenetic Correlation Coefficient to embed the label hierarchy into features using a powerful hyperbolic geometry based tree regularization objective. Our proposed objective can easily be combined and optimized with any classification loss for improving representation learning. We also empirically demonstrate that HypCPCC accurately embeds the hierarchical relationships between the labels in web-scale real-world vision datasets and leads to learning semantically rich features that result in simultaneous improvements in the performance of both in-distribution (ID) classification tasks (upto 1%) and AUROC on Out-of-Distribution (OOD) detection tasks (upto 4%). Motivated by the expressiveness of hyperbolic geometry in embedding the tree-based structures, we also propose two principled non-parametric hyperbolic-distance based OOD detection scores: HypKNN+ and HypDist-O. We demonstrate that using these scores can lead to improvements in OOD detection FPR95 upto 2%. Finally, we also empirically show that the learnt features from our proposed methodology are geometrically and semantically more interpretable using hyperbolic visualizations, paving the way for explainable feature learning using hierarchical label information.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2024

Author dc:creator

• Sinha, Aditya

Contributors dc:contributor

• Zhao, Han

• Representation Learning
• Hyperbolic Geometry
• Hierarchical Labels
• Ood Detection

Statement dc:rights

• Copyright 2024 Aditya Sinha

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/124722

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F124722