Meta-learning representations with relational structure

Representation learning has emerged as a versatile tool that is able to take advantage of the vast datasets acquired using digital technologies. The broad applicability of this method stems from its flexibility in use as a subsystem and malleability in incorporating priors in model architectures. Intuitive dependencies within data, such as pixels mostly contributing to the context of their neighbours, may be formalised and embedded to improve generalisation and allow models with great capacity to avoid overfitting. Meta-learning has also been applied to extend these systems to low data settings without loss of performance by viewing specific tasks as realisations of more general problems. This dissertation considers how we may exploit the essential compatibility of these approaches. The primary thesis of this work is that the articulation of computation provided by inductive biases may be used both to improve meta-learning architectures and to directly structure the transfer of past experience and problem solving abilities of meta-learners to new tasks. The methods developed in fusing these approaches are shown to improve performance over baseline models in a broad range of settings and domains. Fusion is realised in three ways. The first identifies polythetic classification as a natural setting and shows how the self-organisation of datapoints under attention can be used to empower meta-learning classifiers. The second uses explicit relational inference to modulate and recombine neural modules for fast and accurate adaptation at test time. And finally, adapting Neural Processes to capture relational and temporal dependencies is shown to improve the accuracy and coherency of predictions and uncertainty estimates. In validating the motivating hypothesis of this dissertation, these contributions find state-of-the-art applications in, among other areas, few-shot image classification, the unsupervised recovery of interactions governing systems of particles, protein-protein interaction site prediction, and the identification and evolution of dynamical systems. In doing so, this work contributes to the effort to bring machine intelligence to bear on an ever broader and finer range of problems — as solutions to the problems considered, as architectural templates for further applications, and as a direction for future research.

Name dc:type.qualificationname

Doctor of Philosophy (PhD)

Level dc:type.qualificationlevel

Doctoral

Grantor dc:publisher.institution

University of Cambridge

Year dc:date.issued

2022

Author dc:creator

• Day, Benjamin

Advisor dc:contributor.advisor

• Lio, Pietro

• machine learning
• representation learning
• meta-learning
• geometric deep learning
• graph neural networks
• neural processes
• neural ordinary differential equations
• few-shot classification

Licence

• https://creativecommons.org/licenses/by-sa/4.0/

Language dc:language

eng

DOI dc:identifier.doi

https://doi.org/10.17863/CAM.93952

OAI identifier oai:identifier

oai:www.repository.cam.ac.uk:1810/346536