Interpretability of Neural Networks Latent Representations
type: CreativeWork
description: Deep neural networks are complicated objects. They typically involve millions to trillions of operations in order to turn their input data into a prediction. Since it is not possible for a human user to analyze each of these operations, the models appear as black- boxes. The field of interpretability has developed with the aim to make these models more transparent by explaining their predictions. However, the field has historically focused on explaining the output of these models, leaving their internal latent representations unexplored. In this thesis, we describe how interpretability research has recently expanded its scope beyond the output of neural networks. We focus on 3 types of interpretability methods: feature importance, example-based and concept-based explanations. Most feature importance methods require a label to select which component of the neural network output to interpret. When interpreting neural networks representations, components corresponds to neuron activations and there is no clear label to select which component to interpret. Hence, these features importance methods can not be directly applied to interpret latent representations. As a remedy, we introduce the Label-Free Feature Importance framework. In this framework, the feature importance is aggregated over all neurons by weighing each neuron contribution with the neuron activation. We show that this yields actionable explanations of latent representations. Many example-based explanations also implicitly rely on a label, as they require the computation of the model loss for the explained example. To circumvent this limitation, we introduce SimplEx, an example-based interpretability method based on the representation spaces of neural networks. SimplEx reconstructs the latent representation of a given example as a weighted combination of latent representations from a corpus of examples. If this corpus corresponds to the model training data, SimplEx then decomposes the representation of a ne…
abstract: Deep neural networks are complicated objects. They typically involve millions to trillions of operations in order to turn their input data into a prediction. Since it is not possible for a human user to analyze each of these operations, the models appear as black- boxes. The field of interpretability has developed with the aim to make these models more transparent by explaining their predictions. However, the field has historically focused on explaining the output of these models, leaving their internal latent representations unexplored. In this thesis, we describe how interpretability research has recently expanded its scope beyond the output of neural networks. We focus on 3 types of interpretability methods: feature importance, example-based and concept-based explanations. Most feature importance methods require a label to select which component of the neural network output to interpret. When interpreting neural networks representations, components corresponds to neuron activations and there is no clear label to select which component to interpret. Hence, these features importance methods can not be directly applied to interpret latent representations. As a remedy, we introduce the Label-Free Feature Importance framework. In this framework, the feature importance is aggregated over all neurons by weighing each neuron contribution with the neuron activation. We show that this yields actionable explanations of latent representations. Many example-based explanations also implicitly rely on a label, as they require the computation of the model loss for the explained example. To circumvent this limitation, we introduce SimplEx, an example-based interpretability method based on the representation spaces of neural networks. SimplEx reconstructs the latent representation of a given example as a weighted combination of latent representations from a corpus of examples. If this corpus corresponds to the model training data, SimplEx then decomposes the representation of a ne…
author: Crabbé, Jonathan
language: eng
keywords: Artificial Intelligence, Computer Science, Interpretability, Machine Learning, Mathematics
license: https://apollo8-f-pro.lib.cam.ac.uk/bitstreams/abca36f7-01d3-4102-af84-65399f49f0da/download
license: https://creativecommons.org/licenses/by/4.0/
date published: 2024-04-01
date modified: 2026-07-22T22:24:07Z
provider: Cambridge University
publisher: Networked Digital Library of Theses and Dissertations
type: WebPage
source organization: University of Cambridge

PropertyValue
property ID: DOI
value: 10.17863/CAM.116188