Interpreting and Editing Memory in Large Transformer Language Models
type: CreativeWork
description: This thesis investigates the mechanisms of factual recall in large language models. We first apply causal interventions to identify neuron activations that are decisive in a model’s factual predictions; surprisingly, we find that factual recall corresponds to a sparse, localizable computation in the MLP weights of the GPT models we study. Harnessing this insight, we then develop methods for efficiently and surgically inserting up to 10,000 new memories into a transformer; these methods perform well in terms of both generalization and specificity. We conclude with some directions for future work.
abstract: This thesis investigates the mechanisms of factual recall in large language models. We first apply causal interventions to identify neuron activations that are decisive in a model’s factual predictions; surprisingly, we find that factual recall corresponds to a sparse, localizable computation in the MLP weights of the GPT models we study. Harnessing this insight, we then develop methods for efficiently and surgically inserting up to 10,000 new memories into a transformer; these methods perform well in terms of both generalization and specificity. We conclude with some directions for future work.
author: Meng, Kevin
license: https://creativecommons.org/licenses/by-nc-nd/4.0/
date published: 2024-05
date modified: 2026-07-22T22:20:53Z
provider: MIT
publisher: Networked Digital Library of Theses and Dissertations
type: WebPage
source organization: Massachusetts Institute of Technology

PropertyValue
property ID: Handle
value: https://hdl.handle.net/1721.1/156794