Fact Recalling in LLMs
Free while signed in. Answers cite the passages they came from.
A mechanistic-interpretability study showing that early MLP layers function as a lookup table for factual recall.
Athletes-to-sports task: Scoped to how Pythia 2.8B recalls which of 3 different sports various athletes play - a clean task for dissecting a single type of factual recall.
Early MLPs as lookup table: Early MLP layers perform a structured lookup rather than distributed reasoning, with specific neurons keyed to entity-attribute pairs.
Multi-token embedding view: Recommends treating factual knowledge recall as operating over multi-token embeddings rather than single-token representations.
Interpretability payoff: Provides a concrete, testable account of where and how facts live inside transformers, enabling targeted editing and auditing of parametric memory.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack