Fact Recalling in LLMs
A mechanistic-interpretability study showing that early MLP layers function as a lookup table for factual recall.
Ask this paper
Athletes-to-sports task: Scoped to how Pythia 2.8B recalls which of 3 different sports various athletes play - a clean task for dissecting a single type of factual recall.
Early MLPs as lookup table: Early MLP layers perform a structured lookup rather than distributed reasoning, with specific neurons keyed to entity-attribute pairs.
Multi-token embedding view: Recommends treating factual knowledge recall as operating over multi-token embeddings rather than single-token representations.
Interpretability payoff: Provides a concrete, testable account of where and how facts live inside transformers, enabling targeted editing and auditing of parametric memory.