Learning without Training

Context as implicit weight update: The authors prove that for contextual blocks, the presence of a prompt modifies the neural network’s behavior equivalently to a rank-1 update of its weight matrix. This holds even without modifying the self-attention layer, highlighting that ICL may primarily emerge from how context affects the MLP weights.

Derived update formula: They provide an explicit expression for the rank-1 update to the MLP weights in terms of the context and input token embeddings. The result holds both for standard blocks and for transformer blocks with skip-connections.

ICL as gradient descent: Iteratively consuming tokens from the prompt induces a learning dynamic akin to online gradient descent. Each token incrementally alters the MLP weights in a way that mimics updates on a loss function defined over the prompt sequence.

Empirical validation: Using a synthetic task (learning linear functions), the authors show that a trained transformer’s prediction with a context is identical to the prediction from the same model without the context but with MLP weights updated via the derived ∆W formula. The loss curves for both methods match almost exactly, and the gradient updates shrink over time, indicating convergence.

Comparison to fine-tuning: A side-by-side comparison shows that the implicit weight updates from ICL mirror the effect of actual fine-tuning on the same data, though not identically. Both methods reduce loss on a test query as more examples are consumed, suggesting that ICL may serve as a form of “fast weights” mechanism.