Training

Symbol tuning

First page

Paper summary

Fine-tunes LMs on in-context input-label pairs with natural-language labels replaced by arbitrary symbols.

Ask this paper

Key points

Symbolic abstraction: Replacing semantic labels with random symbols forces the model to rely on the demonstrations rather than label priors.

ICL improvements: Boosts performance on unseen in-context learning tasks where the model must infer label semantics from examples.

Algorithmic reasoning: Particularly improves algorithmic reasoning tasks that require following abstract patterns.

ICL mechanism insight: Provides evidence about how ICL works and how to train models that better generalize the mechanism.

Every Monday