Training LLMs with Pause Tokens
Free while signed in. Answers cite the passages they came from.
CMU shows that adding a learnable `<pause>` token during both pretraining and fine-tuning gives the model extra "thinking time" and improves reasoning.
Learnable pause token: Inserts a `<pause>` token into the input; the model processes these tokens but doesn't treat them as meaningful content, letting it compute more before answering.
CommonsenseQA and math gains: Produces measurable performance gains on CommonsenseQA and math word problems - both tasks that benefit from extra internal computation.
Pretraining is required: The benefit only materializes if pauses are introduced in both pretraining and fine-tuning - adding them only at inference doesn't work.
Compute-aware decoding: Positions pause tokens as a simple inference-time knob for trading compute against accuracy, foreshadowing many 2024 "thinking time" tricks.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack