Learning to Reason and Memorize with Self-Notes

Self-note generation: The model can pause processing input and generate explicit reasoning or memory notes in-stream.

On-the-fly recall: Enables the LM to recall past information and perform reasoning when needed, not just in dedicated thinking phases.

Length generalization: Scales better to longer sequences unseen during training than plain reasoning approaches.

Scratchpad precursor: An intellectual precursor to 2024's reasoning models like o1 that produce long internal thinking traces.