System 2 Attention (S2A)
Free while signed in. Answers cite the passages they came from.
Meta's S2A uses the LLM's own reasoning to decide what context actually matters, regenerating a clean prompt before the final response step.
Two-pass prompting: First pass uses the LLM to filter/regenerate the input context, removing irrelevant or misleading content; second pass generates the final answer from the clean context.
Addresses distraction: Directly targets the well-known problem that LLMs attend to irrelevant or manipulative content (e.g., opinion-laden context that biases answers).
Factuality gains: Increases factuality on QA and reduces the model's sensitivity to biased framing or distractors inserted into the prompt.
Math word problems: Outperforms standard attention-based LLMs on math word problems, where filtering irrelevant details is often the hard part of the task.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack