System 2 Attention (S2A)
Meta's S2A uses the LLM's own reasoning to decide what context actually matters, regenerating a clean prompt before the final response step.
Ask this paper
Two-pass prompting: First pass uses the LLM to filter/regenerate the input context, removing irrelevant or misleading content; second pass generates the final answer from the clean context.
Addresses distraction: Directly targets the well-known problem that LLMs attend to irrelevant or manipulative content (e.g., opinion-laden context that biases answers).
Factuality gains: Increases factuality on QA and reduces the model's sensitivity to biased framing or distractors inserted into the prompt.
Math word problems: Outperforms standard attention-based LLMs on math word problems, where filtering irrelevant details is often the hard part of the task.