AI Papers of the Week
Every paper worth reading in AI, hand-picked one week at a time.
Demystifying RL in Agentic Reasoning
This paper studies what actually works when using RL to improve tool-using LLM agents, across three axes: data, algorithm, and reasoning mode. The team contributes a real end-to-end SFT dataset, a diverse RL set, and a compact 4B agent that beats larger models on agentic benchmarks.
Emergent Coordination in Multi-Agent LLMs
A neat, information-theoretic probe for “is this just a pile of agents or a real collective?” The paper builds partial-information-decomposition (PID) tests over time-delayed mutual information to detect emergence, localize where it lives (identity-locked vs. mere temporal coupling), and tie it to performance. Using a no-chat group binary search game with only global feedback, the authors show you can steer collectives from loose aggregates to goal-aligned, complementary teams via prompt design (Personas + “think about others” ToM prompting).
LLMs Can Get “Brain Rot”!
The authors test a clear hypothesis: continual pretraining on trivial, highly engaging web text degrades LLM cognition in ways that persist even after mitigation. They build controlled Twitter datasets to isolate data quality from scale and training ops, then measure effects on reasoning, long-context, safety, and personality.
Hybrid Reinforcement
HERO (Hybrid Ensemble Reward Optimization) is a reinforcement learning framework that combines binary verifier feedback with continuous reward-model signals to improve LLM reasoning. By using stratified normalization and variance-aware weighting, HERO balances correctness and nuance, outperforming verifier-only and RM-only methods on diverse math reasoning benchmarks and enhancing performance on both verifiable and ambiguous tasks.
Kimi-Dev
Kimi-Dev introduces agentless training as a skill prior to software engineering LLMs, bridging workflow-style and agentic paradigms. Trained with structured, verifiable single-turn tasks, it achieves 60.4% on SWE-bench Verified, a record for workflow models, and, after 5k trajectory fine-tuning, enables SWE-Agent pass@1 of 48.6%, rivaling Claude 3.5 Sonnet. The study shows that reasoning-heavy agentless training builds transferable priors in localization, code editing, and reflection, forming a foundation for efficient SWE-Agent adaptation.
Tiny Recursive Model
A simple, data-efficient alternative to the hierarchical hearoning model (HRM) that uses a single tiny 2-layer network to iteratively refine a latent state and the predicted answer. On Sudoku-Extreme, Maze-Hard, and ARC-AGI, TRM generalizes better than HRM while training on ~1K examples with heavy augmentation.
Agentic Context Engineering (ACE)
Presents a modular context-engineering framework that grows and refines an LLM’s working context like a playbook, not a terse prompt. ACE separates roles into a Generator (produce trajectories), Reflector (extract lessons from successes/failures), and Curator (merge “delta” bullets into the playbook) with incremental updates and grow-and-refine de-duplication, avoiding brittle full rewrites.
Reasoning over Longer Horizons via RL
The authors show that you can scale long-horizon reasoning without step labels or heavy scaffolding. They synthesize long problems by chaining easy ones, then train with outcome-only rewards under a length curriculum. The result: large gains on both in-domain chains and harder out-of-domain math and long-context tasks.
The Markovian Thinker
A new RL thinking environment that keeps an LLM’s effective state constant by chunking long chains of thought and carrying over only a short textual state between chunks. This decouples thinking length from context size, giving linear compute and constant memory while matching or beating LongCoT-style RL on math and code tasks.
Abstract Reasoning Composition
UC San Diego and UMD propose ArcMemo, a test-time memory framework that distills reusable concepts from solution traces, stores them in natural language, and retrieves a relevant subset on future queries. Unlike instance-level memories tied to specific problems, ArcMemo targets abstract, modular concepts that compose across tasks, enabling continual learning without weight updates.
Webscale-RL
Webscale-RL introduces a scalable data pipeline that transforms web-scale pretraining text into over 1.2M diverse, verifiable QA pairs for reinforcement learning across 9+ domains. Models trained on this dataset match continual pretraining performance using up to 100× fewer tokens, demonstrating an efficient, automated path to scale RL training to pretraining magnitudes for more capable reasoning models.
DeepSeek-V3.2-Exp
DeepSeek adds a fine-grained sparse attention mechanism (DeepSeek Sparse Attention, DSA) to the V3.1 “Terminus” backbone and shows large cost reductions on 128K context without notable quality loss. Model and inference code are released.
The Era of Real-World Human Interaction
This work presents a post-training recipe that learns directly from real user conversations instead of static annotator labels. RLHI combines user-guided rewrites (using follow-ups as corrections) with persona-based rewards (ranking sampled candidates via a persona-conditioned reward model). Trained on WildChat conversations, it shows strong improvements in personalization, instruction following, and even transfers to reasoning tasks.
DeepSearch
DeepSearch integrates Monte Carlo Tree Search directly into RL with verifiable rewards, but during training rather than only at inference. The result is broader exploration, better credit assignment, and higher sample efficiency on math reasoning vs strong 1.5B baselines.
Reasoning Traces Tailored for Small Models
Small models often get worse when you SFT them on long, high-quality CoT from big teachers. This paper pinpoints why and fixes it with Reverse Speculative Decoding (RSD): let the teacher propose tokens, but let the student approve them only if they are probable under the student. Result: traces that stay correct while matching the student’s distribution, which small models can actually learn from.
Tool-Use Mixture (TUMIX)
TUMIX is an ensemble recipe for reasoning that mixes text, code execution, and web search, running 15 diverse agents in parallel and passing intermediate answers across rounds. An LLM-judge controls early stopping, giving up to +3.55% accuracy gains over strong tool-augmented baselines on HLE, GPQA-Diamond, and AIME 24/25 while cutting inference cost by ~50%.
PrompCoT 2.0
PromptCoT 2.0 introduces an EM-based loop for synthesizing harder and more diverse reasoning prompts, replacing manual heuristics from PromptCoT 1.0. It enables both self-play and SFT training regimes, achieving new SOTA on reasoning benchmarks like AIME, HMMT, LiveCodeBench, and Codeforces, showing prompt synthesis as a new scaling axis for LLM reasoning.
ARE
Metal SuperIntelligence Labs presents a research platform and benchmark for building and stress-testing agent systems in realistic, time-driven environments. The paper introduces a modular simulator (ARE) and a mobile-style benchmark (Gaia2) that emphasize asynchronous events, verification of write actions, and multi-agent coordination in noisy, dynamic settings.
Code World Model
Meta FAIR releases CWM, a 32B open-weights coder trained to model code execution and to act inside containers. It mid-trains on Python interpreter traces and agentic Docker trajectories, then upgrades with multi-turn RL across SWE, coding, and math. CWM is both a strong coder and a testbed for world-model-style reasoning in software environments.
Teaching LLMs to Plan
A training recipe that teaches LLMs to plan in Planning Domain Definition Language (PDDL) by making them write explicit state–action–state chains and checking each step with an external verifier (VAL). The result: big jumps in plan validity on PlanBench domains, especially when feedback explains why an action failed rather than just saying it failed.
ARK-V1
ARK-V1 is a lightweight agent that helps language models answer questions by actively walking through a knowledge graph instead of relying only on memorized text. This is especially useful for long-tail entities (less common stuff) where the model’s pretraining knowledge falls short.
Language Models that Think, Chat Better
A simple recipe, RL with Model-rewarded Thinking, makes small open models “plan first, answer second” on regular chat prompts and trains them with online RL against a preference reward. On Llama-3.1-8B and Qwen-2.5-7B, this consistently beats standard RLHF on chat, creative writing, and general knowledge, with the best 8B model topping some frontier systems on WildBench and AlpacaEval2.
Embodied AI: From LLMs to World Models
This paper surveys embodied AI through the lens of LLMs and World Models (WMs). It highlights how LLMs enable semantic reasoning and task decomposition, while WMs provide predictive, physics-grounded interaction, and argues for a joint MLLM-WM architecture to advance real-world embodied cognition and applications.
GDPval
GDPval is a new benchmark of 1,320 real-world tasks across 44 occupations in 9 major GDP sectors, graded by industry experts with a 220-task gold set. It shows frontier models improve roughly linearly and are nearing expert parity, with Claude Opus 4.1 preferred or tied 47.6% of the time, while GPT-5 leads in accuracy. Model-plus-human workflows can reduce time and cost, and adding reasoning effort and prompt scaffolding further raises scores, with an open gold set and automated grader available for researchers.