AI Papers of the Week
Every paper worth reading in AI, hand-picked one week at a time.
Discover and explore top AI papers with Claude Code or Codex
npx @dair-ai/mcp setup
LongCLI-Bench
LongCLI-Bench benchmarks how well AI agents handle complex, extended tasks through command-line interfaces. Across 20 demanding tasks spanning initial development, feature expansion, error resolution, and code optimization, leading agents succeed less than 20% of the time. The study finds that most failures occur early in task execution, and human-agent collaboration through plan injection and interactive guidance yields significantly greater improvements than automated self-correction alone.
Gaia2
Meta FAIR introduces Gaia2, a next-generation agent benchmark where environments change independently of agent actions, forcing agents to handle temporal pressure, uncertainty, and multi-agent coordination. GPT-5 leads at 42% pass@1 but struggles with time-constrained tasks, while Kimi-K2 leads open-source models at 21%. Built on the open-source Agents Research Environments (ARE) platform with action-level verifiers, Gaia2 represents a paradigm shift from static benchmarks to dynamic evaluation of agentic capabilities.
Semi-Autonomous Mathematics Discovery with Gemini
This paper from Google DeepMind presents a case study in semi-autonomous mathematics discovery using Aletheia, a specialized math research agent built on Gemini Deep Think. The team systematically evaluated 700 open conjectures from Bloom’s Erdos Problems database, combining AI-driven natural language verification with human expert evaluation, and addressed 13 previously open problems.
OpenScholar
OpenScholar is a fully open, retrieval-augmented language model designed for scientific literature synthesis. It retrieves passages from a datastore of 45 million open-access papers, generates citation-backed responses, and iteratively refines outputs through a self-feedback loop. On ScholarQABench, the first large-scale multi-domain benchmark for literature search, OpenScholar-8B outperforms GPT-4o by 6% and PaperQA2 by 5.5% in correctness, while achieving citation accuracy on par with human experts and being preferred over expert-written answers 51-70% of the time.
How AI Impacts Skill Formation
Researchers from Anthropic conducted randomized experiments to study how AI assistance affects the development of software engineering skills. They find that using AI to complete coding tasks with a new Python library significantly impaired conceptual understanding, code reading, and debugging abilities - without delivering significant efficiency gains on average.
Memory Control for Long-Horizon Agents
This paper introduces the Agent Cognitive Compressor (ACC), a bio-inspired mechanism that addresses degraded agent behavior in long multi-turn workflows caused by loss of constraint focus, error accumulation, and memory-induced drift. ACC replaces continuous transcript retention with a bounded internal state that updates incrementally during each interaction turn.
Benchmarking Agents on Hard CLI Tasks
Terminal-Bench 2.0 presents a carefully curated hard benchmark composed of 89 tasks in computer terminal environments inspired by problems from real workflows. Each task features a unique environment, human-written solution, and comprehensive tests for verification, addressing the gap where current benchmarks either don’t measure real-world tasks or aren’t sufficiently difficult.
Rethinking Multi-Agent Workflows
This paper challenges the assumption that complex tasks require multiple specialized AI agents, demonstrating that a single LLM agent, through iterative dialogue, can match the performance of homogeneous multi-agent workflows while gaining efficiency from KV cache reuse.
AI IDEs vs Autonomous Agents
This empirical study investigates how LLM-based coding agents that autonomously generate and merge pull requests affect open-source projects compared to IDE-integrated AI assistants. Using longitudinal causal analysis with matched controls, the researchers measure development velocity and software quality outcomes.
Agent-as-a-Judge
This comprehensive survey traces the evolution from LLM-based evaluation to agentic evaluation approaches, establishing the first taxonomy for this paradigm shift. As evaluation tasks grow more intricate and specialized, traditional single-pass language model judges become insufficient.
Adversarial Program Evolution with LLMs
Digital Red Queen (DRQ) introduces an algorithm where LLMs evolve assembly-like programs called “warriors” that compete for control of a virtual machine in the game of Core War. Rather than optimizing toward static objectives, DRQ embraces “Red Queen” dynamics where goals continually shift based on competition, demonstrating how adversarial self-play can drive the evolution of increasingly sophisticated programs.
Training AI Co-Scientists Using Rubric Rewards
This paper from Meta Superintelligence Labs presents a scalable method to train language models to generate better research plans without expensive human supervision or real-world execution. The approach automatically extracts research goals and goal-specific grading rubrics from scientific papers, then uses reinforcement learning with self-grading to improve plan generation.
SWE-EVO
SWE-EVO introduces a benchmark for evaluating coding agents on long-horizon software evolution tasks that require multi-step modifications spanning an average of 21 files per task. The benchmark reveals significant limitations of current agents: GPT-5 with OpenHands achieves only 21% on SWE-EVO compared to 65% on SWE-Bench Verified, highlighting the gap between isolated bug fixes and realistic software development scenarios.
Geometric Memory in Sequence Models
This paper identifies a dramatically different form of how deep sequence models store factual information called geometric memory, contrasting with the traditional associative memory view. Models synthesize embeddings encoding global relationships between all entities, even ones that never co-occur in training.
Monitoring Monitorability
OpenAI introduces a framework for measuring how well we can detect misbehavior in AI systems by monitoring their chain-of-thought reasoning. The paper proposes three evaluation archetypes and a new metric (g-mean2) to track monitorability across different models and training regimes.
Empirical Study of Agent Developer Practices
This paper presents the first large-scale empirical study of LLM-based agent frameworks, analyzing 11,910 developer discussions across ten popular frameworks. The research identifies practical challenges developers face and evaluates how well current frameworks meet their needs.
Detailed Balance in LLM Agents
Researchers establish the first macroscopic physical law in LLM generation dynamics by applying the least action principle to analyze LLM-agent behavior. They discover statistical evidence of detailed balance in state transitions, suggesting LLMs implicitly learn underlying potential functions rather than explicit rules.
FrontierScience
OpenAI introduced FrontierScience, a new benchmark measuring AI capabilities for expert-level scientific reasoning across physics, chemistry, and biology. The benchmark consists of over 700 questions created and verified by domain experts, including international olympiad medalists and PhD scientists.
FACTS Leaderboard
Google introduces the FACTS Leaderboard, a comprehensive benchmark suite for evaluating LLM factuality across diverse scenarios. The leaderboard aggregates performance across four specialized sub-benchmarks to provide a holistic measure of how accurately models generate factual text.
ARTEMIS
Stanford researchers conducted the first head-to-head evaluation of AI agents against human cybersecurity professionals on a live enterprise network with approximately 8,000 hosts. Their multi-agent framework ARTEMIS placed second overall, discovering 9 valid vulnerabilities with 82% accuracy and outperforming 9 of 10 human testers at a fraction of the cost (18 dollars per hour vs 60 dollars per hour for professionals).
Towards a Science of Scaling Agent Systems
Researchers from Google present a controlled evaluation framework for agent systems, challenging the assumption that “more agents are all you need.” Across 180 configurations spanning three LLM families and four agentic benchmarks, the study establishes quantitative principles for when multi-agent coordination helps versus hurts performance.
Reasoning Models Ace the CFA Exams
Researchers evaluate state-of-the-art reasoning models on mock CFA exams consisting of 980 questions across all three certification levels. While previous studies reported that LLMs performed poorly on these exams, the latest reasoning models now pass all three levels, with Gemini 3.0 Pro achieving a record 97.6% on Level I.
SUSVIBES: Is Vibe Coding Safe?
Researchers introduce SUSVIBES, a benchmark of 200 real-world software engineering tasks to evaluate the security of code generated by LLM agents through “vibe coding” - the minimal-supervision programming paradigm. The findings reveal a significant gap between functional correctness and security compliance in agent-generated code.
FINDER and DEFT
OPPO AI introduces FINDER, a fine-grained benchmark with 100 expert-curated research tasks and 419 structured checklist items for evaluating deep research agents, along with DEFT, the first failure taxonomy categorizing 14 failure modes across reasoning, retrieval, and generation dimensions.