AI Papers of the Week
Every paper worth reading in AI, hand-picked one week at a time.
Evaluating Language Models in Realistic Conversational Contexts
Ilija Subasic, Andrew Rabinovich, and Zhao Chen (Upwork) release UPHELD, a reference-full benchmark of professionally scripted human-to-human dialogues with 36,000 per-turn human annotations, then show standard automatic metrics and LLM judges correlate poorly with expert judgment.
FaulT-Bench: Towards Benchmarking Network Troubleshooting LLM Agents under Unreliable User Tickets
Kuan-Hao Tseng and colleagues (University of Sydney) build FaulT-Bench, 200 network troubleshooting scenarios across eight topologies that include false fault reports and wrong root-cause claims, then show SADE, ReAct, and Claude Code all collapse when the network is actually healthy.
Narcissus: Program Synthesis Using Context-Aware LLM Approximations
Tilman Hinnerichs, Sebastijan Dumancic, and Neil Yorke-Smith (TU Delft) present Narcissus, an enumerative program synthesizer that keeps LLM proposals as syntax trees and scores each expansion in context, without a single LLM call during search.
Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay
Prateek Chhikara introduces matched trajectory replay, a protocol that holds answer states, evidence, budgets, and action costs fixed so confidence-to-action mappings in retrieval agents can be compared on their trajectory-level consequences rather than in isolation.
Federation Is Nearly Free, Reasoning Is Not: Tradeoffs for AI Co-Scientists in Protein Characterization Workflows
Maia Kapur and colleagues run a controlled ablation on a production agentic science platform, using protein function characterization as a verifiable task to separate what federation topology, harness type, model choice, and prompt expertise each contribute.
Benchmarking AI Agents for Hardware Design Automation via MCP Tool Calling
Leonardo Liparulo and Francesco Pierri (Politecnico di Milano) build an MCP server that mirrors a proprietary hardware design tool and benchmark seven locally deployed open-source models on dependency-ordered engineering workflows, isolating which harness choices actually move reliability.
Same Model, Different Harness: Different Coding-Agent Results
Sydney Lewis holds the model and task fixed and varies only the harness, showing that a coding agent's benchmark number is a property of the model-plus-harness pair rather than of the weights.
Safety Does Not Compose: Non-Decaying Loop State for Autonomous LLM Agents
Chenhao Wu and co-authors prove a separation result: any safety monitor scoped to a single agent trajectory is provably useless against an attack whose evidence is spread across iterations of an autonomous loop.
PILOT in the Loop: Live Self-Improvement for Long-Horizon Agents
Yang Xiao and co-authors present PILOT, a supervisor-worker harness that improves a long-horizon agent while the run is still going rather than after it ends.
ASIL: Replacing Screenshot-and-Click with Structured State and Semantic Actions
Rui Xie and Lu Chen (EMNLP Findings) argue that screenshot-and-click is the wrong interface for software-operating agents and build ASIL, which exposes applications through structured JSON observations and code-executable semantic actions.
Redwood: A Frontier AI Accelerator Designed, Verified, and Deployed from Scratch in 2 Weeks by AI
Architect Labs report Redwood, a frontier inference accelerator whose performance model, RTL, UVM environments, formal proofs, firmware and kernels were generated end to end by an AI system in under two weeks from a specification written by two human architects.
Daydreaming: Stealing Hidden Agent Skills through Black-Box Task Interaction
Yu-Lin Tsai and co-authors (NYCU, Berkeley) present Daydreaming, an execution-only attack that reconstructs a hosted multi-file agent skill purely by submitting the ordinary tasks the service exists to perform.
Tunable Tool-Call Rates in LLM Agents via Representation Steering
Yuqi Chen, Vincent Siu, Dawn Song and Chenguang Wang show that whether an instruction-tuned model calls a tool is controlled by a single linear direction in its residual stream, extractable without training and steerable at inference.
EVOMAL: Self-Poisoning in Self-Evolving Coding Agents
Shared skill libraries are usually treated as a safe way for coding agents to reuse each other's work, and EvoMal shows they propagate malware. A planted malicious skill is never invoked, but the agent retrieves it as an authoring template, writes a new skill that preserves the payload, and each authored copy re-enters the library to be imitated again. Across six models the self-poisoning rate runs 20.3% to 41.8%, deleting every planted skill does not clean it up, and a counter-prompt discouraging banner-style copying drops it to 6.7%.
A Few Pages of Markdown: Committed AI Configuration and Lower Quality Cost after Coding-Agent Adoption
Yegor Denisov-Blanch, Rylan Schaeffer, Sanmi Koyejo and colleagues at Stanford and CMU introduce RAMP, a four-level maturity model for the AGENTS.md-style configuration teams commit, and tie it to measurable divergence in code quality after agent adoption.
Paritok-4B: Intent-Conditioned Context Compression for Coding Agents
Jiayu Shi and Luzhuo Chen release Paritok-4B, a 4B LoRA compressor that shrinks coding-agent context to a quarter of its size by extracting spans rather than paraphrasing them, with weights and data open under Apache 2.0.
From General Agents to RCA Experts: A Self-Evolving Harness for Root Cause Analysis
Haiyu Huang, Zhihan Jiang, Michael Lyu and coauthors show that a general agent like Codex or Claude Code now beats purpose-built RCA agents, and argue the remaining gap lives in the harness, which OpsHarness makes self-evolving.
When Stale Constraints Go Unchecked: Budgeted Verification Failures in Inherited Agent Memory
Kazuki Nakayashiki studies what happens when an agent inherits a consolidated memory containing a constraint that has since been withdrawn, and shows that under a two-record verification budget most agents never look at the provenance path.
Automata from Agent Traces: Failure and Next-Step Prediction
Seonglae Cho and colleagues at Holistic AI collapse an entire corpus of agent traces into a single compact finite-state machine, then use that FSM as a substrate for both next-step and failure prediction.
JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution
Harnesses are hand-built and then frozen, which means one design has to serve deep research, product generation, and long-horizon coding equally well. JIT-Agent is a model whose output is a harness, synthesized per task.
TraceML: An Empirical Analysis of Human-Agent Planning in Machine Learning Development
Jiarui Yan, Weiwei Sun, Sijie Li and Yiming Yang at CMU pair 4,465 human Kaggle trajectories with agent runs on the same competitions under one version-level schema, so the ML-development gap can be read as behavior rather than a final score.
Repair or Resample? Rethinking Failure Debugging in LLM Multi-Agent Systems
Zhongwen Luan, Xiaoyu Zhang, Ming Hu and coauthors ask whether multi-agent repair methods causally fix failures or merely exploit LLM sampling randomness, and build SymTrace to make the distinction measurable.
Recursive Experiential-Working Memory Evolution for Long-Horizon Agent Harnesses
Recuris splits agent memory in two, with a Working Memory tracking task progress and an Experiential Memory holding skills, so skill selection is grounded in the current task state rather than the full growing history. Because skill use is anchored to an explicit state, a failed run points at a specific memory component, and a fixed Meta-Agent turns that evidence into validation-gated updates to Skill Memory. It improves task success in 35 of 37 completed model-benchmark pairs, adding 17.8 points to GPT-5.6 Sol on tau-bench and taking Claude Opus 5 to 87.9%.
FM-Bench: A Benchmark for Long-Horizon Management with Competing Agents
FM-Bench turns football club management into a 20-year test of sustained agent decision-making. Fifteen frontier models operate through 26 tools and hundreds of consequential decisions in a deterministic environment with no LLM judge. The results show that model scale, price, vendor, and token spend do not predict performance; managerial behavior and memory discipline do. Every model also fails to learn hidden market prices from repeated feedback, exposing a concrete limit in long-horizon adaptation.