Federation Is Nearly Free, Reasoning Is Not: Tradeoffs for AI Co-Scientists in Protein Characterization Workflows
Maia Kapur and colleagues run a controlled ablation on a production agentic science platform, using protein function characterization as a verifiable task to separate what federation topology, harness type, model choice, and prompt expertise each contribute.
Ask this paper
Model choice dominates everything else: Opus lands at roughly 92 to 94 percent while o4-mini sits at 40 to 50 percent. Federation topology and prompting move the needle far less, which reframes where co-scientist engineering effort should go.
Federation is nearly free: Communicating across institutional boundaries imposes a negligible performance penalty. That is a useful green light for multi-institution scientific agent deployments.
A PPO policy nearly matches the best LLM: 88 percent accuracy at zero token cost, fastest latency, and perfect consistency, but no reasoning trace. The tradeoff is legibility, not accuracy.
Prompt dependence peaks on hard tasks: Expert-prompted LLMs reach the highest accuracy but are expensive and less consistent, and prompt sensitivity grows with protein novelty.
Why it matters: The actionable guidance is to route routine verifiable steps to a cheap deterministic policy and reserve flexible LLM reasoning for open-ended discovery, rather than running everything through the expensive path.
Abstract
Natural language driven autonomous co-scientist workflows involve a fundamental trade-off between flexibility and reasoning at the expense of determinism, reproducibility, and observability. Such agents increasingly must communicate across institutional boundaries, where federation topology can shape latency and cost. We systematically evaluated these tradeoffs using a controlled ablation on a production agentic platform for science. We use a verifiable task: given a protein sequence, we ask an agent to confidently characterize its function by routing across common tools. We compare federation topology, classic RL vs LLM-driven harnesses, language model, and prompt expertise. We also stratify results by protein novelty. We find that the choice of LLM dominated prediction quality far more than topology or prompting (Opus ~92%-94% vs o4-mini ~40%-50%). The PPO policy was nearly as accurate as the best LLM (88%) at zero token cost, fastest latency, and perfect consistency, but yields no reasoning trace. Expert prompted LLMs reached the highest accuracy but were high-cost and less consistent; prompt dependence was largest when the task was hardest. Federation imposed a negligible penalty on performance. These results offer actionable guidance for deploying agents for scientific workflows: for routine, verifiable tasks, a cheap deterministic policy delivers near-frontier accuracy with complete reproducibility, while flexible LLM reasoning is best reserved for open-ended discovery.