Agent Seer: Synthesizing Scenarios from Specification Understanding
Harish Karumuri, Mahesh Vemula, and David Lopes Pegna show that an MCP specification alone carries enough semantic information to synthesize realistic multi-turn agent evaluation scenarios, with no examples, no live tool access, and no domain tuning.
Ask this paper
The spec is the dataset: Function names, natural-language descriptions, and typed parameter schemas already encode enough structure to generate graded scenarios with synthetic tool outputs and mock-data-grounded dialogues.
Tested across seven MCP specs: Domains and tool-suite sizes vary; the pipeline achieves complete tool coverage on small and medium specifications.
Schema complexity, not suite size, predicts quality: Parameter schema complexity is the strongest correlate of quality variation, with tool-suite size a smaller orthogonal factor.
Argument values are the failure mode: The dominant error is wrong argument values, a sub-dimension that coarse name-match tool-calling metrics cannot see at all.
Why it matters: Hand-built agent benchmarks go stale the moment an API changes. Generating them from the live spec is the only version of this that scales with a tool ecosystem.
Abstract
Evaluating AI agents that use external tools requires realistic test scenarios that capture how practitioners compose tools and iterate across conversation turns. Constructing such scenarios by hand demands deep domain expertise, does not scale across tool ecosystems, and produces static benchmarks that cannot track evolving APIs. We observe that tool specifications -- function names, natural-language descriptions, and typed parameter schemas -- already encode sufficient semantic information to synthesize realistic evaluation scenarios without manual curation or live tool execution. Agent Seer builds off this latent information: from a single Model Context Protocol (MCP) specification, with no examples, no live tool access, and no domain-specific tuning. This pipeline enriches raw schemas, generates graded scenarios with synthetic tool outputs, and expands them into mock-data-grounded multi-turn dialogues that exhibit strong tool-calling correctness and conversational coherence. Evaluation quality is measured by applying this pipeline on seven MCP specifications spanning diverse domains and tool-suite sizes and measuring the tool-calling correctness and conversational coherence. The pipeline achieves strong quality across all domains, with complete tool coverage on small and medium specifications. Two findings emerge within this analysis: parameter schema complexity is the strongest correlate of quality variation -- tool-suite size plays a smaller, orthogonal role -- and argument value accuracy is the dominant failure mode among imperfect scenarios, a sub-dimension invisible to coarse-grained name-match metrics.