Benchmarking AI Agents for Hardware Design Automation via MCP Tool Calling
Leonardo Liparulo and Francesco Pierri (Politecnico di Milano) build an MCP server that mirrors a proprietary hardware design tool and benchmark seven locally deployed open-source models on dependency-ordered engineering workflows, isolating which harness choices actually move reliability.
Ask this paper
Confidentiality forces local models: Component specs and naming conventions cannot leave the building, so hosted frontier APIs are off the table. The paper is one of the few MCP benchmarks written for that constraint rather than around it.
Ablated harness knobs, not model rankings: They sweep system prompts, tool-description detail, context scope, and single-agent versus multi-agent decomposition. Comprehensive tool descriptions consistently reduce failures, which is the cheapest available fix.
Few-shot prompting can backfire: For some models few-shot examples cause severe inaction rather than better calls, and cumulative context actively harms constrained models. Both are counterintuitive enough to be worth testing in your own stack.
Multi-agent helps only weak workers: Decomposition buys reliability for weak models or long sessions at the price of extra calls. For strong models it is overhead.
Why it matters: Most MCP evaluation is frontier-model demo work. This is a stateful, dependency-aware benchmark that answers the practical question of what to change in the harness when your model is fixed.
Abstract
We ask whether AI agents powered by locally deployed large language models can reliably automate expert-defined hardware design workflows in an industry-realistic tool-calling setting. In these environments, engineers issue repetitive, dependency-ordered operations