Speculative Macro Commit for Faster Tool-Using Agents
Zeyu Liu and Peter Beerel (USC) with Souvik Kundu (Intel Labs) extend speculative decoding's idea past the token level to the action level, letting a small drafter pre-execute whole multi-action chains on an environment snapshot while the big actor catches up.
Ask this paper
The latency is in the turns, not the tokens: tool-using agents burn wall-clock on serial action-observation cycles where each call, environment transition and observation delays the next decision.
Macro library mined from traces: recurring multi-action skeletons are extracted from training traces and matched at runtime against the drafter's predicted chains, so speculation is structured rather than free-form.
Commit on first-action match: when the actor's next tool call matches the drafter's first drafted action, the remaining pre-executed steps and their observations are committed to the official trajectory wholesale.
Isolated snapshot keeps it safe: the drafter executes against an environment snapshot, so a wrong speculation costs compute rather than state.
Measured gains: with Qwen3.5-27B INT4 as actor and Qwen3.5-4B as drafter, latency drops 10.23% over the Speculative Actions baseline and 18.59% over sequential on tau2-Bench Telecom, and wall time drops 44.9% over sequential on AppWorld.
Abstract
Tool-using LLM agents spend wall-clock time not only on model inference but also in serial action--observation turns, where each tool call, environment transition, and observation can delay subsequent decisions. We introduce \textbf{Speculative Macro Commit} (SMC), a runtime mechanism for a two-tier agent system: a large authoritative actor model produces the official trajectory, while a faster speculative drafter model continuously predicts and executes future action chains on an isolated environment snapshot. SMC mines recurring multi-action skeletons from training traces and stores them in a macro library used to match against action chains predicted by the drafter at runtime. When the actor's next tool call matches the first drafted action, SMC commits the remaining pre-executed draft steps, together with their observations, to the official trajectory. Using Qwen3.5-27B INT4 as the authoritative actor model and Qwen3.5-4B as the speculative drafter model, SMC matches the sequential agent's overall accuracy while reducing latency by 10.23\% over the Speculative Actions (SA) baseline and 18.59\% over sequential execution on the $τ^2$-Bench Telecom subset. On AppWorld, SMC reduces wall time by 7.7\% over SA baseline and 44.9\% over sequential execution, with a small reduction in task completion. Overall, SMC provides a practical way to reuse multi-step speculative execution and reduce agent latency beyond single-step speculative actions. Our code is publicly available \href{https://github.com/zeyuliu1037/speculative-macro-commit}{\textcolor{magenta}{here}}.