Structural Planning for LLM Agents
Free while signed in. Answers cite the passages they came from.
This paper introduces Routine, a structured planning format designed to improve the stability and accuracy of LLM agents executing multi-step tool-calling tasks in enterprise settings. Traditional agent planning approaches often fail in enterprise scenarios due to unstructured plans, weak instruction following, and tool selection errors. Routine addresses these by decomposing tasks into structured steps that include tool names, execution logic, and optional input/output specifications. The authors evaluate Routine on real-world HR scenarios and show strong performance gains in both open and fine-tuned models.
Routine provides a clear and modular format for LLM agents to follow multi-step plans, reducing ambiguity and improving tool selection. Each step contains a step number, name, detailed description, and (optionally) inputs, outputs, and the tool to be called.
In a real HR agent scenario with 7 multi-step workflows, adding Routine increased GPT-4o’s accuracy from 41.1% to 96.3% and Qwen3-14B’s from 32.6% to 83.3%. Fine-tuning Qwen3-14B on a Routine-following dataset further increased accuracy to 88.2%; training on a Routine-distilled dataset reached 95.5%.
The framework separates planning (with LLMs) from execution (with small instruction-tuned models), using Routine as the bridge. This enables small-scale models to reliably execute complex plans with minimal resource overhead, especially when using variable memory and modular tools like MCP servers.
An ablation study shows the importance of explicitly including tool names and I/O descriptions in Routine steps. Removing tool names dropped Qwen3-14B’s accuracy from 83.3% to 71.9%. Adding I/O fields provided minor gains, especially for less capable models.
Using AI-optimized Routine (via GPT-4o) to refine user-written drafts resulted in execution accuracy close to human-annotated plans, suggesting Routine authoring can be scaled via LLMs. However, manual review still yields the best results for high-performing models.
When multiple Routine candidates are recalled, accuracy can decline, highlighting the need for high-precision retrieval in memory-based systems. Surprisingly, some smaller models performed better when exposed to more routines due to repeated substeps aiding execution.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack