Do I Need the Cloud? Uncertainty-Aware Step-Level Handoff for Small Language Model Agents
Abolfazl Younesi (Sharif University of Technology) presents StepGate, which lets a small local model run an agent and escalate individual steps to a stronger cloud model when its uncertainty is high, instead of routing whole queries. Presented at the NeurIPS 2026 SLMs for Agentic Systems workshop.
Ask this paper
Step-level routing. Routers usually pick one model per query. StepGate scores each local step and hands off only the uncertain ones, because step difficulty changes with intermediate tool observations.
Single-step result. At a preselected 30% cloud budget with a Qwen2.5 1.5B/7B pair, StepGate reaches 82.7% task success with 30.8% escalation, against 67.3% local-only and 75.4% random escalation at 33.8%.
Multi-turn result. 69.0% trajectory success and 84.0% action success with 30.0% cloud actions, against 48.0%/70.5% local-only and 57.0%/77.1% query-level routing. Strong-only reaches 82.0% at 100% cloud actions.
Share of the gap. StepGate closes over 60% of the gap to the strong model while running about 70% of actions locally.
Limitations. One model family, one strong backend, scripted tasks and small test sets; the author lists these in the abstract.
Abstract
Small language models (SLMs) are attractive as local agent controllers because they reduce remote inference, latency, and deployment footprint, yet structured tool errors can cause an agent step to fail. Existing routers typically select a model once per query. However, agents expose sequential decision points whose difficulty dynamically changes based on intermediate observations. We propose STEPGATE, an uncertainty-aware handoff framework that scores each local SLM action and selectively escalates challenging steps to a stronger model. On a 52-task held-out single-step BFCL-derived test split, the Qwen2.5-1.5B/7B pair attains 82.7% task success with 30.8% escalation, versus 67.3% local-only and 75.4% random escalation (which uses 33.8% escalation). In a separate multi-turn evaluation, STEPGATE achieves 69.0% trajectory success and 84.0% action success using only 30.0% cloud actions, compared with 48.0%/70.5% local-only, 60.0%/78.2% random escalation, and 57.0%/77.1% query-level routing (strong-only achieves 82.0% trajectory success at 100% cloud actions). These results suggest that step-level escalation recovers a large share of the performance gap to the stronger Qwen2.5-7B backend at a matched cloud-action rate while transmitting fewer tokens remotely. However, our evaluation is limited to one model family, a single stronger backend, and scripted tasks. Furthermore, the test sets are small, multi-turn comparisons rely on paired intervals and statistical tests, and our risk tiers serve as research annotations rather than formal safety guarantees.