Adaptive LLM Routing
Free while signed in. Answers cite the passages they came from.
A routing framework that learns online which model to call for each query while honoring a spend limit. It treats routing as a contextual bandit, initializes with human preference data, and adds an online cost policy that allocates budget across queries.
Core idea: Build a shared embedding space for queries and candidate LLMs, align it with offline human preferences, then update LLM embeddings online using bandit feedback. Selection uses a preference-prior LinUCB variant (PILOT) with cosine-similarity rewards.
Budget control: Introduces an online multi-choice knapsack policy (ZCL-style) that filters eligible models by reward-to-cost thresholds and allocates spend in bins so the total stays within budget.
Results: On RouterBench multi-task routing, achieves about 93% of GPT-4 performance at roughly 25% of its cost; on single-task MMLU, about 86% at roughly 27% cost. Cumulative regret is consistently lower than bandit baselines.
Cost policy effectiveness: Online policy matches or outperforms a strong offline P − λC oracle tuned with hindsight across budgets.
Latency overhead: Routing adds little delay relative to inference. Selection takes 0.065–0.239 s vs. ~2.5 s for GPT-4 on MMLU..
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack