The Efficiency Frontier
Context costs dominate production LLM bills, and the right strategy depends on how often preprocessing gets reused. This paper models context-strategy selection as a deployment-aware optimization problem that jointly accounts for task performance, token cost, and reuse, then uses it to compare retrieval-based and preprocessing-based approaches under realistic constraints.
Ask this paper
A reuse-aware cost model: A parameterized log-utility metric captures diminishing returns from more context while charging an amortized preprocessing cost. Varying a reuse parameter lets the framework compare strategies under different deployment patterns on equal footing.
Distinct operating regimes: The analysis reveals clean transition boundaries between retrieval and preprocessing strategies. Which one wins flips depending on how many times you reuse the preprocessed context, so a single default is rarely optimal.
Real token savings: On 5,000 HotpotQA instances, deployment-aware optimization cuts effective token usage by roughly 25% at comparable performance, and amortized memory compression achieves over 50% lower token cost relative to full-context.
Why it matters: Most teams pick a context strategy once and pay for it on every request. Treating context management as an explicit cost-performance optimization turns a guess into a measurable decision, with double-digit savings available on common workloads.