BudgetBench: A Budget-Tiered Protocol and Pilot Harness for Memory Strategy Evaluation in Local Large Language Model Agents
Rao and Jaggi build a measurement harness that makes the per-call input-token budget the independent variable when comparing agent memory strategies, and report budget-violation rates as a first-class outcome rather than a footnote.
Ask this paper
Active context is the scarce resource. For a local model agent, memory capacity, prefill latency, cache growth and service objectives all cap how many input tokens a call can afford, so a memory strategy that ignores the budget is not comparable to one that respects it.
Sweep the budget, hold everything else fixed. With model, task, sampler and decoding held constant, the harness sweeps 2K, 4K, 8K, 16K and 32K tokens and records quality, budget utilization, latency and budget-violation rate.
The reusable surface is the contribution. A swappable MemoryStrategy contract, explicit budget enforcement, deterministic or versioned graders, prompt-audit metadata and reproducibility artifacts are released on GitHub.
Pilot studies rather than rankings. A local qwen2.5:1.5b pilot of 89 items each on SWE-bench Verified and LongBench v2, a hosted 50-item Qwen3 30B-A3B LongBench replication with exact tokenization, and a 500-item LongMemEval study scored by the official GPT-4o evaluator.
What the harness exposes. Budget-compliance failures and non-monotonic behavior across budgets, both of which a fixed-budget comparison would hide.
Abstract
For local large language model agents, active context is a scarce resource: memory capacity, prefill latency, cache growth, and service objectives all constrain how many input tokens each call can afford. We present BudgetBench, an active-budget protocol and reference harness that treats the per-call input-token budget as the independent variable when comparing memory strategies. Holding the model, task, sampler, and decoding fixed, it sweeps budgets over 2K, 4K, 8K, 16K, and 32K tokens and records quality, budget utilization, latency, and, as a first-class outcome, budget-violation rates. The core contribution is this reusable measurement surface: a swappable MemoryStrategy contract, explicit budget enforcement, deterministic or versioned graders, prompt-audit metadata, and reproducibility artifacts, released at https://github.com/aviskaar/budgetbench. We substantiate the protocol with pilot studies rather than final rankings. Across a local qwen2.5:1.5b pilot (89 items each on SWE-bench Verified and LongBench v2), a hosted 50-item Qwen3 30B-A3B LongBench replication with exact tokenization, and a 500-item LongMemEval oracle study scored by the official GPT-4o evaluator, the harness exposes budget-compliance failures, non-monotonic quality curves, and operating points that single-budget evaluation hides. The budgeted-versus-full-context direction remains unresolved: the local slice is near-null and the hosted replication favors full context in point estimate. We report results transparently, including that the early pilot's tokenizer approximation undercounts some served-model prompts, so its violation rows are tokenizer-approximation diagnostics, not claim-bearing results; all timings are operational diagnostics. The reusable contribution is the protocol, harness, and failure-reporting discipline needed to scale fixed-budget memory-strategy evaluation.