Efficient Benchmarking in Production: A Study of an Evolving LLM Agent
Yining She (Carnegie Mellon, work done at Meta) and Lei Lin (Meta) study how to re-evaluate a production analytics agent with tens of thousands of monthly users without rerunning its full benchmark each time, using 574 historical benchmark runs.
Ask this paper
Setup. Runs are split chronologically into calibration and held-out periods, and four approaches are compared: random sampling, historical caching, fixed representative subsets and IRT-based adaptive testing.
Best fidelity. Multidimensional 2PL adaptive testing does best, with 200 questions, 38.5% of a full run, giving 1.03 points of mean absolute error.
What was deployed. The team shipped difficulty-stratified fixed subsets instead, because they are simpler to operate.
Transfer. The fixed subsets transfer without recalibration to five other agent families and stay stable with calibration windows as short as one day.
Practical guidance. The paper closes with recommendations for recurring evaluation of production agents drawn from this deployment.
Abstract
Production LLM agents are evaluated repeatedly as they evolve, but full agent benchmarks are costly to rerun. We study efficient recurring evaluation for a production analytics agent serving tens of thousands of monthly active users and report first-hand deployment experience. Using 574 historical runs of the production benchmark, split chronologically into calibration and held-out periods, we compare random sampling, historical caching, fixed representative subsets, and IRT-based adaptive testing. The results show that multidimensional 2PL adaptive testing achieves the best overall score fidelity: executing 200 questions, 38.5% of a full run, yields 1.03 pp of MAE. We nevertheless deployed difficulty-stratified fixed subsets because of their operational simplicity, and show they transfer without recalibration to five other agent families and remain stable across calibration windows as short as one day. Drawing on this deployment experience, we report practical recommendations for recurring production-agent evaluation.