Towards Compute-Optimal Many-Shot In-Context Learning
Free while signed in. Answers cite the passages they came from.
Proposes practical strategies for reducing the cost of many-shot in-context learning while preserving or improving performance. With long-context LLMs like Gemini Pro and Flash supporting thousands of demonstrations, caching becomes essential, yet naïvely using only random demonstrations misses potential accuracy gains from smarter selection.
The authors introduce two hybrid strategies: similarity-random and similarity-k-means. Both use a small number of similarity-selected examples tailored to each test instance, combined with a large cached set that remains fixed across test queries.
The similarity-random method adds a handful of demonstrations chosen for their semantic similarity to the test input, while the bulk are randomly sampled and cached. This keeps inference costs low and avoids full prompt regeneration.
The similarity-k-means variant replaces the random cached set with demonstrations chosen based on k-means clustering over test sample representations, ensuring greater diversity and relevance while preserving cacheability.
Results show that these methods consistently match or outperform traditional similarity-based selection (which is expensive) and random baselines across four benchmarks (ANLI, TREC, GSM Plus, MetaTool). The Pareto plots on page 2 and performance/cost curves on page 6 highlight this tradeoff: hybrid methods reach near-top accuracy with up to 10× less inference cost.
In low-data regimes (e.g., BBH subset experiments on page 8), tuning the ratio of similar to cached examples yields further gains, up to +6% over using the entire demonstration pool blindly.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack