FreshLLMs (FreshQA)
Free while signed in. Answers cite the passages they came from.
Introduces FreshQA, a dynamic benchmark designed to stress-test LLMs on time-sensitive knowledge.
Dynamic QA benchmark: Continuously refreshes questions so models can't memorize answers - a direct response to the contamination concerns plaguing static benchmarks.
Four question categories: Covers never-changing, slow-changing, fast-changing, and false-premise questions, stressing different aspects of freshness handling.
Reveals freshness gap: Shows that LLMs without search augmentation answer fast-changing questions poorly, while retrieval-augmented models close most of the gap.
FreshPrompt: Proposes FreshPrompt, a simple search-augmented prompting strategy that substantially boosts LLM performance on time-sensitive questions.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack