LLMs Meet New Knowledge
Free while signed in. Answers cite the passages they came from.
A benchmark that evaluates how well LLMs handle new knowledge beyond their training cutoff.
Three-dimensional evaluation: Tests knowledge understanding, knowledge differentiation (old vs. new), and knowledge association across the full set of relations.
Post-cutoff focus: Uses knowledge that appears after the model's training cutoff, avoiding contamination that undermines many LLM knowledge benchmarks.
LLMs struggle with new knowledge: Reveals systematic gaps - even frontier LLMs handle post-cutoff facts significantly worse than pre-cutoff ones, despite strong reasoning.
RAG-oriented motivation: Provides empirical grounding for RAG: parametric memory is tied to training data, so retrieval remains necessary for fresh knowledge.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack