LLMs Meet New Knowledge

Three-dimensional evaluation: Tests knowledge understanding, knowledge differentiation (old vs. new), and knowledge association across the full set of relations.

Post-cutoff focus: Uses knowledge that appears after the model's training cutoff, avoiding contamination that undermines many LLM knowledge benchmarks.

LLMs struggle with new knowledge: Reveals systematic gaps - even frontier LLMs handle post-cutoff facts significantly worse than pre-cutoff ones, despite strong reasoning.

RAG-oriented motivation: Provides empirical grounding for RAG: parametric memory is tied to training data, so retrieval remains necessary for fresh knowledge.