FACTS Leaderboard
Free while signed in. Answers cite the passages they came from.
Google introduces the FACTS Leaderboard, a comprehensive benchmark suite for evaluating LLM factuality across diverse scenarios. The leaderboard aggregates performance across four specialized sub-benchmarks to provide a holistic measure of how accurately models generate factual text. - **Four evaluation dimensions:** FACTS Multimodal tests visual grounding with world knowledge on image-based questions. FACTS Parametric measures closed-book factoid question answering from internal parameters. FACTS Search evaluates factuality when using search APIs. FACTS Grounding v2 checks if long-form responses align with source documents. - **Automated judging system:** Each sub-leaderboard uses automated judge models to score responses. The final FACTS Score averages all four components for balanced assessment. Coverage and No-Contradiction verdicts ensure responses are both complete and accurate. - **Current rankings:** Gemini 3 Pro leads with 68.8% overall, followed by Gemini 2.5 Pro at 62.1% and GPT 5 at 61.8%. The benchmark reveals trade-offs - Gemini models show higher coverage while GPT models achieve better no-contradiction scores. - **Benchmark integrity:** The suite includes public and private test splits to prevent overfitting. Hosted on Kaggle, it remains open for new model submissions while maintaining evaluation integrity through hidden test prompts.
Four evaluation dimensions: FACTS Multimodal tests visual grounding with world knowledge on image-based questions. FACTS Parametric measures closed-book factoid question answering from internal parameters. FACTS Search evaluates factuality when using search APIs. FACTS Grounding v2 checks if long-form responses align with source documents.
Automated judging system: Each sub-leaderboard uses automated judge models to score responses. The final FACTS Score averages all four components for a balanced assessment. Coverage and No-Contradiction verdicts ensure responses are both complete and accurate.
Current rankings: Gemini 3 Pro leads with 68.8% overall, followed by Gemini 2.5 Pro at 62.1% and GPT 5 at 61.8%. The benchmark reveals trade-offs - Gemini models show higher coverage while GPT models achieve better no-contradiction scores.
Benchmark integrity: The suite includes public and private test splits to prevent overfitting. Hosted on Kaggle, it remains open for new model submissions while maintaining evaluation integrity through hidden test prompts.
Abstract
We introduce The FACTS Leaderboard, an online leaderboard suite and associated set of benchmarks that comprehensively evaluates the ability of language models to generate factually accurate text across diverse scenarios. The suite provides a holistic measure of factuality by aggregating the performance of models on four distinct sub-leaderboards: (1) FACTS Multimodal, which measures the factuality of responses to image-based questions; (2) FACTS Parametric, which assesses models' world knowledge by answering closed-book factoid questions from internal parameters; (3) FACTS Search, which evaluates factuality in information-seeking scenarios, where the model must use a search API; and (4) FACTS Grounding (v2), which evaluates whether long-form responses are grounded in provided documents, featuring significantly improved judge models. Each sub-leaderboard employs automated judge models to score model responses, and the final suite score is an average of the four components, designed to provide a robust and balanced assessment of a model's overall factuality. The FACTS Leaderboard Suite will be actively maintained, containing both public and private splits to allow for external participation while guarding its integrity. It can be found at this https URL .
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack