GAIA
Meta's GAIA is a benchmark for general AI assistants that requires reasoning, multimodal handling, web browsing, and tool use to solve real-world questions.
Ask this paper
Real-world questions: Questions are conceptually simple for humans but require integrated reasoning, web research, and tool use - a realistic test for assistant-style AI.
Massive human-model gap: Humans achieve 92% accuracy while GPT-4 with plugins achieves only 15% - the widest human-AI gap on any major 2023 benchmark.
Level-graduated difficulty: Three difficulty levels let researchers measure incremental progress rather than just binary success/failure.
Agent-first evaluation: Explicitly designed to test AI assistants, not base LLMs - a framing that has since become dominant for agent evaluations.