FINDER and DEFT
Free while signed in. Answers cite the passages they came from.
OPPO AI introduces FINDER, a fine-grained benchmark with 100 expert-curated research tasks and 419 structured checklist items for evaluating deep research agents, along with DEFT, the first failure taxonomy categorizing 14 failure modes across reasoning, retrieval, and generation dimensions. - **Benchmark design:** FINDER refines prompts from DeepResearch Bench with explicit guidelines on report length, format, and disciplinary scope. Each task includes 3-5 structured checklists that guide evaluation of report structure, analytical depth, and citation integrity. - **Failure taxonomy construction:** DEFT was built using grounded theory with human-LLM collaborative coding across approximately 1,000 generated reports. The taxonomy identifies failures like Strategic Content Fabrication (19% of errors), Insufficient Information Acquisition (16.3%), and Lack of Analytical Depth (11.1%). - **Key finding - generation bottleneck:** Over 39% of failures occur in content generation, particularly through strategic content fabrication where agents generate unsupported but professional-sounding content. Retrieval failures account for 32% of errors, highlighting challenges in evidence integration and verification. - **Reasoning resilience insight:** The study reveals that current deep research agents struggle not with task comprehension but with evidence integration, verification, and maintaining reasoning consistency across complex multi-step research tasks. - **Benchmark results:** Gemini 2.5 Pro Deep Research leads with 50.95 overall RACE score. MiroFlow-English achieves highest checklist accuracy (72.19%), while models like Kimi K2 show strong reasoning but suffer sharp declines in generation quality.
Benchmark design: FINDER refines prompts from DeepResearch Bench with explicit guidelines on report length, format, and disciplinary scope. Each task includes 3-5 structured checklists that guide evaluation of report structure, analytical depth, and citation integrity.
Failure taxonomy construction: DEFT was built using grounded theory with human-LLM collaborative coding across approximately 1,000 generated reports. The taxonomy identifies failures like Strategic Content Fabrication (19% of errors), Insufficient Information Acquisition (16.3%), and Lack of Analytical Depth (11.1%).
Key finding - generation bottleneck: Over 39% of failures occur in content generation, particularly through strategic content fabrication, where agents generate unsupported but professional-sounding content. Retrieval failures account for 32% of errors, highlighting challenges in evidence integration and verification.
Reasoning resilience insight: The study reveals that current deep research agents struggle not with task comprehension but with evidence integration, verification, and maintaining reasoning consistency across complex multi-step research tasks.
Benchmark results: Gemini 2.5 Pro Deep Research leads with 50.95 overall RACE score. MiroFlow-English achieves the highest checklist accuracy (72.19%), while models like Kimi K2 show strong reasoning but suffer sharp declines in generation quality.
Abstract
Deep Research Agents (DRAs) aim to automatically produce analyst-level reports through iterative information retrieval and synthesis. However, most existing DRAs were validated on question-answering benchmarks, while research on generating comprehensive reports remains overlooked. Worse, current benchmarks for report synthesis suffer from task complexity and subjective metrics -- this fails to reflect user demands and limits the practical utility of generated reports. To address these gaps, we present Fine-grained DEepResearch bench (FINDER), an enhanced benchmark consisting of 100 human-curated research tasks with 419 structured checklist items that standardize report structure, analytical depth, and factual grounding. Based on approximately 1,000 reports produced by mainstream DRAs, we further propose Deep rEsearch Failure Taxonomy (DEFT), the first failure taxonomy for deep research agents. DEFT contains 14 fine-grained failure modes across reasoning, retrieval, and generation, and is built upon grounded theory with human-LLM co-annotating and inter-annotator reliability validation. Our experimental findings reveal that current DRAs struggle not with task comprehension but with evidence integration, verification, and reasoning-resilient planning.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack