SciDocBench: A Workflow-Centered Benchmark and Data Pipeline for Scientific Document Understanding
Shenxi Wu and colleagues at The Chinese University of Hong Kong and Shanghai AI Laboratory build a benchmark that tests scientific document understanding as a research-assistant workflow rather than as isolated perception, retrieval and reasoning tasks, and ship the training data to improve it.
Ask this paper
SciDocBench: 124 expert-authored, difficulty-screened questions in seven research-assistant capability groups and 19 subtasks across five scientific domains. Each question is instantiated under four matched conditions crossing English or Chinese with all-images-first or interleaved document representations, giving 496 evaluation instances.
Headroom: The strongest evaluated system reaches 62.6/100, with the weakest areas being document perception, evidence grounding, verification and cross-document reasoning.
SciDocIR: A typed evidence-graph representation that preserves document objects, layout and cross-reference relations, and provenance, so evidence stays traceable through the pipeline.
SciDocDataset: Roughly 15K supervised fine-tuning samples and 8K reinforcement-learning samples across 14 verifiable subtasks, built on SciDocIR, turning the diagnostic into training signal. Project page at github.com/InternLM/SciDocBench.
Abstract
Scientific papers require models to reason jointly over text, equations, figures, tables, code, and datasets while preserving the provenance of supporting evidence. Existing benchmarks typically evaluate these capabilities in isolation, leaving unclear whether multimodal models can support realistic scientific-reading workflows. We introduce SciDocBench, a workflow-centered benchmark for scientific document understanding. It contains 124 expert-authored and difficulty-screened questions organized into seven research-assistant capability groups and 19 subtasks across five scientific domains. Each question is instantiated under four matched conditions combining English or Chinese questions with all-images-first or interleaved document representations, yielding 496 evaluation instances for controlled analysis. The strongest evaluated system achieves only 62.6/100, with pronounced weaknesses in document perception, evidence grounding, verification, and cross-document reasoning. To translate these diagnostics into scalable training signals, we introduce SciDocIR, a typed evidence-graph representation that preserves scientific document objects, layout and cross-reference relations, and provenance. Building on SciDocIR, we construct SciDocDataset, comprising approximately 15K supervised fine-tuning samples and 8K reinforcement-learning samples across 14 verifiable subtasks. Together, SciDocBench, SciDocIR, and SciDocDataset form an evaluation-to-training framework for diagnosing and improving scientific-document assistants. The project page is available at https://github.com/InternLM/SciDocBench.