TruthInsightBench: An Evidence-Grounded Benchmark for Automated Evaluation of Open-Ended Scientific Discovery Agents
Zhibo Yang and colleagues build a benchmark for scientific discovery rather than reproduction: agents see a neutral objective and frozen data, with the source study's conclusions, expected values, and analysis path withheld.
Ask this paper
The design difference. Existing AI-scientist benchmarks build tasks and rubrics around a hidden target study and reward recovering its result, which measures execution rather than discovery.
The tasks. 40 blind tasks drawn from 40 peer-reviewed studies across 10 scientific domains.
Automated grading. A fixed LLM judge scores the evidentiary maturity of the agent's own claims along six dimensions, operationalized as 29 artifact-grounded items with deterministic aggregation and no per-instance human grading.
The finding. Four coding agents on one frozen base model form a narrow plateau at 58.4 to 60.3 out of 100 with no reliable pairwise separation. They document analyses competently but lack controls, robustness checks, falsifiability, and cross-dataset generalization.
The diagnosis. The bottleneck is scientific judgment, not coding ability. Data and scoring code at github.com/TruthInsight-stack/TruthInsightBench.
Abstract
Autonomous coding agents are increasingly proposed as AI-scientist systems that conduct analyses and write research reports, but executing a prescribed analysis is not the same as making a discovery. Existing benchmarks are configured for reproduction: tasks, data, and rubrics are built around a hidden target study, and recovery of its result is rewarded. We present TruthInsightBench, a benchmark configured for discovery. Its 40 blind tasks, drawn from 40 peer-reviewed studies across 10 scientific domains, expose only a neutral scientific objective and frozen data; source conclusions, expected values, and analysis paths are withheld, leaving the agent to determine what claim the data support. A fixed LLM-based judge scores the evidentiary maturity of an agent's own claims along six dimensions, operationalized as 29 artifact-grounded items, with automated, deterministic aggregation and no per-instance human grading, so evaluation can be repeated automatically as agents evolve. On one frozen base model, four coding agents form a narrow plateau (58.4-60.3 of 100) with no statistically reliable pairwise separation: they execute and document analyses competently, with comparatively strong evidence auditability and novelty, but largely lack the discriminating acts that establish a trustworthy claim (controls, robustness, falsifiability, and cross-dataset generalization). The bottleneck is scientific judgment rather than coding, and genuine discovery remains out of reach. TruthInsightBench makes this gap a measurable target; data and scoring code are at https://github.com/TruthInsight-stack/TruthInsightBench.