Inspire: Benchmarking Scientific Literature Search for Open Research Problems
Jianrong Ding (CUHK, Microsoft Research Asia intern) and colleagues at Microsoft Research Asia and CUHK introduce INSPIRE, a benchmark where agents search an open, date-gated corpus for prior work that later solved a redacted research problem, scored separately on exposure, selection and ranking.
Ask this paper
Task. Each instance gives a research brief with the solution removed and a search cutoff three months before the target paper. Agents must find the papers that the target later cited as antecedents, without knowing the target.
Three-stage scoring. Logged trajectories measure exposure (did useful papers appear in search), selection (were they kept) and ranking (final order), so losses can be located.
Results. The strongest model reaches 0.284 nDCG@10 and finds at least one antecedent for 72.3% of targets, but three for only 32.1% and five for 6.7%. It recovers 21.5% of labeled antecedents on average.
Exposure is the main loss. The share of targets where no relevant antecedent is ever surfaced ranges from 17.4% to 56.1% across models, so most evidence is lost before selection.
Training signal. SFT on INSPIRE trajectories raises exposed gain from 0.153 to 0.251 and cuts zero-exposure episodes from 56.1% to 29.4% on held-out targets.
Abstract
Scientific literature search often begins with an open research problem rather than a known target paper or a fixed candidate set. We introduce INSPIRE, a benchmark for evaluating agents that search prior literature to make progress on solution-redacted research problems. Each instance pairs a research brief with a target-specific cutoff three months before a later paper and evaluates ranked outputs against graded cited antecedents from that paper's realized research lineage. Search proceeds over an open corpus, while the identity of the target paper and membership of its cited antecedents remain hidden from the agent. Beyond end-to-end retrieval quality, INSPIRE uses logged search trajectories to distinguish three coupled stages: resource exposure, whether useful antecedents are surfaced during search; selection, whether exposed antecedents are retained; and ranking, how effectively retained papers are ordered. Across 476 computer-science targets under a shared search interface and budget, the strongest evaluated agent achieves 0.284 nDCG@10. Results show that current agents more readily recover an isolated antecedent than assemble a broader portfolio of relevant prior work. The stagewise analysis identifies resource exposure as the largest observed bottleneck, with further losses in selection and ranking. We additionally construct replay-valid hindsight demonstrations and show that they improve held-out search without changing test-time information, establishing that the benchmark provides an actionable learning signal. INSPIRE therefore enables both end-to-end comparison and stage-resolved diagnosis in a setting where the agent must construct its own working criterion of relevance.