SemVerBench: Benchmarking LLM Comprehension of Version-Constraint Resolution Semantics
Qibai Chen (independent researcher) and Zeming Liu (Brown University) measure how well frontier LLMs resolve package version constraints across npm, PEP 440 and Cargo, and find predictable rule-specific blind spots that tool delegation removes.
Ask this paper
Benchmark: 240 machine-checkable items with unique answers, drawn from each ecosystem's official test suite plus three LLM proposers, and labeled by two independent resolver implementations.
Shared blind spot: The partial-comparator carry rule in Cargo, where >1.2 means >=1.3.0, drops every model to about 60%.
Model gaps: On PEP 440 zero-pad and post-release corner cases GPT-5.1 scores 0 of 26 while Claude stays at 97 to 100%. Opus beats all other models, and Sonnet beats the OpenAI models.
Application gap: Injecting the rule or a light hint recovers most errors and models are at ceiling on basic forms, so the failures come from applying rules they know.
Recommendation: A free resolver brings accuracy to about 100%, so coding agents should call a resolver instead of reasoning about versions themselves.
Abstract
Large language model (LLM) coding agents constantly decide whether a version satisfies a constraint such as ^1.2.3 or >=2.0,<3, yet their grasp of version-constraint semantics has never been measured directly. We introduce SemVerBench, the first benchmark of LLM version-constraint resolution semantics across three ecosystems (npm, PEP 440, Cargo): 240 machine-checkable items with unique answers, built author-neutrally from four balanced sources (each ecosystem's official test suite plus three frontier LLM proposers) and labeled by a non-circular two-implementation oracle. Evaluating six frontier models, we find systematic, predictable per-mechanism blind spots: a partial-comparator carry rule (>1.2 means >=1.3.0) traps every model on Cargo (near 60%), and although standard PEP 440 prefix matching is universal, on zero-pad/post-release corner cases GPT-5.1 collapses (0/26) while Claude stays at 97-100% (verified on a 67-item oracle-validated set). Opus significantly outperforms all other models, and Sonnet outperforms the OpenAI models (McNemar). The failures look more like an activation/application gap than a knowledge gap: injecting the rule or a light correct hint recovers most errors, whereas interval decomposition does not, and models are at ceiling on the basic forms of the same rules. An author-stratified analysis finds no statistically significant self-favoritism. Because the task is verifiable and a free, 100%-correct resolver exists, tool delegation reaches ~100%: coding agents should delegate version resolution to a resolver rather than reason about versions in-head.