SemVerBench: Benchmarking LLM Comprehension of Version-Constraint Resolution Semantics
This paper provides a detailed analysis of how different AI models handle version constraint resolution, a critical task for coding assistants. It offers concrete performance data that developers can use to evaluate which model is best for their needs.
A new benchmark called SemVerBench has been introduced to measure how well large language models understand version constraint resolution. The benchmark evaluates six leading models across three ecosystems: npm, PEP 440, and Cargo. The study found that models like GPT-5.1 struggled with zero-pad and post-release corner cases, while Claude performed well. Opus and Sonnet outperformed OpenAI's models. The research suggests that models have blind spots in applying version rules rather than lacking knowledge, and that coding agents should delegate this task to a dedicated resolver.
Large language model (LLM) coding agents constantly decide whether a version satisfies a constraint such as ^1.2.3 or >=2.0,<3, yet their grasp of version-constraint semantics has never been measured directly. We introduce SemVerBench, the first benchmark of LLM version-constraint resolution semantics across three ecosystems (npm, PEP 440, Cargo): 240 machine-checkable items with unique answers, built author-neutrally from four balanced sources (each ecosystem's official test suite plus three frontier LLM proposers) and labeled by a non-circular two-implementation oracle. Evaluating six frontier models, we find systematic, predictable per-mechanism blind spots: a partial-comparator carry rule (>1.2 means >=1.3.0) traps every model on Cargo (near 60%), and although standard PEP 440 prefix matching is universal, on zero-pad/post-release corner cases GPT-5.1 collapses (0/26) while Claude stays at 97-100% (verified on a 67-item oracle-validated set). Opus significantly outperforms all other models, and Sonnet outperforms the OpenAI models (McNemar). The failures look more like an activation/application gap than a knowledge gap: injecting the rule or a light correct hint recovers most errors, whereas interval decomposition does not, and models are at ceiling on the basic forms of the same rules. An author-stratified analysis finds no statistically significant self-favoritism. Because the task is verifiable and a free, 100%-correct resolver exists, tool delegation reaches ~100%: coding agents should delegate version resolution to a resolver rather than reason about versions in-head.
