这论文给了一个能自动造VLM压力测试的框架，比现有基准更能暴露模型短板，做评测的可以看看。
SABRE是一个可扩展的自动化流水线，能把Markdown任务设计转化为结构化规格、图像和问答对，用于构建视觉语言模型压力测试。该流程通过自动化过滤和人工审查保证题目质量。作者用SABRE-Prior实例测试模型是否依赖世界先验而非视觉证据，包含600张图像和1000个问题。六个VLM在宏观平均准确率上仅达17.8%至31.3%，平均22.6%。SABRE-Counting和SABRE-Spatial的试点证明该工作流可扩展至其他压力测试场景。
SABRE: Scalable and Automated Benchmarking of VLMs under Stress
Vision-language models (VLMs) are improving rapidly, but benchmark development lags behind, making weaknesses hard to identify. Building stress tests is costly: samples must satisfy controlled conditions, remain answerable, and challenge current models. We present SABRE, a scalable, automated pipeline that converts a Test Primer (a Markdown Task Design with Data Schema) into structured specifications, generated or edited images, and question-answer pairs. Automated filtering removes candidates solved by a Filtering VLM, while human review verifies candidate validity and supports annotation correction and localized image repair. We instantiate SABRE-Prior to test whether VLMs follow visual evidence instead of relying on world priors -- learned expectations about familiar objects and scenes. Its 600 images and 1,000 questions span Context (unexpected entities in familiar scenes), Texture (counterfactual materials), Attribute (noncanonical component counts), and Language Elicitation (answers suggested by language but unsupported by the image). Across six VLMs, macro-average accuracy ranges from 17.8% to 31.3% (22.6% mean). A real-image Attribute control is comparably difficult for the Filtering VLM. SABRE-Counting and SABRE-Spatial pilots show that the workflow supports other stress-test settings. These results establish SABRE as a reusable framework for constructing and refreshing VLM stress tests rather than a single fixed benchmark.