研究人员搞了个叫OenoBench的葡萄酒知识测试题库，用这个题库测了16个大模型，发现o3考得最好，DeepSeek R1在推理题上表现突出，还发现模型对自己出的题有偏好，挺有意思的。
研究团队发布OenoBench，一个包含3266道选择题的葡萄酒领域知识评测基准。该基准基于38104个原子事实构建，涵盖六个支柱和四个难度层级。评估16个前沿模型配置，发现整体准确率在53%-84%之间，o3以83.6%领先。DeepSeek R1在推理模式下提升6.8个百分点，而Claude Opus和Gemini Pro无提升。Anthropic在自身问题上偏好度+9pp，Google则-8pp。开源模型与专有推理模型共享成本-准确率帕累托前沿。所有配置在闭卷可解项目上平均提升33pp。
OenoBench: A Wine-Domain Benchmark for Knowledge-Grounded Evaluation of Large Language Models
We introduce OenoBench, a wine-domain knowledge benchmark of 3,266 multiple-choice questions across six pillars (regions, grape varieties, viticulture, winemaking, producers, business) and four difficulty tiers. The corpus is built from 38,104 atomic, source-anchored facts extracted by 35 provenance-verified scrapers from government registries (INAO, TTB, OIV), peer-reviewed journals, and Wikipedia/Wikidata. Our methodological contribution is an LLM-driven pipeline in which language models reformat verified facts and audit the result, but never serve as the source of truth: every claim traces to a URL, every question is generated by one of five strategies across five generator families, and every question is scored by a nine-agent audit calibrated against a human gold sheet via Cohen's $κ$. Evaluating sixteen frontier configurations, we find: (i) overall accuracy spans 53%-84%, led by o3 at 83.6%; (ii) reasoning-mode lift concentrates in DeepSeek R1 (+6.8pp) and is absent in Claude Opus and Gemini Pro; (iii) Anthropic shows +9pp self preference on its own questions while Google shows -8pp inverse preference; (iv) frontier open-weight models share the cost-vs-accuracy Pareto frontier with proprietary reasoning models; and (v) every config gains around 33pp on closed-book solvable items, revealing a parametric-recall ceiling that only the contextual slice avoids. We release corpus, audit findings, human-review app, and construction code under CC-BY-SA-4.0.