DAIR.AI发布了关于智能体驯服的论文，探讨了评估中的问题，值得一看。
DAIR.AI发布论文《There Is No Neutral Harness》，讨论了智能体驯服评估中的问题。十二个开放权重模型在26种可辩护的驯服配置下，对ARC、HellaSwag、MMLU和TruthfulQA的3,679个问题进行回答。gemma4-31b在不同驯服配置下的准确率从31%到89%不等。论文链接：arxiv.org/abs/2608.21382
Great paper on agent harnesses.
Great paper on agent harnesses. DAIR.AI @dair_ai // There Is No Neutral Harness // Great work discussing some of the issues in harness evaluation. Twelve open-weight models answer the same 3,679 items from ARC, HellaSwag, MMLU, and TruthfulQA under 26 equally defensible harness configurations. Items, weights, and greedy decoding stay fixed. Only option order, prompt wording, and whether the answer is read from generated text or per-option likelihoods change. gemma4-31b lands anywhere from 31% to 89% depending on the harness alone. On the items that two adjacent models both answer stably, the pair is tied. Config-fragile items carry 95.7% of the gap between them, and four of the twelve models reach rank one under some configuration. Item discrimination, the property benchmark-compression methods maximize when picking a representative subset, correlates with fragility. Compressed benchmarks are selecting for the items most sensitive to configuration. Paper: arxiv.org/abs/2608.21382 Track more trending AI papers in our academy: academy.dair.ai 🔗 View Quoted Tweet 💬 2 🔄 2 ❤️ 11 👀 1422 📊 4 ⚡