以后新模型上线几小时就能看到评分了，比之前等几天快得多，而且数据来自真实用户偏好，可信度高。
LMSYS 发布 AutoEval，一种使用奖励模型基于数百万真实 Arena 用户偏好数据的新评估方法。它在文本、视觉、图像和代码领域均能与实时人工评估高度对齐，且将评估时间从数天缩短至数小时。AutoEval 使新模型能更快获得评分并直接显示在排行榜上。
Today we’re launching AutoEval: a new evaluation methodology that ranks models using reward models b...
Today we’re launching AutoEval: a new evaluation methodology that ranks models using reward models based on millions of real Arena user preferences. Highlights: - High-quality evaluation signals calibrated on real preference data - Strong alignment with live human evaluations - Evaluations that are several orders of magnitude faster (hours instead of days) - Support for Text, Vision, Image, and Code Arena AutoEval enables us to evaluate newly launched models much faster and share results with the community sooner. We’ll now show AutoEval estimated scores for new models directly on the leaderboard. More details in the thread. 🧵 💬 9 🔄 8 ❤️ 61 👀 5818 📊 17 ⚡