Notion搞了个新评测，用真实用户任务测模型，看谁成功率高又省时省钱，比传统benchmark更实用。
Notion发布了Knowledge Board评测项目。它从实时流量中随机抽取少量匿名数据，并在各模型间均匀分配。该评测衡量真实任务的成功率、所需时间和成本，而非依赖固定基准。其目的是帮助用户根据自身工作负载找到性价比合适的模型。
Most model evaluations rely on fixed, curated tasks. They’re useful, but they don’t show how models ...
Most model evaluations rely on fixed, curated tasks. They’re useful, but they don’t show how models perform on the messy, varied work people actually give them. That’s the motivation behind @NotionHQ ’s Knowledge Board. It uses a small, random, anonymized sample of live traffic, split evenly across models, to measure whether real world tasks are successfully resolved, as well as the time and cost required. Rather than offering a typical leaderboard, the project aims to answer a more practical question: Which model offers the right tradeoff for the work you’re trying to do? labs.notion.com/knowledge-board 💬 0 🔄 2 ❤️ 11 👀 1640 📊 3 ⚡