这条推文直指 Opus 5 和 Fable 5 的基准与体验反差，还爆出 Anthropic 新的蒸馏训练路径，以及 RLHF 被弱化导致的体验变化，值得看看。
Kun Chen 指出 Opus 5 在多项公开基准上超越 Fable 5，但在实际使用中远不如 Fable 5，认为现有基准已几乎无用。他信任基于私有数据集的领域特定评估。他还透露 Anthropic 在 5 系列中先训练 Mythos 再蒸馏到 Sonnet 和 Opus，导致 Sonnet 5 失败、Opus 5 评价混杂。Chen 表示 Claude 系列使用体验变差，Grok 和 Kimi 目前在“愉悦感”维度上更优。
非常吊诡的是，Opus 5 几乎在每一项指标上都超越了 Fable 5 也许这只能说明我们的所有指标都已经失效了 而且 Claude 系列的模型用起来的味道不太对劲，现在味道比较对的是 Kimi 和...
非常吊诡的是，Opus 5 几乎在每一项指标上都超越了 Fable 5 也许这只能说明我们的所有指标都已经失效了 而且 Claude 系列的模型用起来的味道不太对劲，现在味道比较对的是 Kimi 和 Grok 对 RLVR 而言，重要的只有结果本身，而人类的偏好并没有那么重要 对齐计划，失败了吗？ Kun Chen @kunchenguid opus 5 is a VERY interesting release for a few reasons 1. it showed that the general benchmarks we use today are almost completely useless now opus 5 is nowhere near fable in practical use, not even close. anyone who’s used it meaningfully can tell this very quickly after a few tasks. yet opus beats fable on many benchmarks i now trust domain specific benchmarks built with private datasets a lot more than the popular ones. perhaps the future is everyone running their own evals because the public ones are really not telling us much 2. it seems with the 5 series, anthropic is trying a new way of training models previously, the same generation of sonnet and opus were often released at the same time or sonnet comes out before opus, which indicates sonnet and opus were trained by separate pipelines in parallel with the 5 series, it was very clear that they trained mythos first, and then distilled it into sonnet and opus. it seems this approach has a big influence on the models seeing sonnet 5 being a flop and opus 5 getting pretty mixed reviews already, i’m not sure this is working out 3. “how pleasant is it to work with the model” used to be a strength in claude, but now it’s not. honestly, grok is my favorite right now on the “pleasant” dimension. kimi is not bad either it feels like both anthropic and openai are giving RLHF less care, in favor of scalable RL that’s machine verifiable this almost looks like AI is directing humans to build a world that’s more friendly for machines rather than humans, and most humans don’t even realize they are being manipulated to help with that almost every new generation of frontier models now talk more jargons, need more steering to do what you want, and are just less fun to work with if this continues, AI will start to speak their own language that looks like English but average humans can’t understand. they will choose to do things that their human user never asked for. are we already failing at alignment? 🔗 View Quoted Tweet 💬 1 🔄 0 ❤️ 5 👀 2608 📊 2 ⚡