Palisade Research：衡量 agent 进步该看真实工作而非跑分
你现在每天用 agent 干哪些活？ https://t.co/CXziWnkJWk
Palisade 的 Jeff Ladish 说他自从 Fable 出来后每天用 agent 干活，还追问 Anthropic 工程师到底怎么用，观点挺实在的。
Palisade Research 的 Jeff Ladish 认为，与其看 40 个 AI benchmark，不如看 agent 能否自主完成真实经济价值的工作。他提到自己的 agent 使用频率在 Fable 模型出现前后发生明显变化，从偶尔用变成每天用。他回忆早期 METR 让软件工程师评估 AI 完成长时程任务的方式，并主张回到用真实工作当基准的思路。
你现在每天用 agent 干哪些活？ https://t.co/CXziWnkJWk
你现在每天用 agent 干哪些活？ x.com/MTSlive/status… MTS @MTSlive Forget the 40 AI benchmarks. Can the agents actually do the work? Palisade Research's @JeffLadish argues that's the real test of AI progress: "The biggest jump was the difference between pre-Fable and post-Fable to me was huge. I just went from using agents sometimes to using agents every single day." "Take us to Anthropic. Show us what is it like being an Anthropic engineer? How many agents are they using every day? What does that actually look like?" "I remember back in the day when the way we measured AI capabilities was mostly METR bringing in software engineers and having them do these long time horizon tasks. It was not like you have to look at these 40 benchmarks." "It was like, can the AIs autonomously do the work? That is the thing I most care about." "How do we get back to the world where the benchmark is mostly, let's take real economically valuable work and see if the AIs can do it autonomously." @PalisadeAI Your browser does not support the video tag. 🔗 View on Twitter 🔗 View Quoted Tweet 💬 1 🔄 0 ❤️ 1 👀 204 📊 1 ⚡
