Google Cloud 发布 PLE 基准：85 个真实工程任务，最佳智能体仅得 50.6%
Google Cloud 搞了个 PLE 基准，让 agent 干真实工程活，最好的模型也只有 50.6 分，比那些刷爆的教科书题难多了。
Google Cloud AI Research 推出名为 PLE（Physical-Engineering's Last Exam）的基准，包含 85 个覆盖电气、土木、结构、机械工程的任务。每个任务要求产出可计费的工程交付物，例如持证审查人会签字的设计包。任务环境从多格式项目语料中构建，须满足物理和建筑规范要求，部分任务由执业工程师端到端编写，其余由对抗性 Creator-Solver 管线生成并经专家验证。在 7 种公开 agent 框架下测试前沿模型，最佳配置平均得分 50.6%，单任务消耗 27M tokens、耗时 63 分钟。
Banger paper from Google Cloud AI Research on agents doing professional engineering work.
(bookmark it)
For professional engineering deliverables, the best agent configuration averages 50.6%, using 27M tokens and 63 minutes per task.
They introduce a benchmark called PLE (Physical-Engineering's Last Exam). It has 85 tasks across electrical, civil, structural, and mechanical engineering. Solving a task means producing a billable deliverable, such as a design package a licensed reviewer would sign off on, from a multi-format project corpus with physics and building-code requirements.
Some tasks are written end to end by practicing engineers. Others start from an engineer's problem statement, and an adversarial Creator-Solver agent pipeline builds the project environment, grader, and reference deliverables before experts validate them and make them harder.
Frontier models were tested under seven public agent harnesses. Most existing engineering benchmarks are textbook QA or toy design problems that leading agents already saturate.
https://t.co/pyHoLmZlFZ
Chat with Paper: https://t.co/kUKgle8wdR
