ModelsSelected

Y Combinator 创业公司 Synthetic Sciences 发布 FrontierML 基准,5 个前沿模型无一达标

Why we picked it

25 个 ML 研究题,5 个前沿模型全挂,因为评分目标被藏起来,Agent 只能自己造测试、自己判断测试靠不靠谱,还给 12 小时和真 H100 随便跑。做 Agent 评测或搞 ML 研究自动化的团队应该看看它的任务闭环怎么设计的。

Synthetic Sciences 是一家 Y Combinator 孵化的创业公司,其推出的 FrontierML 基准包含 25 个机器学习研究问题,实测的 5 个前沿 AI 模型没有任何一个在任何问题上达到强结果门槛。该基准的核心设计是把评估目标隐藏起来,Agent 拿不到自己将被打分的标准,必须自行设计测试并判断这些测试的可信程度。

Body

每个任务以固定流程运行:Agent 在 Lab0 环境中获得一个开放式 ML 问题和 12 小时时限,先自行设计进度代理指标,再在真实 H100 GPU 上跑实验并校准该代理指标的可靠性,最后提交模型或方法,由 FrontierML 在 held-out 评估集上打分。

Synthetic Sciences 认为第三步——校准代理指标的可信度——正是实证研究的主要发生地,而评分很大程度取决于此。这实质上是检验 AI Agent 能否在完全看不到分数的情况下独立完成机器学习研究。

Original · rohanpaul_ai

Synthetic Sciences, a Y Combinator startup, has launched FrontierML, a benchmark so hard that none of the 5 frontier AI models it tested hit the strong-result bar on any of its 25 ML research problems.

They hide each objective, so agents must build their own tests and decide how much to trust them.

This is a test of whether AI agents can do machine learning research on their own without ever seeing the score they’ll be graded on.

Each agent gets 12 hours and real GPUs, and has to work out for itself how to measure progress.

FrontierML runs each task as a loop with the objective kept out of reach:

> The agent gets an open-ended ML problem and 12 hours in Lab0 > It designs a local proxy for progress > It runs experiments on H100s and calibrates how far that proxy can be trusted > It submits a model or method > FrontierML scores it on a held-out evaluation

Synthetic Sciences argues step 3 is where much of empirical research happens. Here, a lot of the score rides on it.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Models" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include FrontierML, Synthetic Sciences, Y Combinator, benchmarks.

Source attribution: curated from rohanpaul_ai (@rohanpaul_ai).