Google 发布 SynthCorp 论文：为数据智能体生成可控基准
做数据分析智能体的人该看看。Google 用模拟业务事件自动生成评测数据，答案从日志自动推导，不用人工标注。链式多事件任务能让 SOTA 智能体翻车，比自己攒数据集靠谱得多。
Google 发布 SynthCorp 论文，针对静态数据智能体基准快速饱和、无法针对特定技能出题的问题，改用生成方式构造评测。系统在虚拟日历上模拟日常业务事件与异常中断，把这些事件写入相互一致的关系型表格、文档存储和数据目录。每个操作都会记录进模拟日志，任务的金标答案直接从日志推导并自动校验，省去人工标注。
首个版本包含 7 个数据领地，最多 1,000 张表，覆盖 5 个领域共 256 个任务。需要串联多个业务事件的任务仍能让当前最先进的智能体失败，说明这类多步推理是现成模型的短板。
对做企业数据智能体的团队来说，这套可控生成框架可以按技能难度定向出题，比固定数据集更难被过拟合，也更容易随模型能力提升持续更新评测。
Great paper from Google on building benchmarks for data agents.
Static data-agent benchmarks saturate quickly and can't target a specific skill.
SynthCorp generates the data instead. It simulates routine business events and disruptions on a virtual calendar, and those events fill relational tables, a document store, and a data catalog that stay consistent with each other.
Every operation goes into a simulation log, so the gold answer for each task is derived from the log and checked without manual annotation.
The first release has seven data estates with up to 1,000 tables and 256 tasks across five domains. Tasks that chain several business events together still make state-of-the-art agents fail.
https://t.co/rNDrl68o0m
Chat with Paper: https://t.co/HjPByJi267
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include SynthCorp, Google, benchmarks, agents.
Source attribution: curated from DAIR.AI (@dair_ai).
