研究团队用政府开放数据搞了个新测评，发现现在的大模型遇到多表格和外部知识就掉链子，专治各种表面功夫，推荐给关心模型真实落地能力的朋友。
现有数据分柝基准局限于小表格的事实检索，忽略多表、外部知识整合与探索性洞察。新发布的DataGovBench源自政府开放数据，包含Table QA（复杂分解问答与可视化）和Table Insight（探索性分析生成专家级发现）两项任务。实验表明，当前主流LLM（包括带智能体框架的系统）在这两项任务上表现显著不足，距离真实数据分柝需求仍有很大差距。该基准为推进LLM处理复杂分柝查询与数据洞察提供了挑战性测试。
Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities
Current benchmarks for evaluating Large Language Models (LLMs) in data analysis often fail to reflect real-world settings. They typically focus on fact retrieval from small tables and overlook the challenges of large multi-tabular datasets, external knowledge integration, and exploratory insight discovery. We introduce DataGovBench, a benchmark derived from governmental open data designed to evaluate LLMs in practical scenarios. The benchmark includes two tasks: Table QA that requires solving complex decomposable questions and producing textual answers or visualizations, and Table Insight that evaluates the ability of models to generate expert-level findings through exploratory data analysis. Comprehensive experiments with state-of-the-art LLMs, both with and without agentic frameworks, reveal significant performance gaps across both tasks. These results suggest that current LLM-based systems remain far from satisfying the demands of real-world data analytics. DataGovBench provides a challenging benchmark for advancing research on LLMs capable of both answering analytical queries and discovering insights from data. Code and sample data are available at https://github.com/SoHasegawa/datagovbench.