TraceBench发布了首个LLM智能体时间序列根因评估框架，包含数据集、实验结果和排行榜，tracebench.github.io可访问。
TraceBench是一个基于模拟的框架，用于生成受控的根因归因任务。研究团队使用三个可解释的机械系统生成任务，评估了四个LLM智能体在受控实验条件下的表现。结果显示，智能体从领域上下文中获益显著，主要通过数值控制台输出而非可视化探索数据。当要求生成Python脚本映射时间序列样本时，智能体表现通常较差。
TraceBench: Controlled Evaluation of LLM Agents for Time-Series Root-Cause Attribution
LLM agents are increasingly applied to anomaly detection and root-cause analysis in time-series observations collected from real-world systems; however, their performance on these tasks has not been systematically evaluated under controlled conditions. We introduce TraceBench, a simulation-based framework for generating controlled root-cause attribution tasks. In each generated task, an agent receives time-series observations produced by simulating a physical dynamical system and must determine whether a system parameter was altered during the simulation and, if so, which one. Using TraceBench, we generate tasks from three interpretable mechanical systems and systematically evaluate four LLM agents across controlled experimental conditions, yielding new insights into how these agents analyze time-series observations from dynamical systems. Our results show that agents benefit substantially from domain context and explore data primarily through numerical console output rather than visualizations. We also find that agents generally perform worse when required to produce a Python script that maps each time-series sample to a predicted root-cause label than when they submit predictions directly. We release our datasets, agent trajectories, experimental results, and a leaderboard on our website, tracebench.github.io.