CatchBench基准测试为评估代理失败的可捕捉性提供了新的视角，涵盖了多个模型和配置，是了解代理失败捕捉的重要参考。
CatchBench提出了一种审计方法，针对代理失败的可捕捉性进行评估，涵盖运行前的声明配置（PRE）、运行中的增长前缀（LIVE）和完成后的跟踪（POST）。该基准测试了72个参赛者，包括规则扫描器、结构模型和11个LLM评委，覆盖9个模型家族和超过1187个声明配置。大部分结果未排序，其中47个对比未解决。该基准测试强调了标签过程的重要性，并报告了所有排序和预测。
CatchBench: When Can an Agent Failure Be Caught?
When can an agent failure be caught? An audit is usually limited by the record rather than by the method. CatchBench therefore puts one auditor's question to three information states: the declared configuration before a run (PRE), a growing prefix of its trace (LIVE), and the finished trace (POST). Prior benchmarks fix one of these states or vary the telemetry; to our knowledge none scores all three under one task-method interface. Each state admits different questions, so seven task contracts carry their own labels and metrics rather than one leaderboard. Four are evidential; three are Gold-derived mechanism diagnostics. The release scores 72 entrants, from rule scanners and structural models to eleven LLM judges across nine model families (GPT, Claude, Gemini, Gemma, Llama, Qwen, DeepSeek, Mistral, Nova), over 1187 declared configurations and 1162 recorded runs. Most of the arena does not order: 47 of 118 pre-declared contrasts separate, and the rest are published unresolved rather than ranked. The two sharpest results cut against our own data. One rule ignores every name and permission; it flags each capability declared after the first. On one of six configuration sources it reaches a perfect F1, so a score there measures how the corpus was built rather than how well a method reasons. Our admissibility bar then rejected one injected substrate and withheld evidential status from the other. A benchmark number is therefore not interpretable until the process behind its labels is published and tested for the shortcut it may leave. We report both, and regenerate every ordering from released predictions with no model call.