研究揭示大语言模型在长流程操作推理上的短板
Tasks over Application Manuals: Revealing Gaps in Long-Horizon Procedural Reasoning for Language Models
朋友，这篇论文挺有意思的，它指出现在的大模型在处理长流程操作时很吃力，比如遵循几十页的规则手册，准确率很低，这提醒我们不要过度依赖它们做这类复杂任务。
这篇论文提出了一种新基准（TAM），用于测试大语言模型在遵循长流程手册时的能力。他们在医疗编码和联邦量刑两个真实领域构建了任务，发现即使使用GPT-5等模型，在ICD-10-CM编码任务上的准确率也仅为1%，在量刑任务上为15.5%，表明当前基准可能高估了LLM的推理能力。
Tasks over Application Manuals: Revealing Gaps in Long-Horizon Procedural Reasoning for Language Models
Large language models (LLMs) have achieved strong performance on a wide range of natural language tasks, and recent benchmarks suggest that they are increasingly adept at multi-hop reasoning. However, these benchmarks are typically short-horizon, requiring only a small number of retrieval or inference steps, and provide limited evidence of reliability on real-world tasks that involve following manuals spanning hundreds of pages with complex, interdependent guidelines. In this paper, we introduce Tasks over Application Manuals (TAM), a benchmark for evaluating long-horizon procedural reasoning. We construct TAM by curating real-world tasks from two domains: ICD-10-CM clinical coding (mapping medical conditions to diagnostic codes) and U.S. federal sentencing (computing crime sentencing guideline outcomes, specifically offense levels), with human-validated labels. Each task requires following an authoritative manual with tens of thousands of rules and executing a sequence of interdependent steps across different sections to produce an exact answer. We evaluate general-purpose prompting approaches, including retrieval-augmented generation, ReAct-style prompting, and an agent-harness baseline on GPT-5, and find that the best exact-match performance remains extremely low: 1% on ICD-10-CM coding and 15.5% on sentencing tasks. These results show that current benchmarks may overestimate LLM reasoning ability and miss a key challenge: reliably following long, rule-based procedures. The complete TAM data and code are publicly available.
