论文精选

SkillsBench 研究:技能可用性显著提升 LLM Agent 任务成功率,呈现粒度影响有限

Skill Availability and Presentation Granularity in Large-Language-Model Agents: A Controlled SkillsBench Study

精选理由

做 LLM Agent 开发的团队终于有了实证依据:给智能体塞技能文档比纠结怎么写更管用。建议直接参考这个实验设计来优化自己的 RAG 或工具调用策略。

AI 摘要

一篇来自 arXiv 的论文系统研究了技能文档的呈现粒度对大型语言模型智能体任务成功率的影响。实验基于 SkillsBench 基准,包含 30 个领域平衡的任务,测试了 GPT-5.5 和 DeepSeek V4-Flash 两种模型。结果显示,提供技能文档相比无技能条件,任务平均通过率提升 18 到 36 个百分点,效果显著。然而,技能文档的抽象程度(低抽象 vs 高抽象)以及是否包含示例对成功率的影响很小且统计上不显著。该研究表明,技能可用性是关键因素,而呈现细节的调整影响有限且依赖模型。

原文 · arXiv: DeepSeek

Skill Availability and Presentation Granularity in Large-Language-Model Agents: A Controlled SkillsBench Study

Skill documents provide procedural knowledge to large-language-model agents at inference time. This article studies whether the presentation granularity of controlled skill knowledge changes downstream task success. The experiment uses a pinned SkillsBench version, a 30-task domain-balanced subset validated by official oracle runs, two reasoning-enabled model configurations, six skill conditions, and five trials per task-condition-model cell. Skill availability is the clearest empirical signal. Relative to no skill, skill conditions increase task-mean pass rate by 26.7 to 36.0 percentage points for GPT-5.5 and by 18.0 to 26.0 percentage points for DeepSeek V4-Flash. The final data contain 1,800 rows, with 900 rows for each model. The task is the inference unit. Five trials are aggregated within each task-condition-model cell before paired contrasts are estimated over 30 tasks. The primary presentation contrasts are smaller and uncertain. Low-abstraction guidance differs from high-abstraction guidance by +0.7 percentage points for GPT-5.5 and -6.7 percentage points for DeepSeek V4-Flash, with both 95% bootstrap confidence intervals crossing zero. Adding one worked example to medium-abstraction guidance differs from the no-example variant by +0.7 and +1.3 percentage points. Mean-reward robustness checks preserve the same substantive conclusion. In this controlled subset, skill availability is associated with higher success than no skill, while the tested presentation-granularity changes yield small, uncertain, and model-dependent effects.