Papers

面向智能体工作负载的 LLM 推理平台验证式部署方法

Validation-driven Deployment of LLM Inference Platforms for Agentic Workloads

Why we picked it

做自托管推理的团队值得看:论文用数据说明录制回放比实时重跑稳定得多(1.18x vs 4.98x),还实测了 llm-d 上四个 MoE 模型在真实智能体负载下的真实并发和吞吐差距,工具和数据集都开源了。

一篇新论文提出面向智能体 LLM 服务的端到端评估流水线,覆盖从小规模测试、容量规划、预生产到生产的全生命周期验证,目标是帮助企业把智能体工作负载从封闭 API 迁移到自托管的开源权重模型部署。

Body

核心发现是流量回放的价值:回放已捕获的会话轨迹在 95 分位下输入 token 量与原始负载偏差仅 1.18 倍,而实时重跑偏差达 4.98 倍,说明少量录制会话即可产生可复现的单会话 token 负载。对 vLLM 的代码覆盖率分析还显示,实时、回放和合成三类工作负载覆盖的 Python 服务路径互不重叠,各有互补价值。

在一个在 llm-d 上部署四个大型 MoE 模型的案例研究中,捕获的真实智能体工作负载下模型只达到最优并发的一半左右,吞吐量约为公开轨迹测试结果的 60%。相关工具已开源并附带轨迹数据集,可直接用于自托管推理容量的验证。

Original · arXiv cs.AI

Validation-driven Deployment of LLM Inference Platforms for Agentic Workloads

Enterprises are moving agentic workloads from closed APIs to their own open-weight model deployments. For this reason, lifecycle-wide validation, from small-scale testing through sizing and pre-production to production, becomes all the more important. Typically, rollouts are evaluated with ad hoc tools whose accuracy is rarely checked. We present an end-to-end evaluation pipeline for agentic LLM serving and empirically ground the design choices that connect its stages. Our experiments show that replaying captured traces reproduces the per-session load of the original agent sessions: at the 95th percentile replay stays within 1.18x of the recorded input tokens while live re-runs reach 4.98x, so a small set of recorded sessions yields a repeatable per-session token load. A code-coverage analysis of vLLM further shows that the evaluated live, replayed, and synthetic workloads are complementary in this configuration: each exercises Python serving paths not reached in the measured runs of the others. In a case study rolling out four large mixture-of-experts (MoE) models on llm-d, experiments show that the target models reached about half the optimal concurrency, and correspondingly about 60% of the throughput, on captured workloads compared with public traces. We open-source the relevant tools and release a trace dataset.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include vLLM, llm-d, MoE, inference deployment.

Source attribution: curated from arXiv cs.AI (@Shai Gretz, Alon Halfon, Yoav Katz, Lena Dankin, Michal Jacovi, Rachel Tzoref-Brill, Kfir Toledo, Elron Bandel, Natan Ergas, Michal Malka, Pravein Govindan Kannan, Maroon Ayoub, Michal Shmueli-Scheuer).