这篇论文的Talaria系统把工具代理场景下百亿参数模型的p50延迟从1000秒降到189秒，挺实用的加速方案。
Talaria是一个会话感知的无服务器多模型推理系统，针对工具使用代理的会话连续性优化。在单台TP=8服务器上，用30个SWE-Bench会话（960次调用）测试三个超百亿参数模型。相比轮询调度器（无会话预填充、主机KV恢复和D2D分段），p50会话完成时间从1000秒降至189秒（加速5.3倍），p95从2296秒降至867秒（加速2.6倍）。
Talaria: Session-Aware Serverless Serving of Hundred-Billion-Parameter LLMs
Serverless multi-model LLM systems multiplex popularity-skewed model catalogs over shared GPU pools, yet typically schedule each request independently. Tool-using agents break this abstraction: a session repeatedly calls an LLM across short tool gaps, carries a long reusable KV prefix, and is judged by session completion time (SCT). Load-only routing can separate a continuation from both its model and KV state, while round-based model multiplexing can delay even a correctly placed continuation until the target model's next slot. Both failures are especially costly for hundred-billion-parameter models: their weights constrain residency, while long-context KV is expensive to reconstruct or move. We present Talaria, a session-aware serverless multi-model serving system that makes session continuity a joint placement-and-admission decision. Its router ranks placements by model residency, KV locality, and instance pressure, while soft reservations account for likely returns in the last serving instance's admission budget. Session-prefill (SP) admits budget-eligible continuations before the active model slot closes. An instance-local substrate keeps HBM addresses stable, preserves host-restorable KV, and stages weights across model switches. On a single TP=8 server, we replay 30 SWE-Bench model-sessions (960 calls) over three models, each with more than 100B total parameters. Against an otherwise identical round scheduler with SP, host-KV restoration, and D2D staging disabled, Talaria cuts p50 SCT from 1000 s to 189 s and p95 from 2296 s to 867 s, speedups of 5.3x and 2.6x.