Chinese contentMeta 公开 Cap-and-Track 推理调度设计,分类服务少用 43% H100
Meta 把自家生产环境跑了几个月的推理调度方案写成了论文,亮点是完全不需要预测输出长度,只靠后端拒绝信号自动跟容量。同一流量下省了 43% 的 H100,跑大规模推理服务的团队可以直接抄设计思路。
Meta 把自家生产环境跑了几个月的推理调度方案写成了论文,亮点是完全不需要预测输出长度,只靠后端拒绝信号自动跟容量。同一流量下省了 43% 的 H100,跑大规模推理服务的团队可以直接抄设计思路。
The non-obvious insight is that tail calibration, not average throughput, is the right knob for meeting streaming SLOs in production inference. Engineers building interactive LLM services or operating disaggregated prefill/decode stacks can apply its scheduling approach directly to reduce SLO violations without extra capacity. The evaluation covers open-weight models widely deployed in the region, including Qwen3-8B and DeepSeek variants, making the results directly comparable for teams benchmarking serving layers against vLLM-style baselines.