固定GPU预算下临床 LLM 服务的容量规划研究
Capacity Planning for Resource-Constrained Clinical LLM Serving
这篇论文用 24 组配置实测告诉你：GPU 预算有限时，先想好买什么型号的卡，路由只是 secondary。做临床或垂直场景部署的可以看。
一篇 arXiv 论文在四 GPU 的受控环境中研究临床 NLP 负载下 LLM 服务的容量分配问题，负载覆盖短文本事实核查、生物医学语义匹配和长上下文临床摘要。实验系统改变机型组合、路由策略和负载水平，共测试 24 种配置，结果显示机型组合对首 token 时间、延迟和长上下文覆盖率的影响大于路由策略。全小模型组合吞吐和响应最好但长上下文覆盖明显下降，混合机型则可能因大模型容量不足形成瓶颈。在针对瓶颈的专项评估中，容量感知路由降低了平均延迟并提升吞吐，代价是长上下文覆盖率小幅下降。
Capacity Planning for Resource-Constrained Clinical LLM Serving
Deploying large language models (LLMs) under a fixed accelerator budget requires jointly deciding how computational capacity is allocated across model sizes and how incoming requests are routed across that capacity. Although prior work has extensively studied LLM routing and serving efficiency, the relative importance of fleet composition and routing under resource constraints remains less understood. We investigate this question using a controlled four-GPU serving environment with heterogeneous clinical NLP workloads spanning short factual verification, biomedical semantic matching, and long-context clinical summarization. We systematically vary fleet composition, routing policy, and offered load while holding the workload trace and hardware budget fixed. Across 24 configurations, descriptive main-effect shares attribute more variation to fleet composition than to the evaluated routing policies in time to first token, latency, and long-context coverage. All-small fleets provide the best responsiveness and throughput but substantially lower long-context coverage, whereas larger-model capacity improves coverage at the cost of serving efficiency. Heterogeneous fleets provide intermediate operating points but can introduce strong-model bottlenecks when large-model capacity is scarce. In a focused evaluation of this bottleneck, capacity-aware routing reduces mean latency and increases throughput while incurring a small reduction in long-context coverage. Within this measured regime, the results support prioritizing capacity allocation, with routing managing demand within the provisioned capacity.