NVIDIA这篇博文把长上下文模型变慢的根源讲透了，训练前选对4个架构参数，服务成本能差很多。
NVIDIA在AI Model Co-Design系列中发文指出，长上下文模型的推理速度在训练前就已被架构选择决定。随着上下文窗口扩大，注意力机制在推理成本中的占比快速上升，逐渐成为主要负担。文章梳理了4个决定性能上限的架构参数：group size、head dimension、KV-cache size和并行策略。这些选择同时影响系统吞吐量和单个用户的响应速度。
A long-context model's serving speed is largely decided before training starts. Attention used to b...
A long-context model's serving speed is largely decided before training starts. Attention used to be a small part of a model's inference cost, but its share grows sharply as context windows expand. Once attention is the majority of the work, faster kernels stop being enough, and the shape of the attention mechanism itself sets the ceiling. We're continuing our series on AI Model Co-Design with a post on the four architecture choices that set that ceiling: group size, head dimension, KV-cache size, and parallelism strategy. It also covers how the right choices improve both system throughput and per-user responsiveness. You can read the full piece here: vist.ly/5dfk2 💬 5 🔄 2 ❤️ 20 👀 2105 📊 7 ⚡