Kog@AI 把推理速度从 300 拉到 3000 tokens/s，做模型部署和推理优化的团队值得研究他们的内存流方法，直接看原文能学到如何消除 GPU 瓶颈。
Kog@AI 在 8×AMD MI300X GPU 上实现了 3000 tokens/s 的推理速度，8×NVIDIA H200 上为 2100 tokens/s（FP16，无推测解码），远超高端 GPU 通常的 100-300 tokens/s。他们将 LLM 解码视为内存流问题，通过将整个 token 生成循环保留在单个持久 GPU 程序中，消除了内核启动、CPU 调度和中间内存写入的开销。同时，通过让每个计算单元只等待所需数据，并针对 MI300X 的芯片拓扑优化内存访问，减少了同步浪费。模型架构延迟了张量并行通信，使 all-reduce 在后台进行而不阻塞每一层，这要求运行时、GPU 代码和模型设计协同优化。这一突破展示了通过软硬件协同设计大幅提升推理效率的潜力。
Some truly massive inference numbers here. @Kog__…
Some truly massive inference numbers here.
@Kog__AI just achieved 3,000 tokens/s on 8× AMD MI300X GPUs and 2,100 on 8× NVIDIA H200 (FP16, no speculative decoding) with a 2B model.
For comparison, typical GPU decoding speed for 2B to 8B models on high-end GPUs is around 100 to 300 tokens/s per sec.
They achieved it by treating LLM decoding as a memory-streaming problem: keep the whole token-generation loop inside one persistent GPU program, so kernel launches, CPU scheduling, intermediate memory writes, and sampling interruptions mostly disappear.
Then they cut synchronization waste by making each compute unit wait only for the exact data it needs, while mapping memory access to the MI300X’s chiplet topology so the GPU stops paying avoidable cross-die latency.
Finally, their model architecture delays tensor-parallel communication so all-reduce work happens in the background instead of blocking every layer, which is why the runtime, GPU code, and model design all have to be co-designed.