Kog AI 把 GPU 推理的隐藏效率瓶颈挖出来了，做 LLM 推理优化的开发者可以直接关注他们的技术预览，看看 monokernel 和延迟张量并行能否复现到自己的模型上。
Kog AI 在 8× AMD MI300X GPU 上实现了 3000 tokens/s 的推理速度，在 8× NVIDIA H200 上达到 2100 tokens/s（FP16，无投机解码）。这一速度比常规低批次解码快 10-30 倍。其核心创新是将 LLM 解码视为内存流问题而非数学问题，通过 monokernel 技术将整个解码过程作为单个持久 GPU 程序运行，消除了内核启动、CPU 调度和中间内存往返的开销。他们还优化了同步机制和内存访问，并设计了 Laneformer 模型使用延迟张量并行技术。目前技术预览基于 2B 模型，但声称可扩展到大型 MoE 模型。
I had to test it myself to believe this unreal inf…
I had to test it myself to believe this unreal inference speed.
3,000 tokens/s for 1 user on standard datacenter GPUs.
They leveraged a hidden efficiency gap in how GPUs generate tokens.
@Kog__AI just achieved 3,000 tokens/s on 8× AMD MI300X GPUs and 2,100 on 8× NVIDIA H200 (FP16, no speculative decoding). Their tech preview is on a 2B model, and they show how their techniques will scale to large frontier MoE models at similar speeds.
That's a huge number because normal low-batch GPU decoding for 2B to 8B models is usually closer to 100 to 300 tokens/s per request, so Kog is claiming something like a 10X to 30X jump in the speed one user actually feels.
Their trick: they are getting the speed by treating LLM decoding as a memory streaming problem, not mainly a math problem.
For 1 user at batch size 1, the GPU is not doing big, efficient matrix-matrix work like in training or large-batch serving; it is repeatedly pulling the model’s active weights from high-bandwidth memory for each new token, so speed depends on how smoothly those weights keep flowing.
Normal inference stacks keep breaking that flow. They run many separate GPU programs for different parts of the model, move intermediate results through memory, wait at synchronization points, talk back to the CPU for scheduling or sampling, and then repeat this token after token.
Kog’s answer is to co-design 3 things that are usually tuned separately: the runtime, the low-level GPU code, and the model architecture.
The biggest engineering move is the monokernel, where the whole decode pass runs as 1 persistent GPU-resident program, including sampling, so the system does not keep stopping for kernel launches, CPU scheduling, and intermediate memory round trips.
They also rebuilt synchronization, because their own measurements say grid sync was eating around 35% of token-generation time; instead of making every compute unit wait at a broad barrier, each unit waits only for the exact data it needs.
On AMD MI300X, they also map memory access around the chiplet layout, because memory latency changes depending on which die makes the request.
Then their Laneformer model uses Delayed Tensor Parallelism, which lets cross-GPU communication happen in the background instead of blocking every layer.