这篇论文说用简单填充法就能让LLM推理结果确定，不用牺牲速度，比之前的方案快了不少，还在Qwen和DeepSeek上试过。
CoRun是一个面向LLM推理的调度系统，通过隔离预填充和固定形状批处理解码实现确定性输出。相比现有batch-invariant内核方案，CoRun无需牺牲性能即可保证结果一致。在Qwen和DeepSeek等模型上，CoRun吞吐量提升15%至324%，首token延迟平均降低51.8%，每token输出延迟平均降低48.6%。其核心洞察是大多数内核虽非批次不变，但具有位置不变性。
CoRun: Padding is Simple and Efficient for Deterministic LLM Inference
Despite fixed sampling parameters and random seeds, Large Language Model (LLM) inference exhibits output inconsistency, which undermines downstream tasks such as model evaluation and reinforcement learning. A major source of this nondeterminism is batch-dependent GPU execution: dynamic input shapes change kernel tiling and floating-point reduction orders. Existing systems address this problem with batch-invariant kernels, but these kernels restrict optimized tiling and split reductions, increasing more than 2$\times$ latency and reducing serving throughput by up to 74 %. This paper observes that although most kernels are not batch-invariant, they are position-invariant. Leveraging this property, we present CoRun, a scheduling-based system that achieves deterministic inference without requiring batch invariance. CoRun employs isolated prefill and fixed-shape batched decode to handle the two stages of LLM inference, respectively, leveraging CUDA graphs for efficient execution and simplified implementation. Experiments on LLMs with diverse architectures, including Qwen and DeepSeek, show that CoRun ensures determinism while improving throughput by 15-324 % over batch-invariant approaches, reducing time-to-first-token by 51.8 % and time-per-output-token by 48.6 % on average.