vLLM push-based NIXL connector cuts TTFT in disaggregated inference
Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM
Shifting KV-cache transfer coordination off the critical path removes a hidden latency tax in prefill-decode disaggregation, and the 3x TTFT gain applies to any team serving LLMs at scale on vLLM. Practitioners running inference on AWS can adopt the NIXL connector on EC2 P5en with EFA networking directly from the guide. Compared with pull-based transfer, the push design needs no receiver-side polling, which is what unlocks the latency savings.
vLLM's prefill-decode disaggregation splits LLM inference across engine instances and must transfer the KV cache between them. AWS introduced a push-based NIXL connector that moves transfer coordination off the critical path. In benchmarks on Amazon EC2 P5en instances with AWS EFA, the approach delivers up to 3x better TTFT and 30% better TPOT. The post explains how the connector works and how to deploy it in vLLM.
Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM
vLLM's prefill-decode disaggregation splits LLM inference across engine instances but must transfer the KV cache between them. This post introduces a push-based NIXL connector that moves transfer coordination off the critical path, delivering up to 3x better TTFT and 30% better TPOT in benchmarks on Amazon EC2 P5en instances with AWS EFA.
