How-toSelected76°9月26日 15:03

vLLM Adds Elastic Expert Parallelism for Live MoE GPU Scaling

Official accountPyTorch@PyTorch

Post· 2 sources

Why we picked it

Serving MoE models usually means over-provisioning GPUs to absorb traffic spikes, so resizing clusters without downtime directly cuts inference cost. Engineers running vLLM in production get a concrete mechanism via NIXL EP to scale expert-parallel workers on demand. The PyTorchCon 2026 session will be the reference for implementation details and known limitations.

vLLM now supports Elastic Expert Parallelism, allowing GPUs to be added to or removed from an active Mixture-of-Experts deployment with minimal interruption to serving.

Body

NVIDIA's Itay Alroy will present the architecture and implementation details at PyTorch Conference North America 2026.

The talk will cover what happens when EP size changes and how NIXL EP enables growing or shrinking clusters under live traffic.

Open challenges and the future roadmap for Elastic EP will also be discussed.

Original · PyTorch

Elastic Expert Parallelism in @vllm_project lets you add or remove GPUs from an active Mixture-of-Experts deployment during traffic with minimal interruption to serving and minimal downtime.

At #PyTorchCon North America 2026, Itay Alroy (@nvidia) will present “Elastic Expert Parallelism in vLLM,” covering the architecture, key implementation details, open challenges, and future roadmap for Elastic EP.

Alroy will also examine what happens when EP size changes and how NIXL EP enables grow/shrink under live traffic.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "How-to" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include vLLM, NVIDIA, Mixture-of-Experts, inference.

Source attribution: curated from PyTorch (@PyTorch). Cross-source corroboration: 2 independent sources have covered this development (including arXiv cs.LG), confirming industry-wide signal strength.

arXiv cs.LG09-25 17:20Source

#vLLM #NVIDIA #Mixture-of-Experts #inference #builders

Keep reading

Recent stories with the same tag or section

Models BuildersSelected05:32

Inkling 2 Small Runs on vLLM From Day Zero

Day-zero vLLM support plus single-node GB200 fit means operational cost of serving this model is far lower than typical 594B-scale deployments. Engineering teams running self-hosted inference should benchmark the 630 tok/s TP4 figure against their current MoE serving setups, since the DSpark speculator and NVFP4 experts are the levers behind it. Teams needing 1M-token multimodal context without multi-node clusters have a concrete new option to evaluate.

#Inkling 2 Small #Thinking Machines #vLLM #open-source models

Official accountvLLM@vllm_projectSource

Models BuildersMulti-source76°05:25

Chinese contentvLLM 第一时间支持 Thinking Machines 的 Inkling-2-Small 模型

Thinking Machines 发了个 500B 的多模态 MoE，vLLM 发布当天就能跑，还带了 FP4 量化和投机解码优化。想自部署大模型的人可以看看集成细节，不用等闭源 API。

#Inkling-2-Small #vLLM #Thinking Machines Lab #MoE

Event

Official accountvLLM@vllm_projectSource

Industry Builders02:18

Chinese contentPyTorch Conference 北美 2026 全部 146 场演讲视频上线

PyTorch 官方把北美 2026 大会的 146 场演讲视频全放出来了，vLLM、DeepSpeed、Ray 这些项目的作者亲自讲训练和推理优化，做工程的人可以挑 kernel 和推理那几场看，比零散博客系统多了。

#PyTorch #vLLM #DeepSpeed #Ray

Official accountPyTorch@PyTorchSource

Products BuildersMulti-sourceSelected01:07

Helion compile-time heuristics beat handwritten Triton kernels

Compile-time kernel selection removes one of the biggest practical barriers to adopting Triton beyond a handful of expert engineers: the tuning sweep that precedes every deployment. Inference and serving teams on H100 hardware can swap handwritten reduction and attention kernels for Helion-generated ones and see gains, not just parity, against vLLM's baseline. The 27% margin over production CUDA code is the concrete comparison that should push platform teams to benchmark Helion on their own workloads.

#Helion #Triton #vLLM #H100

Event

Official accountPyTorch@PyTorchSource

Products BuildersSelected00:51

AI Agents Help Integrate SGLang with IBM Spyre in a Week

The notable point is the integration velocity: a 1,500-line plugin plus AI coding agents replaced what is typically a multi-month porting effort. Teams evaluating non-GPU accelerators for LLM inference can now consider IBM Spyre as a drop-in option with vLLM-parity performance. Engineering leaders building serving infrastructure should watch how plugin-based hardware abstraction in SGLang lowers the cost of supporting new silicon.

#SGLang #IBM Spyre #vLLM #PyTorch

Official accountPyTorch@PyTorchSource

Models BuildersMulti-sourceSelected00:29

MiniMax H3-Image Tops Artificial Analysis Open-Weight Image Editing Rankings

The notable detail is the Vera Rubin support landing in vLLM-Omni ahead of the hardware's broad rollout, so early adopters can benchmark image-editing serving costs on next-gen GPUs from day one. Teams comparing open-weight image editing models should add H3-Image to their shortlist against FLUX-family alternatives. The 0.87-second 4K generation figure is a useful baseline for latency-sensitive production deployments.

#MiniMax #H3-Image #vLLM #image editing

Event

Official accountvLLM@vllm_projectSource