Chinese contentSGLang-Neuron 上线 AWS Deep Learning Container，支持在 Trainium 上部署大模型
SGLang 用户现在可以直接把推理和 RL rollout 跑到 AWS Trainium 上，接口不变、原生 PyTorch、AWS 官方容器开箱即用。想在 GPU 之外降推理成本或统一训练推理硬件的团队可以试试这条路。
#inference deployment
30 stories · 7 in 7 days · 18 in 30 days
Stories, products and papers tagged "inference deployment", newest first.
Nov 13
SGLang-Neuron brings SGLang serving to AWS Trainium
Serving stacks have mostly been GPU-bound, so a Trainium path for SGLang reduces the engineering cost of running inference on AWS's own silicon. Teams standardising on SGLang for RL rollout generation can now target Trainium instances directly. This matters for anyone evaluating Trainium against NVIDIA GPUs for cost or supply reasons, since the same APIs carry over.
CoreWeave webinar on hot-swapping RL checkpoints in production
The non-obvious point is that RL cadence, not compute size, becomes the serving bottleneck once weights update every few hours, and hot-loading removes the downtime tradeoff. ML platform engineers and inference teams running post-training pipelines should attend to copy the Dynamo and RL Rollouts setup directly, including cost figures. Comparing on-policy weight transition approaches before committing to an orchestration stack could save a costly rework later.
Nov 11
vLLM Adopts PyTorch LibTorch Stable ABI to Cut Rebuilds
The change removes a recurring pain point: version lockstep between inference engines and the PyTorch runtime. Platform engineers and ML infrastructure teams deploying vLLM can now mix torch versions with less risk, accelerating security and performance updates. Distributors and package maintainers benefit most, since rebuild effort drops sharply across release cycles.
Nov 10
Chinese contentAI 低延迟需求增长，城市内边缘数据中心角色受关注
这篇讲的是数据中心怎么摆位置的问题：训练放郊区，推理搬进市中心来压低延迟。做实时 AI 应用或者关心东南亚机房布局的人可以看看，思路和 AWS Local Zones 这类边缘方案是相通的。
Chinese contentShopify 在 PyTorchCon 上分享 Flywheel 持续学习系统
Shopify 在 PyTorchCon 上公开了他们的 Flywheel 系统，用 PyTorch 和 vLLM 把线上失败案例变成再训练数据，以比调用前沿模型便宜的成本做到更准。自己部署过模型的团队应该看看，比单纯堆 API 有参考价值。
Nov 7
Chinese contentBaseten 将在 AIE CODE 分享自托管推理工作流
用 Cursor 这类工具又想跑自己的开源模型？Baseten 的人教你把推理搬到自己的机器上，11 月 10 日旧金山见。
Nov 6
Chinese contentPyTorch Conference 2026 主旨演讲：开源是 AI 加速的核心驱动力
PyTorch 大会开场演讲，把开源讲成 AI 提速的关键，讲的是任意芯片任意云都能跑模型这套生态，看录像能了解工具链走向。
Nov 5
Chinese content阿里云灵骏真武 M890 SuperPOD 首日适配 Kimi K3.1 Pro
阿里云的灵骏 M890 SuperPOD 在 Kimi K3.1 Pro 发布当天就能跑这个 2.8T 参数的模型，用 vLLM 和 SGLang 部署，要自建推理集群的可以看看这套方案。
Chinese content华为昇腾 0 Day 适配 Kimi K3.1 Pro 推理部署
Kimi K3.1 Pro 发布当天昇腾就能跑推理了，2.8 万亿参数、百万上下文，用 vLLM Ascend 或 SGLang 可直接部署。
IT之家Source
Oct 23
MiniMax M3.1 Flash Runs in Production on vLLM
Production-grade serving of a MiniMax model through vLLM signals that the open-source inference stack can now carry commercial workloads with SLA guarantees. Teams evaluating cheaper open-weight serving options can look at Inferact's vLLM-based offering as a reference setup. Upstream contributions also mean the wider vLLM community inherits the reliability hardening that came out of this deployment.
Oct 21
Chinese contentPyTorchCon Demo Theater 展出 14 场厂商演示，涵盖推理、编译器与部署
PyTorchCon 下午还有 IBM、Modal、Red Hat 等 5 场演示，从 vLLM 集成到量化实测，日程表可以直接查。
Chinese contentPyTorchCon 2026：IBM、Google Cloud、Red Hat 联合演示 llm-d 生产级推理
IBM、Google、Red Hat 三家在 PyTorchCon 现场跑 llm-d 推理演示，做推理部署的可以看看这套开源方案
vLLM TorchTPU backend brings Google Cloud TPUs into vLLM ecosystem
Serving infrastructure built around NVIDIA GPUs now has a maintained TPU path, lowering the switching cost for teams that want Google's accelerator pricing and capacity. Contributors can apply for the free TPU pool to test and port models. Practitioners running vLLM in production should evaluate TPU serving costs against their current GPU fleet.
Oct 20
Chinese contentllm.d 1.0 发布，CoreWeave 作为创始成员支持开源推理生态
CoreWeave 参与的 llm.d 出 1.0 了，做大规模推理部署的可以看看这个开源项目。
Oct 19
vLLM cuts startup time 14–18% with precompiled Python bytecode
Image-level bytecode caching means faster pod spin-up and less wasted GPU time for anyone running vLLM at scale, especially on autoscaling or serverless deployments where cold starts hurt. Engineering teams deploying inference can pick it up by using release images from v0.32.0 or the CUDA nightlies, with no workflow change beyond the image tag.
Oct 16
Chinese content第四范式与瑞莱智慧开源 ModelSphere 异构大模型推理平台
第四范式和瑞莱智慧把自家的 LLM 推理平台按 Apache 2.0 开源了，一套框架能跑昇腾、摩尔线程、海光这些国产芯片，做推理部署的可以看看。
Oct 14
Intel Inference Microservices auto-tune vLLM on Xeon 4-6 CPUs
The tuning work most teams skip — NUMA layout, AMX activation, vLLM flags — is now baked into a container, so existing Xeon fleets can serve LLMs without days of manual optimization. Teams running inference on CPU-only infrastructure should try the detect-hardware and dry-run commands to preview the exact vLLM settings before deploying. It positions Intel's CPU lineup as a cheaper alternative to GPU-based serving for steady-state inference workloads.
IT之家Source
Oct 13
Chinese contentIntel Xe3P 架构 GPU D 系列获 vLLM 支持
Intel 新的 GPU D 系列现在能跑 vLLM 了，部署开源模型又多了一种非 NVIDIA 的硬件选择。
llm-d to share inference scaling update at PyTorch Conference
Beyond the benchmark framing, the notable shift is that open source inference serving now covers day-two operations — multi-tenant QoS and zero downtime — which were previously the gap against managed inference platforms. Platform and MLOps engineers running self-hosted LLM serving on Kubernetes stacks can evaluate llm-d as an alternative to vendor-managed serving. Details will land at the PyTorch Conference sessions.
Oct 4
Chinese contentCohere 发布 vLLM 线上活动报名链接
Cohere 挂了个 vLLM 主题的线上活动报名页，链接是 luma.com/vllm-y009，做推理部署的可以去看看议程。
Sep 28
Red Hat talk on enterprise agentic inference at PyTorch Conference
The gap between pilot demos and always-on enterprise serving is now being closed inside the PyTorch and vLLM ecosystems, meaning teams no longer need proprietary stacks for reliable agentic workloads. Platform engineers and ML infrastructure leads evaluating serving stacks can benchmark vLLM's enterprise features against alternatives. Developers of independent tools can also learn how Ecosystem Landscape membership raises their project's visibility.
Sep 26
vLLM Adds Elastic Expert Parallelism for Live MoE GPU Scaling
Serving MoE models usually means over-provisioning GPUs to absorb traffic spikes, so resizing clusters without downtime directly cuts inference cost. Engineers running vLLM in production get a concrete mechanism via NIXL EP to scale expert-parallel workers on demand. The PyTorchCon 2026 session will be the reference for implementation details and known limitations.
Aug 7
Chinese contentAmazon SageMaker Python SDK v3新增LLM推理推荐
AWS把LLM优化搬进了SageMaker的notebook，跑完基准测试直接给你部署建议，不用再手动调参了。
Jul 28
Chinese contentVercel AI Gateway 区域推理上线，请求可锁定美国或欧盟
Vercel 给 AI Gateway 加了区域推理，一个参数就能把请求锁在美国或欧盟，所有模型通用，省心又合规。
Jul 20
Jun 12
Chinese contentGLM-5 大规模部署中的 Scaling Pain 调试与优化
大模型从实验到生产，Scaling Pain 是绕不过的坎。做推理部署的工程师，这篇博客里的 KV Cache 竞态和 HiCache 同步问题很可能你也会遇到，建议直接收藏。
May 30
Industry11:09
Chinese content黄仁勋称Fireworks为“AI工厂的台积电”黄仁勋的比喻点明了AI基础设施服务正在成为新产业链的关键环节，做AI部署和推理优化的团队值得关注Fireworks的模式。
May 29
