Latest

Open-source inference stacks closing the gap with a lab's proprietary serving setup changes the economics of self-hosting large MoE models. Teams weighing DeepSeek's API against running their own GPUs can use these Wide-EP and EPLB results to estimate revenue per GPU at off-peak prices. The GB200 and GB300 benchmarks also give infrastructure planners concrete throughput targets for the latest NVIDIA hardware.

官方账号vLLM@vllm_project7 sourcesSource

The same serving stack now optimises both NVIDIA and AMD paths, giving operators a concrete reason to benchmark MI355X deployments against GB300 rather than defaulting to CUDA. Teams running agentic inference traces can enable the mode through SGLang and expect P90 latency gains without changing models. The plugin system also means third-party kernels — starting with GLM-5.3, K3, and MiniMax M3 — will arrive without waiting for a core release.

官方账号LMSYS Org (SGLang)@lmsysorg11 sourcesSource

The design lesson is that tiered model routing beats a single flagship model: classification, routing and field extraction do not need top-tier reasoning, and cheaper Chinese models handle them well enough to cut inference cost by double digits. Engineering leads building internal agents can copy the four principles, especially single-pipeline architecture and shipping features behind default-off flags. It is also a concrete case study of small teams automating sales ops rather than headcount.

shao__meng@shao__meng8 sourcesSource

This release significantly improves inference efficiency through advanced attention mechanisms and memory management, enabling faster processing of large models. Developers working with LLMs in production environments will benefit from the optimized serving capabilities and experimental features like engine snapshots, particularly for models requiring high-throughput processing.

官方账号vLLM@vllm_project7 sourcesSource

A frontier-class open-weight MoE model with only 25B active parameters means lower inference cost per token, which directly affects how cheaply developers can run agentic coding workloads. Practitioners using Cline can test it against Kimi K3 and DeepSeek V4 Pro at no charge before 13 October and decide whether it earns a paid slot in their stack.

Cline@cline2 sourcesSource

This move significantly reduces dependency on Nvidia chips for AI training, potentially reshaping hardware competition in China. Developers can now optimize models for Ascend chips using DeepSeek's tools, though the licensing status of some projects limits immediate adoption. The timing coincides with Huawei's expected delivery of training chips this quarter.

rohanpaul_ai@rohanpaul_ai10 sourcesSource