AITOPAI热报
How-toSelected

SGLang adds scoring API and multi-item scoring for decision models

Official accountLMSYS Org (SGLang)@lmsysorg
Post
View the post
Why we picked it

Classification and ranking workloads waste GPU cycles recomputing shared context per candidate, and MIS cuts that cost by an order of magnitude at higher candidate counts. Developers serving scoring endpoints for agent action selection or reranking can switch to /v1/score and measure the p95 gains directly using the benchmark commands in the blog. The LinkedIn contribution shows production demand for this pattern beyond the LMSYS ecosystem.

LMSYS published a blog on scaling JEV-like decision models with SGLang, contributed to by the LinkedIn team.

Body

SGLang's new /v1/score endpoint returns scores for exact requested labels such as Yes/No or A/B/C, addressing cases where Generate plus top-k logprobs can drop the needed label.

Its multi-item scoring (MIS) computes shared context once while keeping each candidate isolated.On Qwen3-8B, 16-candidate p95 latency dropped from 54.

1 ms with Generate to 20.6 ms with MIS, and MIS p95 stayed under ~100 ms under load on Qwen3-0.

6B versus seconds for Generate and SIS.

Original · LMSYS Org (SGLang)

🚀 New blog: Scaling JEV-like decision models with SGLang

Decision models need a score, not prose. Classification, ranking, and agent action selection all ask the same thing: which option wins?

Open-Jev, for example, scores each candidate separately with a Yes/No prompt. Serving this well raises two issues: Generate + top-k logprobs can drop the label you need, and the shared context can be recomputed for every candidate.

SGLang addresses both: - /v1/score returns scores for the exact labels you request (Yes/No, A/B/C) - Multi-item scoring (MIS) computes the shared context once and keeps each candidate isolated - MIS latency stays nearly flat from 2 to 16 candidates, with 16-candidate p95 on Qwen3-8B dropping from 54.1 ms (Generate) to 20.6 ms (MIS) - MIS p95 stays under ~100 ms as load rises on Qwen3-0.6B, vs. seconds for Generate and SIS

Huge thanks to the @LinkedIn team for contributing! Benchmarks and launch commands in the blog 👇

Industry Coordinates & Source Verification

This report is categorized under AITOP's "How-to" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include SGLang, Qwen3, LMSYS, inference optimization.

Source attribution: curated from LMSYS Org (SGLang) (@lmsysorg).

#SGLang#Qwen3#LMSYS#inference optimization#coding assistants

Keep reading

Recent stories with the same tag or section
ProductsBuildersOfficial06:27
Chinese contentSGLang-Neuron 上线 AWS Deep Learning Container，支持在 Trainium 上部署大模型

SGLang 用户现在可以直接把推理和 RL rollout 跑到 AWS Trainium 上，接口不变、原生 PyTorch、AWS 官方容器开箱即用。想在 GPU 之外降推理成本或统一训练推理硬件的团队可以试试这条路。

#SGLang#AWS#Trainium#推理部署
Event
Primary sourceAWS Machine Learning Blog@Alex NailsSource
ProductsBuildersMulti-sourceSelected06:06
SGLang-Neuron brings SGLang serving to AWS Trainium

Serving stacks have mostly been GPU-bound, so a Trainium path for SGLang reduces the engineering cost of running inference on AWS's own silicon. Teams standardising on SGLang for RL rollout generation can now target Trainium instances directly. This matters for anyone evaluating Trainium against NVIDIA GPUs for cost or supply reasons, since the same APIs carry over.

#SGLang#AWS Trainium#TorchNeuron#inference serving
Event
Official accountAmazon Science@AmazonScienceSource
IndustryBuildersMulti-sourceSelected02:57
SGLang Consortium formed with NVIDIA, AMD, Google and Meta

Hardware vendors co-funding a single open-source inference engine reduces the risk that serving optimizations remain locked to one chip ecosystem, which matters for teams planning multi-vendor GPU fleets. Engineering leads running large-scale inference can track the consortium's roadmap to anticipate support for new models and distributed serving features. Organizations in Southeast Asia building inference capacity on heterogeneous hardware will benefit from upstream optimizations landing in SGLang rather than vendor-specific forks.

#SGLang#NVIDIA#AMD#open-source models
Event
Official accountLMSYS Org (SGLang)@lmsysorgSource
ProductsBuildersSelected02:43
LMSYS releases SGLang v1.0 and Miles v0.2 at SGLang Summit

The pairing matters: a hardened v1.0 inference engine plus a production-focused RL framework gives teams a single open-source alternative to closed serving and post-training stacks. Engineers building agent or chat products on self-hosted GPUs can benchmark SGLang v1.0 against vLLM for throughput and memory gains. Teams running RL post-training can evaluate Miles v0.2 for their specific hardware mix rather than stitching together ad-hoc pipelines.

#SGLang#LMSYS#Miles#reinforcement learning
Official accountLMSYS Org (SGLang)@lmsysorgSource
ProductsBuildersSelected00:51
AI Agents Help Integrate SGLang with IBM Spyre in a Week

The notable point is the integration velocity: a 1,500-line plugin plus AI coding agents replaced what is typically a multi-month porting effort. Teams evaluating non-GPU accelerators for LLM inference can now consider IBM Spyre as a drop-in option with vLLM-parity performance. Engineering leaders building serving infrastructure should watch how plugin-based hardware abstraction in SGLang lowers the cost of supporting new silicon.

#SGLang#IBM Spyre#vLLM#PyTorch
Official accountPyTorch@PyTorchSource
How-toBuildersSelected01:21
SGLang Megakernels Lift Muse Spark Open Decoding to 1,109 Tokens/s

Kernel-launch overhead, not compute, is often the real ceiling in multi-GPU inference, and this write-up shows concrete numbers and code proving it. Engineers building or tuning serving stacks on Blackwell-class hardware get CUDA/PTX examples and measured timelines they can adapt. Teams benchmarking against vLLM or TensorRT-LLM deployments will find the launch-count reduction a useful reference point.

#SGLang#Muse Spark Open#GB300#inference optimization
Official accountLMSYS Org (SGLang)@lmsysorgSource