Anyscale session covers Ray Serve LLM deployment and inference scaling
The session bundles inference infrastructure decisions most teams learn the hard way: when disaggregated prefill/decode and KV-cache-aware routing pay off, and whether self-hosting beats token-based API pricing. Engineers deploying open-source models with vLLM or SGLang can register to get a hands-on walkthrough of Ray Serve LLM and agentic-platform integration. Registration is open now via the Anyscale link.
Anyscale AI engineer Marwan will lead a session on building and scaling LLM serving with Ray Serve. The session covers Ray Serve LLM deployment on engines like vLLM and SGLang, plus optimizations including KV cache, paged attention, prefix caching, and continuous batching. Advanced topics include KV-cache-aware routing, prefill/decode disaggregation, and wide expert parallelism for mixture-of-experts (MoE) models. It also demonstrates deploying custom LLMs on Anyscale and integrating them with agentic platforms such as Claude Code, Cursor, and Codex, with a cost comparison of self-hosted GPUs, subscriptions, and token-based API billing.
In this session, @Marwan1112, AI Engineer at Anyscale, will walk through: - Building Ray Serve applications with deployments, composition, and load-aware request routing - Deploying and scaling LLM endpoints with Ray Serve LLM on engines like @vllm_project and @sgl_project - Optimizing the inference engine with the KV cache, paged attention, prefix caching, and continuous batching - Scaling frontier models with KV-cache-aware routing, prefill/decode disaggregation, and wide expert parallelism for mixture-of-experts (MoE) models - Deploying Custom LLMs on Anyscale and Integrating Them with Agentic Platforms (Claude Code, Cursor, Codex) - Comparing costs across self-hosted GPUs, subscriptions, and token-based API billing.
Register: https://t.co/uKeb1R6hQ5
