这个系统把DeepSeek和Opus组合起来，用投机解码省了近一半API成本，还快了一倍，质量也有提升，而且开源可部署。
RLM-Cascade是一个代理层投机解码系统，在响应级别优化LLM API调用。它使用DeepSeek作为草稿模型、Opus作为验证模型，并通过轻量复杂度路由器选择路径。在Claude Code生产环境中，系统达到88.8%的草稿使用率，API成本相比直接使用Opus降低45.8%。P50延迟从3698毫秒降至2026毫秒，实现1.83倍加速。在20个Code/Math/Instruct任务基准上，RLM-Cascade通过率达100%，高于Opus的95%。
RLM-Cascade: Response-Level Speculative Decoding for Cost-Efficient LLM API Serving
We present RLM-Cascade, a proxy-layer system that applies speculative decoding at the response level to reduce LLM API costs without requiring model architecture access or a shared vocabulary. A fast, inexpensive draft model generates a candidate response; a capable verify model accepts, enhances, or is bypassed entirely depending on a lightweight complexity router. On a real-world agentic coding workload (Claude Code), RLM-Cascade achieves a draft-use rate of 88.8% across 125 production requests, reducing API cost by 45.8% relative to a direct Opus baseline. Counter-intuitively, the proxy also reduces end-to-end latency: median response time is 2,026 ms versus 3,698 ms for Native Opus -- a 1.83X speedup at p50 -- because the SKIPPED path (DeepSeek only, no Opus call) dominates the workload distribution. Quality matches or exceeds the Opus baseline: 100% pass rate on a 20-task Code/Math/Instruct benchmark versus 95% for Native Opus. We further describe a rule-based complexity router that selects the SKIPPED path for simple agentic turns and a hybrid tool-call strategy that bypasses the speculative pipeline for schema-critical tool-selection turns. RLM-Cascade is deployed in production as an enterprise AI infrastructure component and published as open source with a live metrics dashboard and Prometheus endpoint.