长上下文 LLM 推理的内存瓶颈终于有了实用解法——OSCAR 在 2-bit 量化下几乎不损失精度，做长文档/多轮对话推理的团队可以直接集成，显著降低硬件成本。
Together AI 开源了 OSCAR，一种面向长上下文 LLM 推理的 INT2 KV 缓存量化方法。与依赖数据无关的 Hadamard 变换不同，OSCAR 通过离线估计注意力感知的协方差结构，为键和值分别推导旋转矩阵。在 Qwen3-4B-Thinking-2507 和 Qwen3-8B 上，OSCAR 以每 KV 元素 2.28 比特的精度，将 BF16 精度差距分别缩小至 3.78 和 1.42 分。该方法可实现约 8 倍的 KV 内存缩减，并在 100K 上下文长度下带来最高 3 倍的解码加速。
Together AI Open-Sources OSCAR: An Attention-Aware 2-Bit KV Cache Quantization System for Long-Context LLM Serving
Together AI has released OSCAR (Offline Spectral Covariance-Aware Rotation), an INT2 KV cache quantization method for long-context LLM serving. Unlike prior rotation-based approaches that apply data-oblivious Hadamard transforms, OSCAR derives separate rotations for keys and values from attention-aware covariance structures estimated offline. At 2.28 bits per KV element, OSCAR reduces the BF16 accuracy gap to 3.78 points on Qwen3-4B-Thinking-2507 and 1.42 points on Qwen3-8B, while delivering approximately 8× KV memory reduction and up to 3× decode speedup at 100K context length. The post Together AI Open-Sources OSCAR: An Attention-Aware 2-Bit KV Cache Quantization System for Long-Context LLM Serving appeared first on MarkTechPost .