VBR新KV缓存格式：动态逐层量化优化VRAM使用

精选理由

SpiritBun 搞了个新 KV 缓存格式 VBR，逐层动态量化，在显存限制内榨出最高质量，已开源可用。

AI 摘要

Clement Delangue 转推了 SpiritBun 的 VBR（Variable Bit Rate）KV 缓存格式。该格式在会话增长过程中逐层动态量化 KV 缓存，在 VRAM 限制内最大化生成质量。目前代码已合入主分支并可用。

原文 · Clement Delangue

is this useful?

is this useful? buun @spiritbuun Introducing a new KV format: VBR (Variable Bit Rate) VBR dynamically quantizes your KV cache layer-by-layer as your session grows, giving you the highest quality possible within your VRAM constraints. This is my dream format. Pushed to master. Available now. 1/15 🧵 Your browser does not support the video tag. 🔗 View on Twitter 🔗 View Quoted Tweet 💬 5 🔄 1 ❤️ 10 👀 4107 📊 5 ⚡

查看原推