精选理由
AWS官方教你用Curvine在HyperPod上搞分层KV缓存，省GPU内存还能提速，跑大模型推理的可以看看。
AWS博客介绍了一种在Amazon SageMaker HyperPod上使用Curvine构建分层KV缓存的方法。该方法将KV缓存扩展到共享分布式NVMe池中，使副本能以接近本地磁盘的速度复用缓存。这解决了大规模LLM推理中GPU实例过大或首token延迟高的问题。通过使用成本更高效的实例，该方法在保持性能的同时降低了推理成本。
原文 · AWS Machine Learning Blog
Tiered KV cache for large LLMs on Amazon SageMaker HyperPod with Curvine
Running large language model inference at scale forces a KV cache trade-off: oversized GPU instances or slow time-to-first-token. This post builds a tiered KV cache on Amazon SageMaker HyperPod that extends the cache into a shared, distributed NVMe pool with Curvine, so replicas reuse cache at near-local-disk speeds on cost-efficient instances.