做向量搜索的团队终于不用为内存账单发愁了——HFresh 把 HNSW 的内存占用砍到零头，十亿级向量也能跑在更小的机器上，成本敏感或写入密集的场景尤其值得一试。
Weaviate 发布了名为 HFresh 的新型向量搜索索引，它通过将向量存储在磁盘上，仅在内存中保留紧凑的质心索引，大幅降低了内存需求。HFresh 将向量划分为多个小区域（postings），利用内存中的 HNSW 索引定位相关区域，再从磁盘获取数据，并采用两级旋转量化压缩。相比传统 HNSW 索引，HFresh 在十亿级向量规模下仍能保持可预测的延迟，尤其适合高维嵌入、成本敏感部署和写入密集型场景。目前 HFresh 已在 Weaviate Cloud 中提供，建议在非生产环境中测试。
Spending half your budget on memory just to keep your vector index running? There's a better way. ...
Spending half your budget on memory just to keep your vector index running? There's a better way. HNSW is the gold standard for vector search, but it needs everything in memory. As datasets grow, that gets expensive. HFresh flips the model by storing vectors on disk while keeping only a compact centroid index in memory. 𝗛𝗼𝘄 𝗶𝘁 𝘄𝗼𝗿𝗸𝘀 • Divides vectors into small regions called postings • Uses a compact in-memory HNSW index over centroids to identify relevant regions • Fetches only the relevant postings from disk for search • Applies Rotational Quantization at two levels for compression The result is significantly lower memory usage with predictable latency, even at billion-vector scale 𝗪𝗵𝗲𝗻 𝘁𝗼 𝗰𝗼𝗻𝘀𝗶𝗱𝗲𝗿 𝗛𝗙𝗿𝗲𝘀𝗵 𝗟𝗮𝗿𝗴𝗲 𝗱𝗮𝘁𝗮𝘀𝗲𝘁𝘀 𝘄𝗶𝘁𝗵 𝗵𝗶𝗴𝗵-𝗱𝗶𝗺𝗲𝗻𝘀𝗶𝗼𝗻𝗮𝗹 𝗲𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴𝘀 — Memory savings become substantial compared to HNSW 𝗖𝗼𝘀𝘁-𝘀𝗲𝗻𝘀𝗶𝘁𝗶𝘃𝗲 𝗱𝗲𝗽𝗹𝗼𝘆𝗺𝗲𝗻𝘁𝘀 — Run the same workload on smaller infrastructure 𝗪𝗿𝗶𝘁𝗲-𝗵𝗲𝗮𝘃𝘆 𝘄𝗼𝗿𝗸𝗹𝗼𝗮𝗱𝘀 — The cluster-based design avoids the write amplification that HNSW can experience during large imports HFresh is currently a weaviate.io/blog/weaviate-… we recommend testing in non-product console.weaviate.cloud/signin?utm_sou… Read our blog for more details: https://t.co/lg1jmppEfa Try HFresh now in Weaviate Cloud https://t.co/wigAn5wXmR 💬 0 🔄 21 ❤️ 63 👀 3079 📊 22 ⚡