Weaviate introduces HFresh vector index
HFresh offers memory efficiency for large datasets with frequent changes, ideal for production systems constrained by infrastructure costs or memory limitations.
Weaviate has developed HFresh, a disk-based vector index alternative to HNSW. HFresh uses a compact HNSW index to route queries to relevant centroids, then searches compressed postings on disk. RQ8 reduces centroid vectors by approximately four times, while RQ1 cuts posting-vector data by up to 32 times compared to 32-bit floats. The index maintains balance through background split, merge, and reassign operations without periodic full rebuilds.
Is the fastest vector index always the right vector index? HNSW is difficult to beat when an application needs the lowest latency and highest query throughput. But production systems optimise for different constraints: 1. Some need to support very large datasets without keeping the full index in memory. 2. Some are limited by available memory or infrastructure cost. 3. Some change frequently and cannot depend on disruptive full-index rebuilds. HFresh takes a different approach. It uses a compact HNSW index to route each query towards the most relevant centroids, then searches compressed postings stored on disk. Underneath that architecture: - RQ8 makes centroid vectors approximately four times smaller before overhead. - RQ1 reduces stored posting-vector data by up to 32 times compared with 32-bit floats. - Background split, merge, and reassign operations keep the index balanced incrementally. The result is a disk-based index designed for lower heap usage, controlled query I/O, and large mutable datasets—without periodic full rebuilds. The choice is not HFresh or HNSW in every situation. Choose HNSW when peak query performance is the priority. Choose HFresh when memory efficiency matters more than peak throughput. Read how HFresh works: weaviate.io/blog/hfresh?ut… 💬 0 🔄 1 ❤️ 3 👀 158 📊 1 ⚡
