AITOPAI热报
ProductsSelected

Milvus 3.0 External Collection

Milvus@milvusio
Post
View the post
Why we picked it

Developers managing large vector datasets can now handle incremental updates without service disruption or full data reloads.

Milvus 3.0 External Collection introduces an explicit Refresh boundary to avoid exposing partially indexed data. When new files arrive in S3, Refresh identifies changed fragments while reusing work for unchanged ones. Queries continue using the previous version until the new version is fully prepared. The system works with Milvus Tiered Storage to reduce memory footprint through lazy loading.

Original · Milvus

A new file lands in S3. Should production search expose it before its indexes are ready? 𝗠𝗶𝗹𝘃𝘂𝘀 𝟯.𝟬 𝗘𝘅𝘁𝗲𝗿𝗻𝗮𝗹 𝗖𝗼𝗹𝗹𝗲𝗰𝘁𝗶𝗼𝗻 avoids that partial state through an explicit Refresh boundary. When new files, updated metadata, or new embeddings arrive in the lake, Refresh reads the external metadata and identifies which source fragments changed. Milvus can reuse the segment and index work for unchanged fragments, while processing only the new or modified ones. During that work, queries continue using the previous published version. Once Refresh completes, the new version becomes available as a whole, so applications do not see a mixture of old and partially prepared data. External Collection also works with Milvus Tiered Storage to reduce the serving footprint. QueryNodes can initially load lightweight metadata, then fetch fields and indexes at the chunk level as queries need them. Frequently accessed data stays cached, while colder data can remain remote or be evicted. Together, incremental Refresh and Lazy Loading let Milvus keep lake-resident data searchable without rebuilding everything after each batch or loading the entire dataset into memory. See ho milvus.io/blog/milvus-3-… 0: https://t.co/VWDpFH8iAV 💬 0 🔄 0 ❤️ 0 👀 55 ⚡

#Milvus#vector database#External Collection#Refresh boundary#lazy loading

Keep reading

Recent stories with the same tag or section
How-toBuildersSelected23:50
Zilliz tests Jev model as semantic cache gateway guard

Semantic caching usually forces a tradeoff between cost savings and serving wrong answers to slightly different queries; a cheap guard model shifts that balance toward accuracy. Engineers running RAG or agent workloads can plug this pattern into existing caches like GPTCache to cut LLM spend without degrading quality. Milvus and Zilliz have active developer communities across ASEAN, so the benchmark and blog are directly accessible to teams in the region.

#Milvus#Zilliz#Jev#semantic caching
Milvus@milvusioSource
How-toBuildersSelected01:06
Milvus Function Chain Reranks Search with XGBoost in One Call

Reranking inside the vector database itself removes the need for a separate reranking service and an extra network hop, which simplifies production search stacks. Builders running e-commerce, feed, or RAG systems can reuse the open-source demo to blend relevance with revenue-driving signals. It compares with the common pattern of exporting candidates from Milvus and reranking them in an external model server.

#Milvus#XGBoost#RAG#vector database
Milvus@milvusioSource
How-toBuildersSelected00:11
Milvus benchmark tests Jev against LLMs in agentic search

The benchmark suggests a practical routing rule: swap general-purpose LLM calls for Jev only on narrow, high-frequency decisions like stopping checks, and keep dedicated rerankers like Voyage rerank-3 for re-ranking. Teams running RAG pipelines can use the latency and cost figures to decide where Jev cuts spend without hurting recall. Full results are on Milvus's blog.

#Jev#Milvus#RAG#Voyage
Milvus@milvusioSource