AITOPAI热报
ProductsMulti-sourceSelected

Milvus 3.0 indexes external data

Milvus@milvusio
Post· 3 sources
View the post
Why we picked it

This eliminates data duplication costs and simplifies workflows for organizations using vector databases with large datasets stored in data lakes.

Milvus 3.0 now supports indexing and searching data without moving source rows into Milvus. The new External Collection feature connects directly to data stored in Parquet, Apache Iceberg, Lance, Vortex, or Milvus snapshots. Teams can use familiar Milvus APIs for ANN vector search, scalar filtering, BM25 search, and hybrid retrieval without copying source data or maintaining sync pipelines.

Original · Milvus

𝗠𝗶𝗹𝘃𝘂𝘀 𝟯.𝟬 𝗰𝗮𝗻 𝗻𝗼𝘄 𝗶𝗻𝗱𝗲𝘅 𝗮𝗻𝗱 𝘀𝗲𝗮𝗿𝗰𝗵 𝗱𝗮𝘁𝗮 𝘄𝗶𝘁𝗵𝗼𝘂𝘁 𝗳𝗶𝗿𝘀𝘁 𝗺𝗼𝘃𝗶𝗻𝗴 𝘁𝗵𝗲 𝘀𝗼𝘂𝗿𝗰𝗲 𝗿𝗼𝘄𝘀 𝗶𝗻𝘁𝗼 𝗠𝗶𝗹𝘃𝘂𝘀. External Collection connects Milvus directly to data stored in Parquet, Apache Iceberg, Lance, Vortex, or a supported Milvus snapshot. You define the external source, map its columns to a Milvus schema, and create the indexes your application needs. Milvus then builds and serves the retrieval layer while the original records remain in the lake. From the application side, the experience stays familiar. Once the collection is refreshed and loaded, teams can use the same Milvus search and query APIs for: • ANN vector search • scalar and JSON filtering • BM25 and full-text search • hybrid retrieval and ranking For large, batch-produced datasets, this provides a production retrieval path without copying the source rows into Milvus or maintaining another synchronization pipeline. Keeping the lake as the shared data foundation is a key milvus.io/blog/milvus-3-… s 3.0’s Vector Lakebase architecture. Read more: https://t.co/VWDpFH7KLn 💬 0 🔄 0 ❤️ 0 👀 19 ⚡

#Milvus#Milvus 3.0#vector databases#External Collection#data lakes

Keep reading

Recent stories with the same tag or section
How-toBuildersSelected23:50
Zilliz tests Jev model as semantic cache gateway guard

Semantic caching usually forces a tradeoff between cost savings and serving wrong answers to slightly different queries; a cheap guard model shifts that balance toward accuracy. Engineers running RAG or agent workloads can plug this pattern into existing caches like GPTCache to cut LLM spend without degrading quality. Milvus and Zilliz have active developer communities across ASEAN, so the benchmark and blog are directly accessible to teams in the region.

#Milvus#Zilliz#Jev#semantic caching
Milvus@milvusioSource
How-toBuildersSelected01:06
Milvus Function Chain Reranks Search with XGBoost in One Call

Reranking inside the vector database itself removes the need for a separate reranking service and an extra network hop, which simplifies production search stacks. Builders running e-commerce, feed, or RAG systems can reuse the open-source demo to blend relevance with revenue-driving signals. It compares with the common pattern of exporting candidates from Milvus and reranking them in an external model server.

#Milvus#XGBoost#RAG#vector database
Milvus@milvusioSource
How-toBuildersSelected00:11
Milvus benchmark tests Jev against LLMs in agentic search

The benchmark suggests a practical routing rule: swap general-purpose LLM calls for Jev only on narrow, high-frequency decisions like stopping checks, and keep dedicated rerankers like Voyage rerank-3 for re-ranking. Teams running RAG pipelines can use the latency and cost figures to decide where Jev cuts spend without hurting recall. Full results are on Milvus's blog.

#Jev#Milvus#RAG#Voyage
Milvus@milvusioSource