技巧精选

Milvus 引入 Force Merge Compaction 优化查询性能

𝗠𝗶𝗹𝘃𝘂𝘀 𝗰𝗼𝗺𝗽𝗮𝗰𝘁𝘀 𝘀𝗲𝗴𝗺𝗲𝗻𝘁𝘀 𝗱𝘂𝗿𝗶𝗻𝗴 𝗻𝗼𝗿𝗺𝗮𝗹 𝗼𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝘀, ...

精选理由

如果你在用 Milvus 做知识库或批量加载后想加速查询,试试 Force Merge Compaction,它能在数据稳定后帮你合并段,减少查询扇出。

AI 摘要

Milvus 在写入数据后默认不会主动合并段,导致查询时需扫描过多段,影响性能。新推出的 Force Merge Compaction 功能允许用户手动触发段合并,在数据稳定后重塑存储布局。单节点基准测试显示,合并和索引重建会占用全部 16 个核心。该功能适用于写入完成后转为读密集的场景,可显著提升搜索吞吐量,目前处于公开预览阶段。

原文 · Milvus

𝗠𝗶𝗹𝘃𝘂𝘀 𝗰𝗼𝗺𝗽𝗮𝗰𝘁𝘀 𝘀𝗲𝗴𝗺𝗲𝗻𝘁𝘀 𝗱𝘂𝗿𝗶𝗻𝗴 𝗻𝗼𝗿𝗺𝗮𝗹 𝗼𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝘀, ...

𝗠𝗶𝗹𝘃𝘂𝘀 𝗰𝗼𝗺𝗽𝗮𝗰𝘁𝘀 𝘀𝗲𝗴𝗺𝗲𝗻𝘁𝘀 𝗱𝘂𝗿𝗶𝗻𝗴 𝗻𝗼𝗿𝗺𝗮𝗹 𝗼𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝘀, 𝗯𝘂𝘁 𝘄𝗶𝘁𝗵𝗶𝗻 𝗰𝗲𝗿𝘁𝗮𝗶𝗻 𝗹𝗶𝗺𝗶𝘁𝘀. 𝗢𝗻𝗰𝗲 𝘁𝗵𝗲 𝗱𝗮𝘁𝗮 𝘀𝗲𝘁𝘁𝗹𝗲𝘀, 𝘁𝗵𝗲 𝗳𝗿𝗮𝗴𝗺𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻 𝗹𝗲𝗳𝘁 𝗯𝗲𝗵𝗶𝗻𝗱 𝘁𝗮𝘅𝗲𝘀 𝗲𝘃𝗲𝗿𝘆 𝗾𝘂𝗲𝗿𝘆. 𝗔𝗱𝗱𝗿𝗲𝘀𝘀 𝗶𝘁 𝘄𝗶𝘁𝗵 𝗙𝗼𝗿𝗰𝗲 𝗠𝗲𝗿𝗴𝗲 𝗖𝗼𝗺𝗽𝗮𝗰𝘁𝗶𝗼𝗻. During ingestion, Milvus writes data into segments and seals them as part of the ingestion path. Merging them aggressively at that point would compete with ingestion resources, so the system leaves them as they are. That is the right default while data is still changing. The problem: Milvus does not have your application-level signal that ingestion is complete. After a bulk load wraps up, a knowledge-base refresh completes, or a migration lands, those write-optimized segments stay in place. Search queries may continue to fan out across more sealed segments than necessary. 𝗙𝗼𝗿𝗰𝗲 𝗠𝗲𝗿𝗴𝗲 𝗖𝗼𝗺𝗽𝗮𝗰𝘁𝗶𝗼𝗻 𝗶𝘀 𝗵𝗼𝘄 𝘆𝗼𝘂 𝗽𝗮𝘀𝘀 𝘁𝗵𝗲 𝘀𝘆𝘀𝘁𝗲𝗺 𝗮 𝘀𝗶𝗴𝗻𝗮𝗹 𝗶𝘁 𝗰𝗮𝗻𝗻𝗼𝘁 𝗶𝗻𝗳𝗲𝗿 𝗼𝗻 𝗶𝘁𝘀 𝗼𝘄𝗻: 𝘁𝗵𝗲 𝗱𝗮𝘁𝗮 𝗵𝗮𝘀 𝘀𝗲𝘁𝘁𝗹𝗲𝗱, 𝗿𝗲𝘀𝗵𝗮𝗽𝗲 𝘁𝗵𝗲 𝗹𝗮𝘆𝗼𝘂𝘁 𝗳𝗼𝗿 𝗿𝗲𝗮𝗱𝘀. The tradeoff is explicit. In our benchmark, the merge and index rebuild used all 16 cores during the merge and rebuild in our single-node benchmark. On a cluster, you can isolate that cost on separate DataNode and QueryNode resources. On a single node, time it for a quiet window. 𝗪𝗵𝗲𝗻 𝗙𝗼𝗿𝗰𝗲 𝗠𝗲𝗿𝗴𝗲 𝗳𝗶𝘁𝘀: • The collection has shifted from writes to reads • Search throughput is the bottleneck, not ingestion • You have a maintenance window or separated cluster resources • You can measure segment cou milvus.io/blog/force-mer… confirm the gain 𝗪𝗵𝗲𝗻 𝗶𝘁 𝗱𝗼𝗲𝘀 𝗻𝗼𝘁: • The collection still takes steady writes or updates • Resources are already tight with no room for the rebuild • You are in peak traffic with no quiet window 𝗙𝗼𝗿𝗰𝗲 𝗠𝗲𝗿𝗴𝗲 𝗶𝘀 𝗶𝗻 𝗽𝘂𝗯𝗹𝗶𝗰 𝗽𝗿𝗲𝘃𝗶𝗲𝘄. 𝗥𝗲𝗮𝗱 𝗮𝗯𝗼𝘂𝘁 𝗼𝘂𝗿 𝗲𝘅𝗽𝗲𝗿𝗶𝗺𝗲𝗻𝘁 𝗵𝗲𝗿𝗲: https://t.co/bDv1Y1Qlpn 💬 0 🔄 0 ❤️ 0 👀 59 ⚡