Milvus 3.0 内置 MinHash 近重复检测,无需独立去重流水线
向量数据库 Milvus 3.0 直接内置了 MinHash 去重能力,文本进去就能查重,不用自己搭签名流水线也不用嵌入模型。做爬虫清洗、RAG 语料或 LLM 训练数据的人可以直接在入库前一步完成去重,比自建方案省事不少。
Milvus 3.0 将 MinHash 近重复检测直接集成进向量数据库。此前在网络爬取和内容流水线中,同一文档常因标题更新、样板文字改动或轻微编辑而出现多个变体:精确哈希无法识别这些变体,而全量两两相似度比对随语料规模增长代价极高。
技术机制分两层:MinHash Function 把原始文本转成紧凑的二进制签名,近似估算文本 shingle 之间的 Jaccard 相似度;MINHASH_LSH 索引用局部敏感哈希高效召回候选匹配,避免扫描全部文档对。摄取和搜索接口都直接接受原始文本,签名生成由 Milvus 自动完成,不需要嵌入模型,也不需要单独部署 MinHash 生成流水线。
典型用法是在文档摄取流水线中,插入前先对现有语料做去重检查,再用可选的 Jaccard 精化步骤改善候选排序。适用场景包括网页语料清洗、RAG 知识库建设和 LLM 训练数据准备。对开发者和企业数据团队来说,对比自行搭建 MinHash+LSH 的方案,这一内置能力省去了签名生成和索引维护两个环节。
Near-duplicate detection is a common challenge in large-scale document ingestion. In web crawls and content pipelines, the same document often appears in slightly different versions: updated titles, modified boilerplate, or minor content edits. Exact hashing misses these variants, while exhaustive pairwise similarity comparisons become expensive as the corpus grows. 𝗠𝗶𝗹𝘃𝘂𝘀 𝟯.𝟬 𝗯𝗿𝗶𝗻𝗴𝘀 𝗠𝗶𝗻𝗛𝗮𝘀𝗵-𝗯𝗮𝘀𝗲𝗱 𝗻𝗲𝗮𝗿-𝗱𝘂𝗽𝗹𝗶𝗰𝗮𝘁𝗲 𝗱𝗲𝘁𝗲𝗰𝘁𝗶𝗼𝗻 𝗱𝗶𝗿𝗲𝗰𝘁𝗹𝘆 𝗶𝗻𝘁𝗼 𝘁𝗵𝗲 𝗱𝗮𝘁𝗮𝗯𝗮𝘀𝗲. • 𝗠𝗶𝗻𝗛𝗮𝘀𝗵 𝗙𝘂𝗻𝗰𝘁𝗶𝗼𝗻 converts raw text into compact binary signatures that approximate Jaccard similarity over text shingles. • 𝗠𝗜𝗡𝗛𝗔𝗦𝗛_𝗟𝗦𝗛 𝗶𝗻𝗱𝗲𝘅 uses locality-sensitive hashing to efficiently retrieve likely matches without scanning every document pair. Both ingestion and search accept raw text. Milvus handles signature generation automatically, so there's no need for an embedding model or a separate MinHash generation pipeline. You can use this in document ingestion pipelines to check incoming content against an existing corpus before inserting it, with optional Jaccard refinement to improve candidate ranking. 𝗧𝗵𝗶𝘀 𝗶𝘀 𝗰𝗼𝗻𝘁𝗲𝗻𝘁-𝗼𝘃𝗲𝗿𝗹𝗮𝗽 𝗱𝗲𝘁𝗲𝗰𝘁𝗶𝗼𝗻, 𝗻𝗼𝘁 𝘀𝗲 milvus.io/docs/minhash-f… � #Milvus al #VectorDatabase e #OpenSource , web corpus cleaning, and LLM training data preparation. Explore the implementation: https://t.co/o4G3CnDYFB #Milvus #VectorDatabase #OpenSource
This report is categorized under AITOP's "Products" desk, focusing on China's AI industry and domestic foundation model ecosystem. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Milvus, MinHash, VectorDatabase, RAG.
Source attribution: curated from Milvus (@milvusio).