How-toSelected

Milvus 用 MinHash 和 LSH 检测训练数据里的重复与改写文档

Why we picked it

做数据清洗的朋友可以看看:Milvus 官方演示了怎么用 MinHash 加 LSH 抓出改写过的重复文本,Jaccard 打分从 0.92 到 1.00 分得很清楚,文档可以直接跟着跑自己的数据集。

同一条台灯广告可以被改写成三条不同的“产品评论”,换标题加改写就能绕过精确匹配去重。Milvus 的演示用 MinHash 把文档的词序列转成紧凑指纹,再通过 LSH 索引找出疑似重复项,无需对每一对文档做两两比较。之后 Milvus 按估计的 Jaccard 相似度对候选结果排序:与原评论相比,精确复制的得分为 1.00,轻度编辑后的得分为 0.92,独立撰写的文章得分明显更低。官方文档提供了 MinHash 与 LSH 的使用指南,可以按步骤在自己的数据集上跑同样的流程。

Image · Milvus
Body

对构建 LLM 训练数据的团队来说,这种近重复检测比精确去重更贴近实际:爬取的语料里大量内容是同一文案的微调版本,不去除会放大数据偏差并浪费训练算力。MinHash + LSH 是成熟的传统方案,Milvus 把它集成进向量数据库的索引体系里,让去重可以和相似度检索放在同一套基础设施中完成。

Original · Milvus

The same desk lamp ad can show up in your LLM training data as three different product reviews. A new headline and a few rewritten lines are enough to get past an exact-match check. In our demo, 𝗠𝗶𝗹𝘃𝘂𝘀 𝘂𝘀𝗲𝘀 𝗠𝗶𝗻𝗛𝗮𝘀𝗵 𝘁𝗼 𝘁𝘂𝗿𝗻 𝗼𝘃𝗲𝗿𝗹𝗮𝗽𝗽𝗶𝗻𝗴 𝘀𝗲𝗾𝘂𝗲𝗻𝗰𝗲𝘀 𝗼𝗳 𝘄𝗼𝗿𝗱𝘀 𝗶𝗻𝘁𝗼 𝗮 𝗰𝗼𝗺𝗽𝗮𝗰𝘁 𝗳𝗶𝗻𝗴𝗲𝗿𝗽𝗿𝗶𝗻𝘁 𝗳𝗼𝗿 𝗲𝗮𝗰𝗵 𝗱𝗼𝗰𝘂𝗺𝗲𝗻𝘁. An LSH index finds likely matches without comparing every pair of documents. Milvus can then rank those candidates by estimated Jaccard similarity. Compared with the original review, an exact copy scored 𝟭.𝟬𝟬, a lightly edited copy scored 𝟬.𝟵𝟮, and an independently written article scored � milvus.io/docs/minhash-l… c #VectorDatabase n #Milvus wn dataset with the Milvus MinHash and LSH guide. https://t.co/qTpWDDfm7V #VectorDatabase #Milvus

Industry Coordinates & Source Verification

This report is categorized under AITOP's "How-to" desk, focusing on China's AI industry and domestic foundation model ecosystem. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Milvus, MinHash, LSH, RAG.

Source attribution: curated from Milvus (@milvusio).