Milvus教你按文档类型选分块方法
固定长度分块将文档切成512或1024 tokens的块，但可能把完整答案切半导致检索不完整。滑动窗口分块用50-100 tokens重叠避免断句，但会产生重复块挤占检索结果。语义分块按段落、标题或章节分割保持语义完整，但只适用于格式规整的文档。Milvus建议对技术文档用语义分块+滑动窗口兜底，对话记录用大重叠固定分块，API文档按章节分块。
𝗧𝗵𝗲𝗿𝗲 𝗮𝗿𝗲 𝘁𝗵𝗿𝗲𝗲 𝗰𝗼𝗺𝗺𝗼𝗻 𝘄𝗮𝘆𝘀 𝘁𝗼 𝗰𝗵𝘂𝗻𝗸 𝗱𝗼𝗰𝘂𝗺𝗲𝗻𝘁𝘀 𝗳𝗼𝗿 𝗥𝗔𝗚....
𝗧𝗵𝗲𝗿𝗲 𝗮𝗿𝗲 𝘁𝗵𝗿𝗲𝗲 𝗰𝗼𝗺𝗺𝗼𝗻 𝘄𝗮𝘆𝘀 𝘁𝗼 𝗰𝗵𝘂𝗻𝗸 𝗱𝗼𝗰𝘂𝗺𝗲𝗻𝘁𝘀 𝗳𝗼𝗿 𝗥𝗔𝗚. 𝗪𝗵𝗶𝗰𝗵 𝗼𝗻𝗲 𝘄𝗼𝗿𝗸𝘀 𝗱𝗲𝗽𝗲𝗻𝗱𝘀 𝗼𝗻 𝘄𝗵𝗮𝘁 𝘆𝗼𝘂𝗿 𝗱𝗼𝗰𝘂𝗺𝗲𝗻𝘁𝘀 𝗹𝗼𝗼𝗸 𝗹𝗶𝗸𝗲. 𝗙𝗶𝘅𝗲𝗱-𝗹𝗲𝗻𝗴𝘁𝗵 𝗰𝗵𝘂𝗻𝗸𝗶𝗻𝗴 (𝟱𝟭𝟮 𝗼𝗿 𝟭𝟬𝟮𝟰 𝘁𝗼𝗸𝗲𝗻𝘀 𝗽𝗲𝗿 𝗰𝗵𝘂𝗻𝗸) 𝗶𝘀 𝘁𝗵𝗲 𝗲𝗮𝘀𝗶𝗲𝘀𝘁 𝘁𝗼 𝘀𝗲𝘁 𝘂𝗽, 𝗯𝘂𝘁 𝗶𝘁 𝗰𝗮𝗻 𝘀𝗽𝗹𝗶𝘁 𝗮 𝗰𝗼𝗺𝗽𝗹𝗲𝘁𝗲 𝗮𝗻𝘀𝘄𝗲𝗿 𝗶𝗻 𝗵𝗮𝗹𝗳. Say the steps for configuring Nginx timeouts take 600 tokens. A 512-token chunk cuts right through the middle, and retrieval may only return one side of it. The user gets an incomplete answer, even though the source document had the full information.. 𝗦𝗹𝗶𝗱𝗶𝗻𝗴-𝘄𝗶𝗻𝗱𝗼𝘄 𝗰𝗵𝘂𝗻𝗸𝗶𝗻𝗴 (𝟱𝟬–𝟭𝟬𝟬 𝘁𝗼𝗸𝗲𝗻𝘀 𝗼𝗳 𝗼𝘃𝗲𝗿𝗹𝗮𝗽) 𝘀𝘁𝗼𝗽𝘀 𝗮𝗻𝘀𝘄𝗲𝗿𝘀 𝗳𝗿𝗼𝗺 𝗴𝗲𝘁𝘁𝗶𝗻𝗴 𝗰𝘂𝘁 𝗺𝗶𝗱-𝘀𝗲𝗻𝘁𝗲𝗻𝗰𝗲, 𝗯𝘂𝘁 𝗶𝘁 𝗰𝗿𝗲𝗮𝘁𝗲𝘀 𝗱𝘂𝗽𝗹𝗶𝗰𝗮𝘁𝗲𝘀. The same passage shows up in three overlapping chunks, all three score high, and they fill your top results — leaving no room for other relevant content. 𝗦𝗲𝗺𝗮𝗻𝘁𝗶𝗰 𝗰𝗵𝘂𝗻𝗸𝗶𝗻𝗴 (𝘀𝗽𝗹𝗶𝘁𝘁𝗶𝗻𝗴 𝗯𝘆 𝗽𝗮𝗿𝗮𝗴𝗿𝗮𝗽𝗵, 𝗵𝗲𝗮𝗱𝗶𝗻𝗴, 𝗼𝗿 𝘀𝗲𝗰𝘁𝗶𝗼𝗻) 𝗸𝗲𝗲𝗽𝘀 𝗺𝗲𝗮𝗻𝗶𝗻𝗴 𝗶𝗻𝘁𝗮𝗰𝘁, 𝗯𝘂𝘁 𝗶𝘁 𝗼𝗻𝗹𝘆 𝘄𝗼𝗿𝗸𝘀 𝘄𝗵𝗲𝗻 𝘁𝗵𝗲 𝗱𝗼𝗰𝘂𝗺𝗲𝗻𝘁 𝗵𝗮𝘀 𝗰𝗹𝗲𝗮𝗻 𝗳𝗼𝗿𝗺𝗮𝘁𝘁𝗶𝗻𝗴. Internal wikis, chat logs, and ticket systems usually don't, so there's nothing for semantic chunking to split on. 𝗪𝗵𝗮𝘁'𝘀 𝘄𝗼𝗿𝗸𝗲𝗱 𝗳𝗼𝗿 𝘂𝘀: • 𝗧𝗲𝗰𝗵𝗻𝗶𝗰𝗮𝗹 𝗱𝗼𝗰𝘀: semantic chunking, with sliding-window as a fallback • 𝗖𝗵𝗮𝘁 𝗹𝗼𝗴𝘀 𝗮𝗻𝗱 𝘁𝗶𝗰𝗸𝗲𝘁𝘀: fixed-length, with a larger overlap • 𝗔𝗣𝗜 𝗱𝗼𝗰𝘀 𝗮𝗻𝗱 𝗰𝗼𝗻𝗳𝗶𝗴 𝗿eferences: split by section Sort your documents by type first. The right chunking method follows from there. 💬 0 🔄 0 ❤️ 0 👀 22 ⚡