LEMUR 多向量压缩策略在长尾语料上存在长度偏差问题
做向量检索的团队注意了——LEMUR 在长尾语料上会悄悄按文档长度排序而非相关性，LoTTE 上召回率从 98% 掉到 30.5%。如果你的语料长度差异大（P90/P10 > 20），建议先跑文中的两个检查再决定是否用 LEMUR。
LEMUR 是一种将多向量压缩为单向量的策略，声称无需调参、数据驱动。但研究发现，在文档长度差异大的语料（如 LoTTE，长度范围 400 倍）上，LEMUR 会学习到“长文档得分更高”的偏差，导致按长度而非相关性排序。
在 LoTTE 上，LEMUR 的 nDCG@10 仅 0.109，召回率 30.
5%，而 Exact MaxSim 方法达 0.722 和 98%。
问题根源在于 LEMUR 的训练标签（MaxSim 分数）天然偏向长文档，MLP 学到了长度信号。建议用户检查语料的长度分布（P90/P10 比）和模型的可分离性（MaxSim 标准差），若长度差异大或可分离性高，应改用 TokenANN 或 MUVERA。
已使用 LEMUR 的团队可通过长度分层采样缓解偏差。
𝗟𝗘𝗠𝗨𝗥 𝗶𝘀 𝘁𝗵𝗲 𝗺𝘂𝗹𝘁𝗶-𝘃𝗲𝗰𝘁𝗼𝗿 𝘀𝘁𝗿𝗮𝘁𝗲𝗴𝘆 𝘁𝗵𝗮𝘁 𝗹𝗼𝗼𝗸𝘀 𝗲𝗮𝘀𝗶𝗲𝘀𝘁 𝘁𝗼 𝗮𝗱𝗼𝗽𝘁 — 𝗻𝗼 𝘁𝘂𝗻𝗶𝗻𝗴, 𝗱𝗮𝘁𝗮-𝗱𝗿𝗶𝘃𝗲𝗻, 𝗰𝗼𝗺𝗽𝗿𝗲𝘀𝘀𝗲𝘀 𝘆𝗼𝘂𝗿 𝗺𝘂𝗹𝘁𝗶-𝘃𝗲𝗰𝘁𝗼𝗿𝘀 𝗶𝗻𝘁𝗼 𝘀𝗶𝗻𝗴𝗹𝗲 𝘃𝗲𝗰𝘁𝗼𝗿𝘀 𝗼𝗻 𝗶𝘁𝘀 𝗼𝘄𝗻. 𝗕𝘂𝘁 𝗼𝗻 𝗹𝗼𝗻𝗴-𝘁𝗮𝗶𝗹𝗲𝗱 𝗰𝗼𝗿𝗽𝗼𝗿𝗮, 𝗶𝘁 𝗰𝗮𝗻 𝗾𝘂𝗶𝗲𝘁𝗹𝘆 𝘀𝘁𝗮𝗿𝘁 𝗿𝗮𝗻𝗸𝗶𝗻𝗴 𝗯𝘆 𝗱𝗼𝗰𝘂𝗺𝗲𝗻𝘁 𝗹𝗲𝗻𝗴𝘁𝗵 𝗶𝗻𝘀𝘁𝗲𝗮𝗱 𝗼𝗳 𝗿𝗲𝗹𝗲𝘃𝗮𝗻𝗰𝗲. 𝗟𝗼𝗧𝗧𝗘, 𝗝𝗶𝗻𝗮-𝗖𝗼𝗹𝗕𝗘𝗥𝗧-𝘃𝟮: 📈 𝗘𝘅𝗮𝗰𝘁 𝗠𝗮𝘅𝗦𝗶𝗺 — nDCG @10 = 0.722, R @100 = 98% 📉 𝗟𝗘𝗠𝗨𝗥 — nDCG @10 = 0.109, R @100 = 30.5% 𝗪𝗵𝘆? 𝗜𝘁'𝘀 𝗻𝗼𝘁 𝗮𝗻 𝗶𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻 𝗯𝘂𝗴 — 𝗶𝘁'𝘀 𝗮 𝗹𝗲𝗻𝗴𝘁𝗵 𝗯𝗶𝗮𝘀 𝗯𝗮𝗸𝗲𝗱 𝗶𝗻𝘁𝗼 𝗵𝗼𝘄 𝗟𝗘𝗠𝗨𝗥 𝘁𝗿𝗮𝗶𝗻𝘀. LEMUR's labels are MaxSim scores — for a sampled vector, its highest inner product against any token in a document. A longer document has more tokens, so it's more likely to hold a high value regardless of relevance, a pure statistical effect. The MLP learns "longer scores higher" instead of "more relevant scores higher," and the ranking follows length. 𝗜𝘁 𝗼𝗻𝗹𝘆 𝗯𝗿𝗲𝗮𝗸𝘀 𝘄𝗵𝗲𝗻 𝘁𝘄𝗼 𝘁𝗵𝗶𝗻𝗴𝘀 𝗹𝗶𝗻𝗲 𝘂𝗽: • First, a length-sensitive model, which comes with high separability. On Jina, docs under 50 tokens average a MaxSim of 0.492; docs over 1,000 average 0.666 — a 0.174 gap the model assigns to length alone. • Second, heavy length variance. LoTTE runs 10 to 4,109 tokens, about a 400x range. 𝗟𝗼𝗧𝗧𝗘 𝗵𝗮𝘀 𝗯𝗼𝘁𝗵, 𝗮𝗻𝗱 𝗶𝘁 𝗳𝗮𝗶𝗹𝘀 𝗮𝘁 𝗰𝗼𝗺𝗽𝗿𝗲𝘀𝘀𝗶𝗼𝗻, 𝗻𝗼𝘁 𝗿𝗲𝗿𝗮𝗻𝗸𝗶𝗻𝗴 — 𝗠𝗮𝘁𝗵 𝗥@𝟭𝟬𝟬 𝘁𝗵𝗲𝗿𝗲 𝗶𝘀 𝟮𝟰.𝟳%, 𝘀𝗼 𝘁𝗵𝗲 𝗿𝗶𝗴𝗵𝘁 𝗱𝗼𝗰𝘂𝗺𝗲𝗻𝘁𝘀 𝗮𝗿𝗲 𝗱𝗿𝗼𝗽𝗽𝗲𝗱 𝗳𝗿𝗼𝗺 𝘁𝗵𝗲 𝗰𝗮𝗻𝗱𝗶𝗱𝗮𝘁𝗲 𝘀𝗲𝘁 𝗯𝗲𝗳𝗼𝗿𝗲 𝗿𝗲𝗿𝗮𝗻𝗸𝗶𝗻𝗴 𝗰𝗮𝗻 𝗵𝗲𝗹𝗽. 𝗧𝗵𝗲 𝗰𝗼𝗻𝘁𝗿𝗼𝗹 𝗿𝘂𝗻𝘀 𝗶𝘀𝗼𝗹𝗮𝘁𝗲 𝘁𝗵𝗲 𝗰𝗮𝘂𝘀𝗲: For length-uniform data (SciFact): same LEMUR behaves normally — 0.651 vs BruteForce 0.719. For a low-separability model (AnswerAI on LoTTE): unaffected — R @100 91.5%. 𝗧𝘄𝗼 𝗰𝗵𝗲𝗰𝗸𝘀 𝗳𝗼𝗿 𝘆𝗼𝘂𝗿 𝗼𝘄𝗻 𝗰𝗼𝗿𝗽𝘂𝘀: → 𝗟𝗲𝗻𝗴𝘁𝗵: if P90/P10 is under 5 it's fairly uniform and LEMUR is usually safe; over 20 (LoTTE is ~400), be cautious. → 𝗦𝗲𝗽𝗮𝗿𝗮𝗯𝗶𝗹𝗶𝘁𝘆: sample ~1,000 token vectors, take the std of their MaxSim scores. Under 0.08 is low, LEMUR is safe; over 0.12 is high — use TokenANN or MUVERA. 𝗜𝗳 𝘆𝗼𝘂'𝗿𝗲 𝗮𝗹𝗿𝗲𝗮𝗱𝘆 𝗰𝗼𝗺𝗺𝗶𝘁𝘁𝗲𝗱 𝘁𝗼 𝗟𝗘𝗠𝗨𝗥, 𝗹𝗲𝗻𝗴𝘁𝗵-𝗯𝗮𝗹𝗮𝗻𝗰𝗲𝗱 𝘀𝗮𝗺𝗽𝗹𝗶𝗻𝗴 𝗵𝗲𝗹𝗽𝘀: 𝘀𝘁𝗿𝗮𝘁𝗶𝗳𝘆 𝗯𝘆 𝗹𝗲𝗻𝗴𝘁𝗵 (𝟬–𝟭𝟬𝟬, 𝟭𝟬𝟬–𝟯𝟬𝟬, 𝟯𝟬𝟬–𝟭,𝟬𝟬𝟬, >𝟭,𝟬𝟬𝟬 𝘁𝗼𝗸𝗲𝗻𝘀) 𝗮𝗻𝗱 𝗱𝗿𝗮𝘄 𝗲𝗾𝘂𝗮𝗹 𝘀𝗮𝗺𝗽𝗹𝗲𝘀 𝗽𝗲𝗿 𝗯𝗮𝗻𝗱. 𝗜𝘁 𝘄𝗲𝗮𝗸𝗲𝗻𝘀 𝘁𝗵𝗲 𝗹𝗲𝗻𝗴𝘁𝗵 𝘀𝗶𝗴𝗻𝗮𝗹 𝘄𝗶𝘁𝗵𝗼𝘂𝘁 𝗳𝘂𝗹𝗹𝘆 𝗿𝗲𝗺𝗼𝘃𝗶𝗻𝗴 𝗶𝘁.
This report is categorized under AITOP's "Papers" desk. Key entities and technical topics under continuous tracking include 向量检索, 多向量压缩, 长度偏差, LEMUR.
Source attribution: curated from Milvus (@milvusio).
