LlamaIndex研究了如何提升静态嵌入模型的检索准确性，尝试了三种创新方法，虽然结果不理想但提供了宝贵经验。
静态嵌入模型相比密集嵌入模型具有极高的处理速度和低成本优势。LlamaIndex团队尝试了三种方法改进静态嵌入：在每token嵌入上使用原始MaxSim评分、训练小型适配器模型、改变蒸馏训练目标和教师模型。尽管这些方法未能达到预期效果，但研究深入探讨了静态嵌入的工作原理及其局限性。
static embeddings are extremely fast/cheap compared to dense embeddings, and still quite powerful. ...
static embeddings are extremely fast/cheap compared to dense embeddings, and still quite powerful. one issue is that they lose accuracy over longer text, since the tokens get pooled and averaged late interaction models score with MaxSim instead of average pooling, and avoid the pooling accuracy degradation. Can we apply these concepts to static models? This is a fun writeup by @LoganMarkewich - come check it out! llamaindex.ai/blog/exploring… LlamaIndex 🦙 @llama_index Static embeddings offer unmatched throughput, but they also suffer from accuracy loss compared to traditional embedding models. Can we make them better for retrieval? We tried several things: ✅️ raw maxsim scoring on the per-token embeddings ✅️ training a small adapter model ✅️ changing the distillation training target and teacher While none of these saw the results we wanted, it's an excellent dive into static embeddings and how they do (and don't) work! Blog: llamaindex.ai/blog/exploring… 🔗 View Quoted Tweet 💬 3 🔄 0 ❤️ 16 👀 1736 📊 6 ⚡