做RAG系统的团队，如果发现线上召回率不如测试时，这三个原因能帮你快速定位问题，建议对照排查。
RAG系统上线后召回率下降，常见原因包括：索引过期（新文档加入、旧文档修改或删除，但向量索引未更新）、嵌入模型变更（如OpenAI更新模型导致新旧向量不匹配）、用户提问方式变化（用户群体和产品变化导致查询分布偏移）。此外，测试集可能已偏离真实场景，掩盖了召回率下降的问题。这些因素会导致检索结果不准确，影响RAG系统性能。
Your RAG tested well and went live, but recall is getting worse. 𝗧𝗵𝗿𝗲𝗲 𝗰𝗼𝗺𝗺𝗼𝗻 ...
Your RAG tested well and went live, but recall is getting worse. 𝗧𝗵𝗿𝗲𝗲 𝗰𝗼𝗺𝗺𝗼𝗻 𝗰𝗮𝘂𝘀𝗲𝘀: • 𝗧𝗵𝗲 𝗶𝗻𝗱𝗲𝘅 𝗶𝘀 𝘀𝘁𝗮𝗹𝗲. New docs get added, old ones get edited or deleted, but the vector index is still the one you built three months ago. Retrieval is searching an outdated snapshot. • 𝗧𝗵𝗲 𝗲𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴 𝗺𝗼𝗱𝗲𝗹 𝗰𝗵𝗮𝗻𝗴𝗲𝗱. Providers like OpenAI update their embedding models without much warning. Your existing documents were embedded with the old version, new queries use the new one, and the vectors no longer line up the way they used to. • 𝗨𝘀𝗲𝗿𝘀 𝘀𝘁𝗮𝗿𝘁𝗲𝗱 𝗮𝘀𝗸𝗶𝗻𝗴 𝗱𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝘁𝗹𝘆. Your user base grew, the product changed, and the way people phrase questions shifted with it. The queries your system handles today aren't the same ones it was tested on. 𝗧𝗵𝗲𝗿𝗲'𝘀 𝗮𝗹𝘀𝗼 𝗮 𝗳𝗼𝘂𝗿𝘁𝗵 𝗿𝗲𝗮𝘀𝗼𝗻 𝗿𝗲𝗰𝗮𝗹𝗹 𝗹𝗼𝗼𝗸𝘀 𝘄𝗼𝗿𝘀𝗲. Your test set drifted from reality, or it never covered certain query types to begin with. That doesn't cause recall to drop — it just means you can't see where it already has. 💬 0 🔄 0 ❤️ 0 👀 105 ⚡