面向乳腺病灶管理的 RAG 系统证据感知检索评估方法
How Ground Truth Shapes Retrieval Scores: Evidence-Aware Evaluation of a Retrieval-Augmented Generation System for Breast Lesions
一项临床 RAG 评估研究：混合检索加 cross-encoder 在乳腺病灶文献上拿到 0.94 的 Recall@3，还发现标注范围会直接影响分数，做 RAG 评测的可以看看方法细节。
论文提出一套证据感知检索评估协议，对象是覆盖乳腺病灶管理文献的临床 RAG 系统。评估基于 18 篇文档和 18 条查询的开发基准，检索采用 dense embeddings、BM25 与 cross-encoder 重排序的混合方案。最终选定 384-token 分块、20% 重叠的配置，在 18 条查询中的 17 条于前三个结果内命中可用证据，Recall@3、R-Precision、MRR@3 和 nDCG@3 均达到 0.94。实验还发现，扩充标注使证据片段从 25 增至 37、接受分块从 36 增至 51，但检索排名不变，说明检索分数对 ground-truth 标注覆盖范围敏感。
How Ground Truth Shapes Retrieval Scores: Evidence-Aware Evaluation of a Retrieval-Augmented Generation System for Breast Lesions
Purpose: To describe an evidence-aware retrieval evaluation protocol for a clinical retrieval-augmented generation (RAG) system over breast lesion management literature, addressing the dependence of retrieval scores on annotation coverage. Methods: An 18-document corpus and an 18-query development benchmark, with one Yes/No or WH-question per document, were used. Retrieval combined dense embeddings, BM25, and cross-encoder reranking. Chunk sizes of 128 to 1024 tokens, overlaps of 0 to 20%, and four retriever configurations were evaluated. Relevance was defined by evidence spans mapped to all containing chunks, treating overlapping chunks with equivalent evidence as relevant. Retrieval was evaluated using R-Precision, Recall@K, MRR, and nDCG@K. Separately, generation was assessed on a 512-token index at K=1 to 5 using DeepEval claim-level metrics, with deepseek-v4-pro as the LLM judge, and BLEU/ROUGE metrics. Results: The selected hybrid retriever, using 384-token chunks and 20% overlap, retrieved accepted evidence in the top three results for 17 of 18 queries. Recall@3, R-Precision, MRR@3, and nDCG@3 each reached 0.94. Ground-truth curation increased evidence spans from 25 to 37 and accepted chunk targets from 36 to 51 without changing retrieval rankings. Conclusion: Cross-encoder reranking achieved the highest retrieval scores. Expanded relevance annotations increased recognized evidence without changing rankings, highlighting annotation sensitivity. The small development set and retrieval-informed curation require independent validation, including radiologist assessment of evidence sufficiency.
