FinRank是个新出的金融问答基准，专门考你从SEC文件里找证据的能力。实测连7B模型都只有44.8%的召回率，难度很够。
FinRank基准包含1185个手工问答对，覆盖22家公司的10-K和10-Q文件，每项记录附有参考答案、gold支持段落和人工筛选的困难负样本。在合并证据语料上，7B指令微调嵌入模型的Recall@10仅为44.8%。亚十亿参数编码器相比BM25最高提升3.5个点，而金融适配嵌入模型比BM25低9.7个点。将随机负样本替换为困难负样本后，成对准确率下降13.0至20.5个百分点。该基准分别评测段落检索、重排序和困难负样本判别任务。
FinRank: An Evidence-Grounded Benchmark for Financial Question Answering and Retrieval over SEC Filings
Financial question answering is typically evaluated by answer correctness, yet in SEC filings a plausible and even numerically correct answer can be grounded in the wrong evidence. Similar facts and disclosures recur across sections of a filing, across reporting periods of the same firm, and across comparable firms. FinRank targets this provenance-sensitive retrieval problem by requiring systems to identify evidence for the intended entity, reporting period, and disclosure context. The benchmark contains 1185 manually authored question-answer records over the 10-K and 10-Q filings of 22 companies. Each record includes a reference answer, gold supporting passages, and hand-curated hard negatives drawn from confusable passages within filings, across reporting periods, and across comparable firms. FinRank evaluates passage retrieval, reranking, and hard-negative discrimination as separately measured tasks. Baseline results demonstrate the difficulty of this setting: among the evaluated systems, even a 7B instruction-tuned embedder reaches only 44.8% Recall@10 on the pooled evidence corpus; sub-billion-parameter encoders gain at most 3.5 points over BM25, a finance-adapted embedder trails BM25 by 9.7 points, and pairwise accuracy falls by 13.0-20.5 percentage points when random negatives are replaced with the curated hard negatives. FinRank provides an evidence-first benchmark for developing financial question answering systems that are not only accurate but also grounded in the correct disclosure.