FinRank：面向SEC文件的证据定位金融问答基准
FinRank: An Evidence-Grounded Benchmark for Financial Question Answering and Retrieval over SEC Filings
FinRank是个新出的金融问答基准，专门考你从SEC文件里找证据的能力。实测连7B模型都只有44.8%的召回率，难度很够。
FinRank基准包含1185个手工问答对，覆盖22家公司的10-K和10-Q文件，每项记录附有参考答案、gold支持段落和人工筛选的困难负样本。
在合并证据语料上，7B指令微调嵌入模型的Recall@10仅为44.8%。
亚十亿参数编码器相比BM25最高提升3.5个点，而金融适配嵌入模型比BM25低9.
7个点。将随机负样本替换为困难负样本后，成对准确率下降13.
0至20.5个百分点。
该基准分别评测段落检索、重排序和困难负样本判别任务。
FinRank: An Evidence-Grounded Benchmark for Financial Question Answering and Retrieval over SEC Filings
Financial question answering is typically evaluated by answer correctness, yet in SEC filings a plausible and even numerically correct answer can be grounded in the wrong evidence. Similar facts and disclosures recur across sections of a filing, across reporting periods of the same firm, and across comparable firms. FinRank targets this provenance-sensitive retrieval problem by requiring systems to identify evidence for the intended entity, reporting period, and disclosure context. The benchmark contains 1185 manually authored question-answer records over the 10-K and 10-Q filings of 22 companies. Each record includes a reference answer, gold supporting passages, and hand-curated hard negatives drawn from confusable passages within filings, across reporting periods, and across comparable firms. FinRank evaluates passage retrieval, reranking, and hard-negative discrimination as separately measured tasks. Baseline results demonstrate the difficulty of this setting: among the evaluated systems, even a 7B instruction-tuned embedder reaches only 44.8% Recall@10 on the pooled evidence corpus; sub-billion-parameter encoders gain at most 3.5 points over BM25, a finance-adapted embedder trails BM25 by 9.7 points, and pairwise accuracy falls by 13.0-20.5 percentage points when random negatives are replaced with the curated hard negatives. FinRank provides an evidence-first benchmark for developing financial question answering systems that are not only accurate but also grounded in the correct disclosure.
This report is categorized under AITOP's "Papers" desk. Key entities and technical topics under continuous tracking include FinRank, SEC filings, 金融问答, 基准.
Source attribution: curated from arXiv cs.AI (@Sasan Mansouri, Daniel Saad, Mark Wahrenburg, Manu Weissel, Fabian Woebbeking).