PapersMulti-source

FinVeriQA 开源多智能体金融问答系统:小模型路由、Python 计算、MCP 工具链

FinVeriQA: Financial Research Agents for Question Answering: Routing, Retrieval, MCP Tool Use, and Provenance-Aware Evidence Verification

Why we picked it

这篇把金融问答 Agent 拆成路由、计算、检索、写作、校验五步,用数字说话:小模型路由反而赢了大模型,Mistral-7B 算数只有 29% 正确率。做 RAG 或智能体的人该看看它的校验设计——证据自检靠不住。

开源多智能体原型 FinVeriQA 针对 SEC 文件和实时行情构建金融问答流水线:紧凑决策模型 Laya 负责问题路由并拦截投资建议请求,所有数值由 Python 计算,Mistral-7B 生成带引用的回答,代码校验器核对引用 ID 与计算结果,整条链路通过 MCP 服务器对外暴露。

Body

实测数据显示,Laya 路由准确率 0.85,高于 Mistral-7B 的 0.52(总体 0.86 对 0.68,McNemar p = 0.0003);Mistral-7B 在 100 道金融计算题上只算对 29%,Python 则全对;接入实时报价把价格误差从 210% 降到 0.00%,混合检索在 50 道文件问题上取得 76% Hit@5。

但案例研究暴露了关键问题:两个智能体回答都通过了校验,却把 AMD 的样本数据标成 NVIDIA 和 Alphabet。在 FinRank 基准上的对照实验解释了原因——证据被污染时自一致性仍保持高置信(一致性 0.55-0.59,准确率从 0.50 跌到 0.00),共享证据校验只能抓到 35% 错误,对照原始段落核查可抓到 77%(p = 5.96×10⁻⁸);DeepSeek-R1-8B 为 68% 对 90%。结论是:校验必须独立于写作模型看到的那份证据。

Original · arXiv: DeepSeek

FinVeriQA: Financial Research Agents for Question Answering: Routing, Retrieval, MCP Tool Use, and Provenance-Aware Evidence Verification

Large language models (LLMs) write fluent financial analyses but miscompute ratios, recall stale prices, and route questions to the wrong tool with high confidence. We ask which component should do each job in a financial question-answering assistant, and study it with FinVeriQA, an open-source multi-agent research prototype over SEC filings and live market data: a compact typed-decision model (Laya) routes each question and screens out advice requests, Python computes every number, retrieval and tools fetch evidence, Mistral-7B writes a cited answer, a code validator checks that cited IDs exist and that numbers match computed notes, and a Model Context Protocol (MCP) server exposes the pipeline. In live experiments, Laya routed better than Mistral-7B (0.85 vs. 0.52; overall 0.86 vs. 0.68, McNemar p = 0.0003); a 12B Mistral decided no better, and DeepSeek-R1-8B decided better (0.79 vs. 0.68, p = 0.0192) but about 9.9 times more slowly. Mistral-7B computed 29% of 100 financial calculations correctly against 100% for Python, a live quote cut price error from 210% to 0.00%, and hybrid retrieval reached 76% Hit@5 on 50 filing questions. Yet in a live case study, both agent answers passed validation while describing AMD sample data under NVIDIA's and Alphabet's names. A controlled upper-bound study on the FinRank benchmark explains why: when one figure in the shared evidence is corrupted, self-consistency stays confident (agreement 0.55 to 0.59 while accuracy falls from 0.50 to 0.00), and a shared-evidence check (the kind our validator performs) catches 35% of the errors, whereas a check against the original passage catches 77% ($p = 5.96 \times 10^{-8}$); for DeepSeek-R1-8B, 68% vs. 90%. Verification must be independent of the evidence the writer saw.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include FinVeriQA, MCP and tools, Mistral-7B, DeepSeek-R1-8B.

Primary source verification: published directly by arXiv: DeepSeek with first-party authority. Cross-source corroboration: 11 independent sources have covered this development (including NVIDIA AI, IT之家, DAIR.AI), confirming industry-wide signal strength.