V-FiLLM:金融推理基准,用计算树生成可验证题目

V-FiLLM: Verified Financial LLM Reasoning Benchmark

精选理由

想测金融表格推理?V-FiLLM 用计算树自动生成题目,不用人工标注,还能控制难度,微调后效果提升明显。

AI 摘要

V-FiLLM 是一个新的金融推理基准框架,通过可执行计算树在真实表格上生成题目,答案由构造保证正确,无需人工标注。基准提供四个难度控制维度:计算深度、表达式广度、金融概念复杂度和上下文规模。评测显示,推理深度增加时准确率最高下降51%,对抗性数值扰动下下降47个百分点。对开源模型进行轻量 LoRA 微调后,在留出问题上准确率从81.1%提升至85.6%,并在 FinQA 上比基础模型高5个百分点。

原文 · arXiv cs.AI

V-FiLLM: Verified Financial LLM Reasoning Benchmark

While existing benchmarks have made substantial progress in evaluating LLMs across STEM domains, financial reasoning over structured data remains comparatively less explored. We introduce V-FiLLM, a framework that generates financial reasoning benchmarks from executable computation trees grounded in real tables, yielding items whose answers are correct by construction. Trees are evaluated symbolically to obtain ground truth and rendered into natural-language questions, removing any model from the labeling loop, so items can be generated at arbitrary scale without annotation cost and without inheriting a generator's error rate. V-FiLLM exposes four independently controllable axes of difficulty including computation depth, expression breadth, financial concept complexity, and context size. By evaluating on open-source models, we find that accuracy falls up to 51% as reasoning depth increases, and up to 47% points under adversarial numerical perturbations, highlighting remaining challenges in robust financial reasoning over tables. We further show that lightweight LoRA fine-tuning on verified chain-of-thought traces improves accuracy from 81.1% to 85.6% on held-out problems and outperforms the base model by 5% points on FinQA (Chen et al., 2022a), s), suggesting that targeted, low-cost adaptation is a promising direction for compositional reasoning in financial QA.