开源模型能否媲美闭源模型?金融文本理解基准更新
Can Open-Weight Models Compete on Financial Text Comprehension?
想知道开源模型在金融领域到底行不行?这份基准测试用数据说话,Kimi K2.6和GLM 5的表现可能会让你意外。
更新后的Financial Touchstone基准包含2967个问答对,覆盖495份国际年报。测试了20个模型,包括GLM 4.
7、GLM 5、Kimi K2.6、DeepSeek V3.
2等开源模型及阿里Qwen3-Max。Claude Opus 4.
6准确率最高达88.4%,Gemini 2.
5 Pro幻觉率最低为0.08%。
Kimi K2.6准确率排名第三,GLM 5和Mistral 3分列第四、第五,挑战了推理架构或专有权重的必要性。
信息检索是主要瓶颈,占失败原因的48.9%。
Can Open-Weight Models Compete on Financial Text Comprehension?
Open-weight language models from Chinese AI labs caught up on benchmarks relative to proprietary frontier models in recent months. Yet their reliability on real-world financial tasks remains largely untested. We updated the Financial Touchstone benchmark, which now has 2,967 question context-answer triplets across 495 international annual reports. We also apply a new set of models on the benchmark, expanding coverage from eleven to twenty models across ten providers, including recent open-weight models such as GLM 4.7, GLM 5, Kimi K2.6, and DeepSeek V3.2, as well as Alibaba's proprietary flagship Qwen3-Max. Anthropic's Claude Opus 4.6 achieves the highest accuracy (88.4%), while Google's Gemini 2.5 Pro maintains the lowest hallucination rate (0.08%). Notably, the open-weight Kimi K2.6 ranks third in accuracy, and the non-reasoning models GLM 5 and Mistral 3 rank fourth and fifth, challenging the assumption that reasoning architectures or proprietary weights are a prerequisite for strong financial comprehension. Information retrieval remains the primary bottleneck, accounting for 48.9% of all failures. We also document a new finding: geopolitical content filters in Chinese models refuse legitimate financial questions (0.08% of attempts), sometimes without clear reason, and the refusal behavior depends on the access route as much as on the model. The complete dataset and evaluation framework are publicly available.
This report is categorized under AITOP's "Papers" desk. Key entities and technical topics under continuous tracking include Financial Touchstone, Kimi K2.6, GLM 5, DeepSeek V3.2.
Primary source verification: published directly by arXiv: DeepSeek with first-party authority.