DeepSeek-Backed Benchmark Tests LLM Backtest Auditing With Clean Controls
Can LLMs Catch a Rigged Backtest? A Clean-Control Calibration Benchmark
The non-obvious result is that recall-only evaluations inflate auditor quality: specificity against clean strategies is where models actually diverge, by as much as 79 points. Quants and fintech teams deploying LLMs to audit trading backtests should adopt paired clean-control scoring and clean-aware prompts, since a single-line warning eliminated DeepSeek's false positives in this setup. The benchmark and deterministic scorer offer a reusable template for calibrating any audit-style LLM task.
Researchers built a 96-item paired benchmark where each flawed backtest has a clean control differing by one methodology detail, holding strategy, dates, code style and reporting fixed.
They ran 1440 cached audits across four text endpoints with a deterministic scorer measuring flaw recall, false positives on clean controls, evidence localization and fix relevance.
The primary DeepSeek auditor hit 100.0% closed and clean-aware code recall, but open prompts falsely flagged 93.
8% of clean controls.A clean-aware warning cut DeepSeek's code false positives from 20.
8% to 0.0% at unchanged recall, while a budget anchor still flagged 38 of 48 clean controls.
Reporting recall alone would rank three of the four models identically, whereas the clean-control rate separates them by 79 points.
Can LLMs Catch a Rigged Backtest? A Clean-Control Calibration Benchmark
Backtest auditing is a calibration problem: high flaw recall is not useful when the model falsely flags matched clean strategies. We build a 96-item paired benchmark in which every flawed backtest has a clean control that holds strategy, dates, code style, labels, and reporting scaffold fixed while changing one methodology detail. A deterministic scorer separates flaw recall, clean-control false positives, evidence localization, and fix relevance. Over 1440 cached audits from four text endpoints, the primary DeepSeek auditor reaches 100.0\% closed and clean-aware code recall, but open prompts over-flag 93.8\% of clean code controls, and clean-aware all-three specificity is 87.5\% even where recall saturates. A clean-aware warning drops DeepSeek code false positives from 20.8\% (95\% CI 11.7--34.3) to 0.0\% (0.0--7.4) at unchanged recall, while the budget anchor still flags 38/48 clean controls under the same prompt. Reporting recall alone would rank three of these four models identically; reporting the clean-control rate separates them by 79 points.
This report is categorized under AITOP's "Papers" desk, focusing on China's AI industry and domestic foundation model ecosystem. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include DeepSeek, benchmark, AI safety, reasoning models.
Primary source verification: published directly by arXiv: DeepSeek with first-party authority.
