This paper introduces a new reporting protocol for scientific reports that could significantly improve reproducibility. It's a must-read for anyone interested in improving the quality of scientific reporting and the use of LLMs in research.
This paper proposes claim-locked reporting, a protocol that ensures the statistical evidence in scientific reports is fixed by structured results before prose generation. It improves reproducibility by 37.4 and 20.5 points in fMRI and randomized controlled trial reporting, respectively. DeepSeek's fMRI cost analysis demonstrates lower token use and generation latency with this method.
Large language models (LLMs) can fluently verbalize statistical evidence, yet statistical reports can still drift numerical values, invert effect directions, or restate thresholded contrasts as categorical effects. We frame these failures as a control problem: the evidence-bearing content of a scientific report should be fixed by structured statistical results rather than sampled during prose generation. We therefore use cross-run reproducibility to stress-test whether report-visible numbers and claims are bound before prose generation. Existing controls operate at the text or slot level; a deterministic hybrid template reproduces only 61.1% of report-visible numerical content across seeds because the LLM still selects which findings and numbers the template renders. We propose claim-locked reporting, a provenance-before-prose protocol that fixes the evidence source, numbers, direction, and allowed language strength of each reportable claim before the LLM writes only connective prose. Across fMRI functional-connectivity reporting and randomized controlled trial reporting on Evidence Inference 2.0, claim-locked reporting improves reproducibility over the hybrid template by 37.4 and 20.5 points, respectively. Blinded human audits support the observed direction-preservation and governance trends. In an fMRI cost analysis with DeepSeek, claim-locked reporting also yields the lowest observed token use and median generation latency.