这篇论文对自动化事实核查系统进行了全面的跨基准评估，揭示了不同系统在不同领域的表现差异，对于理解自动化事实核查技术的局限性和改进方向具有重要意义。
本研究评估了九种自动化事实核查系统，涵盖从随机基线到微调Transformer、零样本LLM和AVeriTeC 2025任务中排名前两位的系统，并在四个数据集上进行了测试。研究发现，在气候核查任务中，基于ClimateCheck的模型优于零样本LLM和AVeriTeC 2025系统，表明噪声证据会降低真实性预测；系统排名强烈依赖于领域和指标；用金标签替换检索到的证据可以提高模型的真实性准确性14-22个百分点，证实检索仍然是主要瓶颈。
How Robust Are Automated Fact-Checking Systems? A Cross-Benchmark Evaluation
Automated fact-checking (AFC) systems retrieve evidence and predict claim veracity, yet evaluations omit simple baselines, systems are developed for a single benchmark and cannot be trusted to generalise across domains. No prior work cross-evaluates the full two-stage retrieve-then-verify pipeline across diverse datasets, complementing retrieval-only studies (Thakur et al., 2021) and single-stage benchmarking studies (Calamai et al., 2025). We benchmark nine models, ranging from random and sparse baselines to fine-tuned transformers, zero-shot LLMs, and the two highest-ranked systems from the AVeriTeC 2025 shared task, across four datasets spanning scientific, open-web, and climate domains. Three findings stand out: (1) on ClimateCheck claim-only and fine-tuned models outperform zero-shot LLM and top-performing AVeriTeC 2025 systems, highlighting that noisy evidence can degrade veracity prediction; (2) system rankings are strongly domain- and metric-dependent: the best model on SciFact (macro-F1 0.70) drops to 0.31 on ClimateCheck, while the AVeriTeC 2025 winner and runner-up swap rankings based on evaluation metrics and datasets; (3) replacing retrieved evidence with gold annotations improves veracity accuracy by 14-22 points across models, confirming retrieval remains primary bottleneck. We release code, pre-processed datasets, and all results to support reproducible AFC research.