法律从业者和AI开发者会关心:LLM 在法律场景中的时间失效问题直接关系到合规风险,RAG 方案已被证明能有效缓解,值得在实务中尝试。
大型语言模型在静态知识截止日期与动态法律条文之间存在根本矛盾,导致两种时间失效模式:一是模型在立法修订后仍使用旧规则(后截止日期失效),二是模型偏好新条款而忽略历史版本(近因偏差)。研究者构建了包含312个专家验证的德语法律问答基准,涵盖三类时间敏感问题,并评估了OpenAI、Anthropic和DeepSeek的五种模型。实验发现,在无辅助的推理设置下,模型在后截止日期场景中表现严重下降;检索增强生成(RAG)方法通过提取事实日期和版本过滤显著提升所有问题类型的准确性,而网络搜索则不稳定且加剧近因偏差。研究结论指出,可靠的法律问答必须将时间有效性作为硬约束。
Asking For An Old Friend: Diagnosing and Mitigating Temporal Failure Modes in LLM-based Statutory Question Answering
Large language models are increasingly used for legal research, yet their fixed training cutoffs and reliance on static parametric knowledge are at odds with the evolving nature of statutory law. We study two temporal failure modes: post-cutoff staleness, where models apply superseded rules after legislative amendments, and recency bias, where models prefer newer provisions even when a historical version governs the fact pattern. To this end, we present a benchmark of 312 expert-validated, time-sensitive German statutory QA pairs spanning three categories: Post-Cutoff Amendment Questions, Pre-Amendment Questions, and Multi-Provision Pre-Amendment Questions. We evaluate five LLMs by OpenAI, Anthropic and DeepSeek under four inference settings: Vanilla, Web-search, and two retrieval-augmented variants that enforce temporal validity via a fact date extraction and version filtering. Using an LLM-as-a-judge validated against human expert ratings, we find severe degradation in the Vanilla post-cutoff setting. Both RAG approaches substantially improve performance across all question types, while web search yields unstable gains and exhibits a marked recency bias on historically anchored tasks. Our results indicate that reliable legal QA requires treating temporal validity as a hard constraint.