ImplicitHop 基准揭示 LLM 跨上下文与权重的知识整合能力差距
ImplicitHop: Evaluating knowledge integration across context and memory
给做 agent 和微调的同学：这篇用图基准量化了模型对权重里知识的推理比上下文差多少，写提示词时把关键事实放进上下文能直接提升准确率。
论文提出两个自然语言基准 ImplicitHop，用于评测 LLM 在不同推理深度下的演绎推理能力：一个是可控图拓扑的形式化图基准，另一个基于真实世界知识图谱的知识组合基准。测试对象是一批开源模型，全程不使用思维链。结果显示，随组合长度增加，预训练知识和上下文知识的准确率都急剧下降。上下文信息的推理表现明显优于存储在模型权重中的信息。
ImplicitHop: Evaluating knowledge integration across context and memory
The ability to integrate new information and deduce its consequences is central to learning, understanding, and reasoning in large language models (LLMs). Whether information arrives during training or in-context, models must not only store local facts, but propagate their long-range implications globally. To explore information integration in LLMs, we introduce two complementary natural language benchmarks for large-scale evaluation of deductive reasoning as a function of inference depth. The first is a formal graph benchmark enabling controlled evaluation over a wide range of graph topologies and integration tasks; the second is a knowledge composition benchmark grounded in real-world knowledge graphs. We use these benchmarks to compare knowledge integration from information encoded in context versus information encoded in model weights during pretraining, evaluating a set of open models implicitly, without chain of thought. We find that accuracy decays sharply with increased composition length for both pretrained and in-context knowledge. Information presented in-context, however, substantially outperforms deduction over information encoded in weights. These results quantify a fundamental limitation of current models with direct implications for prompt-based control of agent behavior and for continual learning strategies, and they provide a foundation for developing training, prompting, and integration methods that support deeper reasoning.
