arXiv 论文提出 CrossProfile 基准，揭示 LLM 群体决策记忆机制的失败模式
Rare, Relevant, and Still Wrong: How Memory Fails LLM Group Decisions
一篇挺有意思的论文：测试了 Qwen2.5、Llama-3.1 和 DeepSeek 在群体讨论记忆上的表现，发现相关性筛选会引入干扰信息，去掉归属标签反而准 13-24 分。
arXiv 论文提出 CrossProfile 诊断基准，包含 60 个模板、480 个任务，用于测试 LLM 群体讨论压缩成记忆后的决策表现。在 Qwen2.5-7B、Llama-3.1-8B 和两个 DeepSeek 模型上，按相关性筛选记忆会引入干扰项，在经典 hidden-profile 配置下比按稀有度筛选低 23 到 31 分。按稀有度筛选平均不优于随机选择，而决策模型还会给多人提到的信息额外加权，属于社会证明偏差，移除归属标签可将准确率提升 13 到 24 分。
Rare, Relevant, and Still Wrong: How Memory Fails LLM Group Decisions
When an LLM group's discussion is compressed into a short memory for a final decision maker, what should the memory keep? Two natural heuristics keep the rarest claims or the claims relevant to the decision criteria. On hidden-profile tasks both look attractive, because how widely a fact is held is confounded with whether it helps. We introduce CrossProfile, a diagnostic benchmark of 60 templates with automatically verified answer keys, in which the ownership of supporting, decoy, and irrelevant facts is crossed while their text is held fixed (480 tasks). Across Qwen2.5-7B, Llama-3.1-8B, and two DeepSeek models, both heuristics fail in predictable places. Selecting by relevance readmits decoys, because decoys are on topic: even a perfect relevance filter loses 23 to 31 points to rarest-first selection in the classic hidden-profile configuration. Rarest-first selection is no better than random selection on average and wins only where the rare facts happen to be the useful ones. In both cases accuracy follows the balance of evidence the memory keeps, not the policy that kept it. The decision model adds a third failure, a form of social proof: it gives extra weight to facts attributed to many agents. Removing the attribution tags, or reassigning them at random with line order fixed, raises accuracy in the classic configuration by 13 to 24 points. In natural discussions the evidence is lost before any memory acts: the complete transcript yields lower accuracy than no notes at all.