LoCoMo 得分拆解:评委与答案提示词对分数影响可达 9.3 分
What Moves a LoCoMo Score? Memory Context, Answer Prompts and Judges
如果你常看各家记忆系统的 LoCoMo 分数,这篇会改变你的读法:换个评委分数能差 7.2 分,换个提示词差 9.3 分。作者还把全部答案和评分脚本开源了,做 agent 记忆或跑基准的开发者都该看。
一篇 arXiv 论文系统测量了长对话记忆基准 LoCoMo 分数对评测配置的敏感度。实验覆盖 10 段对话共 1,540 个问题,使用 gpt-4o-mini 与 gpt-4.1-mini 两个答案模型、两种答案提示词和三个 LLM 评委:只更换评委最多使分数变动 7.2 分,只换答案提示词最多变动 9.3 分。
论文以黑盒方式评测了面向编程智能体的团队记忆系统 Becos。Becos 从比整段对话少 90% 字符的上下文中生成答案,在 gpt-4o-mini 作答时显著高于整段对话基线(短提示词下 79.7 对 73.0,差值 95% 置信区间 +3.8 到 +9.5;分步提示词下 88.8 对 77.5),但换用 gpt-4.1-mini 时两者基本持平(83.2 对 83.1,92.5 对 92.1)。
作者对 150 个答案做了盲测人工核查,发现评委在意见一致时可靠,且主要评委在有争议答案上的误判反而不利于 Becos。论文同时指出 Becos 的配置是在 LoCoMo 本身上调出来的,没有留出测试集,并已公开全部答案、判定结果和评分脚本。
What Moves a LoCoMo Score? Memory Context, Answer Prompts and Judges
LoCoMo is a widely reported benchmark for long-term conversational memory, but published scores mix memory quality with choices of answer prompt, judge and baseline layout that are rarely reported together. We measure how much these choices move a score, and evaluate Becos, a team memory system for coding agents, as a black box under all of them: ten conversations (1,540 questions), two answer models (gpt-4o-mini, gpt-4.1-mini), two answer prompts and three LLM judges, against a whole-conversation baseline rerun under identical conditions. Changing only the judge moves a score by up to 7.2 points and changing only the answer prompt by up to 9.3. A blind human check of 150 answers finds the judges reliable when they agree, and the primary judge's errors on disputed answers working against Becos rather than for it. Becos answers from a context with 90% fewer characters than the whole conversation. Its configuration was developed on LoCoMo itself, so there is no held-out set. With gpt-4o-mini answers it scores clearly higher than the whole conversation (79.7 vs 73.0 with the short prompt, paired 95% interval for the difference +3.8 to +9.5; 88.8 vs 77.5 with a step-by-step prompt). With gpt-4.1-mini answers the two score about the same (83.2 vs 83.1; 92.5 vs 92.1). We release every answer and verdict and the scoring script.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include LoCoMo, Becos, gpt-4o-mini, LLM-as-judge.
Source attribution: curated from arXiv cs.AI (@Nikhil Kumar Jha, Amitoj Cheema).