医疗AI开发者终于有了一个能真正测试长期记忆能力的基准——MediLongChat让跨会话推理变得可评估，做医疗对话系统的团队建议直接拿来跑跑看。
研究团队提出一个框架，用于合成高质量、长期医疗对话数据集MediLongChat，以评估医疗AI代理的记忆和推理能力。该框架通过知识引导分解为三个阶段：构建具有多样疾病和并发症轨迹的患者档案、生成每次就诊的多轮对话、整合为连贯的纵向历史数据集。他们设立了三个基准任务（对话内推理、跨对话推理、合成推理）来测试医疗代理的记忆能力。实验表明，即使最先进的LLM在MediLongChat上也表现不佳，凸显了该基准的挑战性和开发定制方法的必要性。
Synthesis and Evaluation of Long-term History-aware Medical Dialogue
An effective healthcare agent must be able to recall and reason over a patient's longitudinal medical history. However, the absence of datasets with realistic long-term dialogue timelines limits systematic evaluation. Real clinical text is constrained by privacy and ethics, while existing benchmarks focus on isolated interactions, failing to capture cross-session reasoning. We introduce a framework for synthesizing high-quality, long-term medical dialogues with LLMs. Our approach entails a knowledge-guided decomposition into three stages: constructing synthetic patient profiles with diverse disease and complication trajectories, generating multi-turn dialogues per encounter, and integrating them into a coherent longitudinal history dataset, MediLongChat. We establish three benchmark tasks-In-dialogue Reasoning, Cross-dialogue Reasoning, and Synthesis Reasoning-to evaluate the memory capabilities of healthcare agents. To assess data quality, we introduce a multi-dimensional evaluation framework combining vector-based metrics with LLM-as-a-judge assessments. Specifically, we define automatic measures-Faithfulness, Coherence, and Diversity-together with two LLM-based evaluations: Correctness and Realism. Benchmark experiments show that even state-of-the-art LLMs struggle with MediLongChat. These findings highlight the benchmark's applicability and underscore the need for tailored methods to advance healthcare agents.