这个新基准EMPATH专测情感支持聊天机器人的安全漏洞，用AI模拟求助者进行多语言多轮对话，发现主流模型评分虚高且不稳定，值得一做。
EMPATH是一个多语言审计-法官基准，用于评估情感支持聊天机器人的安全性。该基准使用审计模型模拟求助用户，基于140个种子指令和34个人设生成多轮对话，法官模型从19个指标（分属五个维度）评分。基准在墨西哥西班牙语和美国英语上构建，研究发现标准评分在19个指标中的10个上存在膨胀，校准后恢复了区分度。在三个前沿模型（含一个开源模型）上测试，聚合分数差异在0.74分内，但具体指标差异可达6分。运行间可靠性差，deepseek-v4-pro在温度0下每次运行生成不同对话。
EMPATH: A Multilingual Auditor-Judge Benchmark for Safety Evaluation of Emotional-Support Chatbots
Safety benchmarks often buy scalability by fixing the prompt, the language, and the turn structure. For emotional-support chatbots, that bargain hides precisely where safety failures emerge: across a multilingual, multi-turn crisis conversation. We present EMPATH, a benchmark for safety evaluation of emotional-support chatbots. An auditor model role-plays help-seeking users, generating multi-turn conversations from 140 seed instructions and 34 personas. A judge model scores each full transcript against 19 metrics across five dimensions: crisis handling, therapeutic quality, conversational integrity, emotional safety, and cultural adaptation. EMPATH is built for Mexican Spanish and US English; the studies reported here run in Mexican Spanish. Auditor and judge are drawn from different model families, and the judge is treated as an instrument to be calibrated rather than trusted. A strict per-criterion rubric reveals material score inflation on 10 of the 19 metrics and restores discrimination. We study the measurement properties of the benchmark through judge calibration and cross-family inter-judge agreement. We also illustrate EMPATH on three frontier models, one of them open-weight. Aggregate scores sit within 0.74 points of one another, but per-metric profiles diverge by up to six points in model-specific places. Under the standard rubric, both the ranking and the weak spots are stable across a second, cross-family judge: 93% of scores fall within plus or minus 1. A five-run test-retest adds a second axis: even the steadiest model swings from 2 to 10 on a crisis metric across identical re-runs, and deepseek-v4-pro returns a different conversation on every run even at temperature 0. Run-to-run reliability is therefore a per-model safety property, not noise to average away. EMPATH is system-agnostic; the pipeline, seeds, personas, and rubrics are released for reuse.