研究用15槽信息框架评估 LLM 危机报告，Gemini 与人类表现打平
Looking Complete Is Not Being Grounded: Evaluating LLM Sitreps Against Human-Authored Situation Reports
一个把 LLM 危机报告和 121 份真实人道主义报告对着比的研究，数字看着打平，其实来源信息全缺，评估指标也翻车了。
该研究从专业人道主义情况报告中提炼出 15 槽信息框架，对 121 份人类报告和 261 份 LLM 报告进行跨 5 场灾害、3 类灾害的评估。在 Cyclone Idai 上，google/gemini-3.7-flash 评判下最优机器报告得 0.827，名义上略超人类上限 0.821，但 95% bootstrap CI [-.049,.059]、p=.852 表明差异不可区分，且该排序在 openai/gpt-5.6-luna 下不复现。三场主要事件中，机器报告仅复现 5.4-11.0% 的当日官方数字，报告日期、发布机构等来源信息在全部 12 个事件-评判组合中均未达到专业报告覆盖度。ROUGE-L 无区分力，BERTScore 经多重性校正后与完整性负相关，说明传统自动指标不适合评估危机报告。
Looking Complete Is Not Being Grounded: Evaluating LLM Sitreps Against Human-Authored Situation Reports
Evaluation for automatic crisis reporting should measure what practitioners need, not what is convenient to compare. Through LLM-assisted, author-approved content analysis, we derive a 15-slot information schema from professional humanitarian situation reports using qualitative content analysis, then evaluate 121 human and 261 LLM-generated reports across five disasters and three hazard types. Under google/gemini-3.7-flash on Cyclone Idai, the best machine arm is nominally above the empirical human ceiling (0.827 vs. 0.821), but the difference is statistically indistinguishable from zero (95\% bootstrap CI $[-.049,.059]$; permutation $p=.852$), and the ordering does not replicate under openai/gpt-5.6-luna. On a blind 100-item Idai sample, the author agrees with the Gemini judge at 0.910 accuracy and unweighted Cohen's $κ=.799$. All four judges clear the plan's raw-agreement criterion of .7; only Gemini reaches .7 on chance-corrected $κ$. Across the three primary events, machine reports reproduce 5.4-11.0% of same-day official figures. Across those same events, raw exact-number matching links 14.3-19.6% of official figures and 85.4-90.1% of machine figures to the social stream; manual audit confirms 45/50 declared absences, 4/25 official matches, and 24/25 machine matches semantically. The issuing-provenance gap is positive in all 12 observed event-judge cells: report date, organisation, series, and authority were not supplied at professional-report coverage in any cell. On the three primary events, ROUGE-L is uninformative, BERTScore is negatively associated with completeness after multiplicity correction, and a generic LLM judge changes sign.
