做对话系统评估的团队终于有了一个可复现、低成本的替代方案——无需调用大模型就能衡量对话的语义进展，建议做客服或问答系统的开发者试试这个指标。
该研究提出了一种基于信息论的多轮对话语义进展度量方法，通过计算对话中问题相关且非冗余信息的累积量来评估对话质量。核心指标使用高斯模型在嵌入空间中近似不确定性减少，具有单调性、可加分解和冗余证据递减等理论性质。实验表明，该方法在MT-Bench、Chatbot Arena和UltraFeedback上与人类判断高度一致，甚至优于部分基于LLM的评判方法。该指标无需自回归推理，仅需轻量级嵌入模型即可在CPU上运行，显著降低了评估成本。
Measuring Semantic Progress in Multi-turn Dialogue via Information Gain
Evaluating multi-turn dialogue is challenging because quality emerges across turns rather than within individual responses. We focus on a key dimension of information-seeking dialogue: semantic progress, defined as the accumulation of new, question-relevant, and non-redundant information over the course of a conversation. We formalize semantic progress as question-conditioned uncertainty reduction and introduce an information-theoretic metric that approximates it in embedding space. Our main estimator uses a tractable Gaussian formulation with closed-form updates, while a complementary maximum-entropy argument shows why log-determinant structure arises more broadly when only second-order embedding information is retained. This formulation yields desirable theoretical properties, including monotonicity, additive decomposition of total information gain across turns, and diminishing returns for redundant evidence. Unlike LLM-as-a-judge approaches, our metric requires no autoregressive inference at evaluation time and is fully reproducible for a fixed embedding model. Experiments on MT-Bench, Chatbot Arena, and UltraFeedback show that the proposed metric achieves competitive agreement with human judgments despite targeting only semantic progress, with improved alignment on MT-Bench and UltraFeedback compared to several LLM-based judges. Notably, the method remains effective with lightweight embedding models under CPU-only execution, indicating that semantic progress can be captured without reliance on large model capacity.