这篇论文很实在，直接指出LLM-as-a-Judge在多语言场景下不靠谱，只找出33篇相关研究还一堆问题。做NLP评估的朋友建议看看。
该论文分析了ACL Anthology中650篇提及LLM-as-a-Judge的论文，发现仅33篇关注低资源或多语言场景。研究发现存在不一致的评价结果、对LLM判断过度信任以及普遍依赖单一评判模型的问题。作者基于分析提出了针对多语言和低资源环境下使用LLM-as-a-Judge的具体建议。
Challenges and Recommendations for LLMs-as-a-Judge in Multilingual Settings and Low-Resource Languages
LLM-as-a-Judge has become the dominant evaluation paradigm for many natural language generation tasks, due to shortcomings of conventional metrics and high correlations with human judgment, albeit mostly in English. There are now attempts to extend LLM-as-a-Judge to multilingual settings including low-resource languages. However, LLMs have limited proficiency in low-resource languages, and there is often no adequate human validation in these settings. To highlight the scope of the problem and current practices, we explore the use of LLM-as-a-Judge evaluators in ACL Anthology papers focusing on multilingual settings and low-resource languages across a diverse set of tasks. Out of 650 papers mentioning LLM-as-a-judge, only 33 of them focus on low-resource or multilingual settings. Our in-depth analysis of these papers indicates inconsistent evaluation outcomes, a tendency to overtrust LLM judgments in multilingual settings, and the widespread reliance on a single judge model per study. To help the NLP community further, we conclude with recommendations about how to use LLM-as-a-Judge in multilingual and low-resource settings.