LLM作为裁判评估中，去偏置会降低判断精度
Debiasing as a Measurement Intervention: Calibrated Ties and Resolution Loss in LLM-as-a-Judge Evaluation
这个研究很实用，它告诉你怎么用LLM作为裁判评估时，去偏置操作会带来什么后果，比如会降低判断精度，所以想用这个方法的人一定要看看。
研究指出，在LLM作为裁判的评估中，通过指令忽略引用格式等呈现线索的去偏置干预，虽然能抑制偏见，但会损害测量工具的分辨率。他们创建了TraceJudgeBench基准，测试了GPT-5.5、Claude Sonnet 4.6和DeepSeek V4-Flash等模型，发现更强的反引用提示词可以将引用不当的获胜率从50.5%降至0%，但在严格压力测试前，一些操作点已经将验证过的中等质量差距决策转为平局，而正确性冲突的准确率仍保持在93.0%以上。人类验证区分了三种平局含义：正确等价平局、校准软边界平局和分辨率破坏平局。
Debiasing as a Measurement Intervention: Calibrated Ties and Resolution Loss in LLM-as-a-Judge Evaluation
LLM-as-a-judge protocols are commonly debiased by instructing judges to ignore presentation cues such as citation formatting, source labels, and evidence-display style. We show that this intervention can suppress bias while damaging the resolution of the measurement instrument. We introduce TraceJudgeBench, a diagnostic benchmark for auditing citation-like artifacts in RAG and agent-workflow evaluation, covering content-equivalent pairs, citation ablations, correctness conflicts, human-validated soft and moderate quality gaps, prompt-strength ladders, decoupled judging, and a controlled workflow-ranking probe. Across GPT-5.5, Claude Sonnet 4.6, and DeepSeek V4-Flash, stronger anti-citation prompts reduce worse-cited wins from up to 50.5% to 0%; yet some operating points already convert validated moderate-gap decisions into Tie before the strict stress-test endpoint, while correctness-conflict accuracy remains at or above 93.0%. A second, 50-pair FinQA moderate-gap construction reproduces the qualitative frontier, and open-weight Qwen2.5-14B-Instruct-AWQ and Gemma-3-12B-IT runs reproduce the central HotpotQA frontier. TRACE-style decoupling recovers 96.5-100.0% better-plain resolution across the reported settings. Human validation separates three meanings of Tie: correct equivalence Tie, calibrated soft-boundary Tie, and resolution-destroying Tie on validated quality gaps. We frame debiasing as a measurement intervention whose bias suppression, resolution retention, Tie cost, and protocol cost must be reported jointly. The supplementary artifact contains benchmark splits, prompts, raw judge outputs, validation summaries, and analysis.
