Papers11月11日 04:54

Meta 发布 AutoRefine：让 LLM 代码评审判官保持校准

elvis@omarsar0

Post

Why we picked it

Meta 这篇论文解决了一个很实际的问题：agent 一更新，judge 提示词就失效。AutoRefine 拿测试通过和 PR 合并这些现成信号自动改提示词，一致性从 0.16 拉到 0.51。做 agent 评测的工程师应该看看。

Meta 发表的 AutoRefine 论文针对 LLM 代码评审中的提示词失效问题：当被评估的编码 agent 更新后，judge 的提示词很快过时，评估结果随之失真。AutoRefine 利用团队已有的信号（测试是否通过、改动是否被合并）自动重写 judge 提示词，无需人工重新标注。在 Meta 已部署的编码 agent 上，judge 与真实结果的一致性从 0.16 提升到 0.51。论文编号为 arXiv:2611.09291。

Body

其核心机制是把人工标注环节替换为部署流水线中天然产生的结果信号，让 judge 提示词随 agent 演化持续刷新，而不是靠人工周期性维护。这解决了 LLM-as-a-judge 在实际生产中最常见的漂移问题。

对任何在 CI 或 agent 流水线中用 LLM 评审代码质量的团队，这个思路可以直接借鉴：不需要额外标注预算，复用已有的测试与合并数据即可校准 judge。

Original · elvis

Great paper from Meta with practical tips for tuning LLM judges. This one is on keeping LLM code judges calibrated and tested on a deployed coding agent. LLM judges often decide whether an agent's code change is good. But their prompts go stale every time the agent changes. AutoRefine fixes the prompt using signals teams already have, like whether tests passed and whether the change got merged. On Meta's deployed agent, agreement between the judge and real outcomes rose from 0.16 to 0.51. arxiv.org/abs/2611.09291 Chat with Paper: academy.dair.ai/papers/autoref…📊 11 ⚡

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include LLM-as-a-judge, Meta, 编码Agent, 提示词工程.

Source attribution: curated from elvis (@omarsar0).

#LLM-as-a-judge #Meta #编码Agent #提示词工程 #AutoRefine

Keep reading

Recent stories with the same tag or section

How-to BuildersMulti-sourceSelected13:35

LangChain Q&A Guide on Agent Evals

The practical takeaway is the failure feedback loop: every production failure becomes a new dataset sample, turning evals into a continuously growing asset rather than a one-off gate. Engineers building or maintaining agents can adopt the graded evaluator stack — deterministic code checks before LLM judges, pairwise comparison over absolute scores — to cut cost and improve stability. It pairs well with LangSmith-style tooling and contrasts with teams that still rely on manual spot checks.

#LangChain #agents #evals #LLM-as-a-judge

Event

shao__meng@shao__mengSource

Products BuildersSelected18:35

DeepEval team releases JevEval, replacing generative LLM judges with Jev

Moving scoring logic out of prompts and into code makes evaluations auditable and reproducible, which matters for teams running agent traces in production. Practitioners building agent evaluation pipelines can use the framework to pin down exactly what their judge checks, per-question weights and N/A handling. Compared with conventional LLM-as-a-judge setups like GPT-4 grading in a single call, JevEval trades flexible free-text verdicts for deterministic, explainable scores.

#JevEval #DeepEval #LLM-as-a-judge #agents

shao__meng@shao__mengSource

Papers Builders09:02

Chinese content新框架显式校正 LLM 评审偏差，用更少对比恢复可靠排名

做模型评测的朋友看看这篇：它把 LLM 当裁判的位置偏好、啰嗦偏好这些偏差直接建模掉，能省大量对比次数。

#LLM-as-a-judge #评测 #偏差校正 #潜变量模型

arXiv cs.LG@Harshita Katoch et al. (4)Source

PapersSelected04:39

Nature paper examines LLM-as-a-judge for clinical AI evaluation

Automated judging could ease the expert-review bottleneck that slows clinical AI validation, though the paper leaves open how reliably models grade safety-critical medical errors. Health AI teams building evaluation pipelines and regulators weighing validation standards for clinical tools will find the methodology directly applicable. It complements LLM-as-a-judge work from general-domain benchmarks like MT-Bench by focusing on medical output quality.

#LLM-as-a-judge #clinical AI #evaluation #npj Digital Medicine