这篇论文用LAMBDA系统在153个任务上测了三种自动评分方法，发现宽松LLM评分召回率97%，严格规则召回率靠关键词提取提高60个百分点。想看AI评分够不够靠谱的可以读。
论文以LAMBDA多智能体数据分析系统在DSGym的153个数值QRData任务上为例，研究自动评分可靠性。三层人机评分级联（严格正则匹配、LLM宽松评分、代码片段人工检查）中，两个自动评分器在70个假阳性上达到100%精确率。宽松评分器相比人工标签召回率为97%。关键词锚定提取方案将严格评分器召回率比最后数字启发式提高60个百分点，迭代提示机制将评分运行成功率从36%提升至97%，宽松通过率从16%提升至46%。变量类型是任务元数据中最一致影响评分动态的字段。
Grading the Grader: Lessons from Evaluating an Agentic Data Analysis System
Agentic data analysis systems produce rich outputs, including code, numerical results, and verbal diagnostics. This makes them more challenging to evaluate than single-turn LLM responses. It is therefore necessary to distinguish genuine disagreement between an agent's output and a ground-truth answer from grading artifacts. We investigate how reliably automated graders assess such a system and what strategies improve grading quality by applying LAMBDA, a multi-agent data-analysis system, on 153 numerical QRData tasks from DSGym. We develop and evaluate a three-layer human-AI grading cascade: strict regex matching, LLM-based lenient grading, and snippet-based human inspection, which combines non-GenAI and GenAI strategies with different failure profiles. Both automated graders achieve 100% observed precision (0/70 false positives). The lenient grader's recall is 97% against human labels. A keyword-anchored extraction pipeline raises the strict grader's recall by 60 percentage points over a last-number heuristic; the lenient grader is architecturally parser-independent. An iterative nudge mechanism raises grading run success from 36% to 97% and lenient-pass rates from 16% to 46%; comparing nudging with and without original-question re-injection shows that re-injection offers no benefit, confirming the nudge as an answer template cue. We further observe in this case study that variable type is the task metadata field most consistently associated with grading pipeline dynamics and observed outcome grades.