Papers10月27日 10:32

研究：用 DeepSeek-V3.2 模拟 ICLR 2024 审稿评分，提示词设计对齐程度差异显著

Interpreting LLM Scores in Peer Review: Alignment and Divergence Across Prompts, Domains, and Manuscript Features

arXiv: DeepSeek@Wenqiang Qin et al. (4)

Paper

Why we picked it

想让 LLM 帮忙审论文或做评分的人该看看：同一批 ICLR 投稿上，提示词怎么写能让 DeepSeek-V3.2 的分差从 1.46 降到 0.2，数据很扎实。

一项研究在 7,083 篇 ICLR 2024 公开投稿上比较四个 LLM 的评分与人类评审的一致性，并用 DeepSeek-V3.2 测试十种提示词条件。结果显示，在提示中加入 Area Chair 指导后，评分均值偏差从 1.460 降至 0.199，绝对误差从 1.577 降至 0.942，而排序相关系数仅从 0.377 变为 0.387。历史引用数据结合向下打分的指令使均值偏差降到 0.052，但排序相关为 0.321。论文还发现标题越长与评分偏差呈正相关，公式密度则与偏差呈负相关，且领域差异在全局校准后依然存在。

Original · arXiv: DeepSeek

Interpreting LLM Scores in Peer Review: Alignment and Divergence Across Prompts, Domains, and Manuscript Features

Large language model (LLM) scores are being studied as information for scholarly evaluation. Interpreting these scores requires evidence about agreement with human judgments and its variation across evaluation conditions. Guided by an argument-based approach to validation, we compare four LLMs on 7,083 public ICLR 2024 submissions under a standard review condition and examine ten prompt conditions using DeepSeek-V3.2. We distinguish average score divergence, paper-level absolute error, and rank correlation, then examine domain differences and adjusted manuscript-feature associations. For DeepSeek-V3.2, Area Chair guidance reduces mean rating divergence from 1.460 to 0.199 points and mean absolute error from 1.577 to 0.942, while rank correlation changes from 0.377 to 0.387. Historical references combined with downward scoring instructions yield mean divergence of 0.052 and rank correlation of 0.321. Domain differences remain after global calibration. Longer titles have positive adjusted associations with rating divergence, and formula density has negative associations across all four configurations. These findings connect score interpretation to the scoring procedure, evaluation context, and manuscript information represented in the comparison. They show why aggregate alignment, manuscript ordering, and correspondence within manuscript groups require distinct evidence when evaluating LLM-generated scholarly assessment information.

#DeepSeek-V3.2 #ICLR #LLM评分 #同行评审 #提示词工程

Keep reading

Recent stories with the same tag or section

Papers Builders10:24

Chinese contentTCR-Former：用因果拓扑注意力让 8.6B 模型时序因果推理超越 GPT-5.4

一篇时序因果推理的论文，8.6B 参数的 TCR-Former 在 TempoCausal 上赢了 GPT-5.4 和 DeepSeek-V3.2，思路是把因果图塞进注意力机制里。

#TCR-Former #TempoCausal #GPT-5.4 #DeepSeek-V3.2

arXiv: DeepSeek@Zhifeng Yang et al. (4)Source

Papers BuildersMulti-source09:36

Chinese contentAI 审批监控器会出现相关性失效，论文建议按证据源计数而非投票

这篇论文测了 16 个模型当支付审批监控器的表现，发现一起失效的根源是共享证据源，结论是数来源别数票，做智能体安全的人可以看看。

#Claude #GPT-6 #DeepSeek-V3.2 #AI安全

Event

arXiv: DeepSeek@Luca Zanolini11 sourcesSource

Papers Builders09:27

Chinese contentIF-MAP：用影响函数思路给多智能体系统排序优化优先级

多智能体系统预算有限时，这篇论文教你怎么算出该先优化哪个 agent，六个工作流实测收益翻倍。

#IF-MAP #多智能体 #Qwen3-8B #DeepSeek-V3.2

arXiv: DeepSeek@Jie Mao et al. (9)Source

Papers BuildersSelected09:29

LatentIndex cuts sparse-attention indexer cache by 61.1% on DeepSeek-V3.2

Sharing continuous latent representations instead of discrete token selections resolves the usual trade-off between cache reuse and per-layer accuracy in sparse attention. Engineering teams running long-context inference on DeepSeek-V3.2 or GLM-5 can apply the training-free calibration directly to cut indexer memory and speed up decoding. The 2.3-2.7 times decode speedups at 128K contexts make it especially relevant for local long-document and agent workloads.

#LatentIndex #DeepSeek-V3.2 #GLM-5 #sparse attention

arXiv: DeepSeek@Zhaohui Wang et al. (4)Source

Models BuildersMulti-sourceSelected10:45

SPLASH system switches attention layouts

SPLASH eliminates the need to restart workers when changing attention layouts, enabling dynamic adaptation to varying workloads. Developers can implement this to optimize LLM serving efficiency across different hardware configurations and model sizes, with particular benefits for handling mixed workloads that transition between short and long requests.

#SPLASH #GLM-5.3 #DeepSeek-V3.2 #attention parallelism

Event

arXiv: DeepSeek@Chuan Liu et al. (9)9 sourcesSource

Papers BuildersSelected18:34

ScopeIF Framework for LLM Instruction Following

ScopeIF addresses a critical gap in LLM training by providing dense supervision for complex constraints that existing binary reward models miss. This approach significantly improves performance on multi-faceted instructions without compromising general capabilities. Developers working with complex task automation can implement this framework to enhance their models' precision, particularly when dealing with segmented constraints across different response parts.

#ScopeIF #Qwen3 #Gemini-2.5-Pro #DeepSeek-V3.2

arXiv: DeepSeek@Bosi Wen et al. (6)Source