D2K-Bench 基准测试：LLM Agent 能否把专家设计转化为高效 GPU 内核
D2K-Bench: Can LLM Agents Turn Expert Designs into Efficient GPU Kernels?
arXiv 上的新基准 D2K-Bench，26 个任务测 LLM 写 GPU 内核，加上专家指导后 GPT-6-Astra 等模型的加速比从 1.69 倍冲到 2.49 倍，结果挺有意思。
D2K-Bench 是一个包含 26 个任务和 85 个工作负载的诊断型基准，用于衡量 LLM Agent 在专家设计指导（L1 算法洞察、L2 数据流设计、L3 底层优化技巧）下生成 GPU 内核的效率。在 NVIDIA B200 GPU 上测试五个模型，专家指导使 130 个模型-任务对的正确率从 93.1% 提升到 98.5%，26 个任务的 Performance Score 从 1.46 升至 1.95。GPT-6-Astra、Claude-Opus-4.8 和 GPT-5.6-Sol 三个前沿模型的几何平均加速比从 1.69 倍提高到 2.49 倍。五个模型的平均综合实现得分从 57 分提升到 70 分（满分 100）。
D2K-Bench: Can LLM Agents Turn Expert Designs into Efficient GPU Kernels?
GPU kernels generated by large language model (LLM) agents can remain less efficient than expert implementations, but runtime alone does not reveal how the gap relates to design discovery and implementation. We introduce D2K-Bench, a diagnostic benchmark of 26 tasks and 85 workloads that measures how effectively agents translate expert design guidance into efficient GPU kernels. The guidance covers L1: high-level algorithmic insights, L2: dataflow design, and L3: low-level optimization tricks, including dependencies among these levels. Pairwise runs with and without guidance share task descriptions, workloads, tools, hardware, and a 350-turn budget. Complementary assessments examine independently proposed designs and the design properties implemented in generated code. Across five models on NVIDIA B200 GPUs, guidance raises correctness over 130 model-task pairs from 93.1% to 98.5% and increases the Performance Score over all 26 tasks from 1.46 to 1.95. For the three frontier models with correct submissions on all 26 tasks in both runs (GPT-6-Astra, Claude-Opus-4.8, and GPT-5.6-Sol), geometric mean speedup increases from $1.69\times$ to $2.49\times$. Across all five models, the mean combined implementation score increases from 57 to 70 out of 100. These results show the value of expert design guidance while identifying design properties that remain unimplemented.
