Sebastian Raschka:评测框架本身就是模型训练的一部分
Raschka 这条推解释了一个常见坑:同一模型换评测框架分数就变。做模型选型或复现论文的人看完会明白为什么要先看作者的测试工具链。
Sebastian Raschka 在 X 上指出,LLM 在训练阶段就针对特定 harness(评测/运行框架)和工具做了微调,harness 已内嵌于训练流程,模型无法将其“吸收”成通用能力。他用 CPU/GPU 与操作系统的关系作类比:框架和模型是配套设计的整体,而非可分离的两层。
这意味着跑分时换一套 harness,成绩可能就不可比;开发者复现论文结果时,忽略作者所用的具体工具链往往会导致性能落差。选择与目标场景匹配的 harness,和选模型本身同等重要。
Yeah. Also, LLMs are specifically fine-tuned to work in specific harnesses and with specific tools.
I.e., the harness is part of the training setup, so the model can't really absorb it.
It’s a bit like the CPU/GPU absorbing the operating system.
This report is categorized under AITOP's "How-to" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include LLM, benchmarks, fine-tuning, model selection.
Source attribution: curated from Sebastian Raschka (@rasbt).