Sebastian Raschka:用基准测试对比 LLM 就像用 IQ 测试招人
Sebastian Raschka 说现在看 LLM 基准分数选模型,就跟用 IQ 测试招人一样不靠谱。做选型的人别光盯着榜单,用自己实际任务测一遍更实在。
知名机器学习研究者 Sebastian Raschka 在 X 上发文指出,用基准测试来对比新发布的 LLM,已经变得像用 IQ 测试来做招聘决策一样不可靠。
他认为基准测试只能作为基线参考,但无法说明某个模型对个人具体任务和需求的实际表现。文中还暗示了前一天某个模型发布所引发的基准刷分讨论,反映出社区对基准分数与真实使用效果脱节的担忧。
Comparing new LLM releases via benchmarks is now a bit like using IQ tests for hiring decisions.
Sure, benchmarks are useful baselines and so on, but beyond that, I don't know how indicative they are of our individual tasks and needs. (Yes, also looking at that release yesterday)
This report is categorized under AITOP's "Industry" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include LLM, benchmarks, model selection, Sebastian Raschka.
Source attribution: curated from Sebastian Raschka (@rasbt).