arXiv 论文提出测试世界模型泛化能力的方法:以零售定价为实验场景
Testing World Models on Actions They Have Never Seen: Held-Out Plans, Forecast Accuracy and Price Rules in Retail
做世界模型评估的人应该看看这篇:2,853 个模型跑下来,证明平时看的预测精度在比较同一模型不同版本时可能完全失灵,甚至会反转到 -0.69。更有用的是它发现注入几条定价规则比 JEPA 预训练管用,零售和制药场景的实验设计可以直接套用。
arXiv 论文 2611.22537 提出一套评估动作条件世界模型的方法:故意把训练时没见过的动作(hold-out plans)留给模型预测,再对照已知答案打分。实验在零售场景进行,动作即价格与促销计划,共拟合了 2,853 个模型。
核心发现有三个。第一,整体预测精度在比较不同类别模型时还算可靠(与未见过计划的误差秩相关为 0.32 到 0.66),但在比较同一世界模型的不同版本时会失灵:当商家把计划排进高需求周时,相关性反转到 -0.69。第二,用 JEPA 预训练、隐空间一致性检查和计划日适配等方式训练模型学习动作效果,对未见过计划上的误差没有改善;真正有效的是把常规价格记录和三条定价规则注入模型或简单模型。第三,在真实数据集(Breakfast at the Frat、The Complete Journey、M5)上,规则把世界模型在纯降价场景的误差从 0.63 降到 0.56,但对持久性价格变动无效。
论文还做了第二域压力测试:在生成的给药剂量世界里,累积血药浓度让误差从 2.86 降到 1.91,但模型仍只能预测真实增益的 0.12 到 0.32。作者认为这套测试可迁移到任何需要评估历史外动作的领域。
Testing World Models on Actions They Have Never Seen: Held-Out Plans, Forecast Accuracy and Price Rules in Retail
A world model is trusted to answer "what if": how its world responds to an action it has never seen. We give a test of that trust for any action-conditioned world model: actions held out of training by design and scored against a known answer; whether forecast accuracy predicts accuracy on those actions; and the same structure given to simpler models. We run it in retail, where the action is a price and promotion plan, across 2,853 fitted models. First, forecast accuracy is a fair guide between kinds of model (rank correlation with error on new plans 0.32 to 0.66) and a poor guide between versions of one world model: where merchants plan into high-demand weeks the link reverses, down to -0.69. Second, training the world model to learn the effect of actions (JEPA pretraining, a latent consistency check, adaptation at the plan date) leaves its error on new plans unchanged. Third, in generated worlds the recorded regular price and three price rules improve it. Given the same rules, a log-linear model is best when its curve can match the truth; in a world whose curve lies outside the rules, the world model with the rules is ahead of the log-linear model with the rules in 36 of 40 worlds, and does as well without its recurrent cell. On real data (Breakfast at the Frat, The Complete Journey and M5) the rules lower the world model's error on pure price cuts deeper than any it saw (0.63 to 0.56 on the Frat; 0.45 to 0.44, within chance, on The Complete Journey), but not on lasting price moves. Fourth, a small share of random test plans helps every model a little. In a generated dosing world, a stress test in a second domain, the first finding repeats, and the built-up drug level lowers the world model's error (2.86 to 1.91) though it still predicts only 0.12 to 0.32 of the true gain. The test carries to any domain where a world model is asked about actions outside its history.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include world models, JEPA, M5, time series.
Source attribution: curated from arXiv cs.LG (@Karthik Iyer).