CaliBench 新基准专测视频模型的物理随机效果靠不靠谱，六个模型里 Veo 3.1 在骰子上翻车。想对比可以参考。
CaliBench 提出用物理可解释的离散空间（如骰子点数、纸牌花色、轮盘颜色）来评估视频世界模型，而不是像 FID 那样的学习特征空间。该基准覆盖九种场景和六个模型（WAN-2.7、SeeDance-2.0、HappyHorse-1.0、Veo 3.1、Runway Gen-4.5、Cosmos3-Super），每个模型生成32次。测试发现模型普遍把概率质量集中在少数结果上，而非复现参考分布；多数场景-模型组合显著失准，Veo 3.1 在骰子场景甚至坍缩到单一结果。在轮盘场景中，生成视频常让球位置含糊不清，导致多个模型的可评分性（scorability）很低。研究者将协议和指标 mnTV（平均归一化总变差）开源，供新模型对比。
CaliBench: Are the Stochastic Dynamics of Video World Models Physically Calibrated?
Video world models approximate the stochastic distribution of physical outcomes through generative sampling, but existing benchmarks score individual generations or compare distributions coarsely over a whole dataset, leaving the fine-grained aleatoric uncertainty of specific phenomena untested. We introduce CaliBench, which scores outcomes in a physically interpretable discrete space - a bin index, a die face, a suit, a colour - rather than a learned feature space such as in FID, so the distance from a known reference distribution is measured directly. We curate outcome spaces whose reference is known in closed form (binomial Galton boards, Bernoulli forks, uniform dice/cards/lottery, a skewed European-roulette colour), enabling an exact calibration test. We decompose performance into two orthogonal axes that a single accuracy metric conflates: scorability, the fraction of generations yielding a scoreable outcome, and calibration, the total variation distance from the reference on that sample. A chi-squared test assesses significance; as calibration is its null hypothesis it can evidence only miscalibration, and at N=32 per cell detects only large deviations. We apply it to nine scenes and six image-to-video models (WAN-2.7, SeeDance-2.0, HappyHorse-1.0, Veo 3.1, Runway Gen-4.5, Cosmos3-Super), 32 generations each. Models consistently concentrate probability mass on a few outcomes rather than reproducing the reference. Most scene-model combinations are significantly miscalibrated, in the extreme collapsing to one outcome, as Veo 3.1 does on dice. On roulette, generations often leave the ball ambiguously placed, giving several models low scorability. Performance varies by scene: no model dominates all nine. We release the protocol and a metric (mean normalised total variation, mnTV) for comparing new models against our results.