这篇论文戳破了时序模型评估的常见坑，用具体数字说明评估协议能颠倒结论，搞时序生成的人都该看看。
该研究指出，许多生成式时序模型基准数据集中，单一值（如零降雨、零订单）占据大量概率质量。标准滚动起点协议可能导致评估窗口与数据集结构严重不符，例如某基准数据集中42%为零，而评估窗口仅13%。这种不匹配甚至逆转了研究结论，使最强模型看似失败。研究者提出一种CRPS不变的对照方法，以衡量时间耦合对统计量的贡献。在五个种子上对七种模型进行匹配协议测试，自回归hurdle模型在六个数据集中的五个上优于条件流模型，最高提升153倍，而流的出现统计量在不同种子间变化高达62%。五种不同出现统计量下模型排序不一致，且构造不同的两种统计量一致性最差。
Evaluating Generative Time-Series Models on Data with Point Masses
Many of the series that generative time-series models are benchmarked on place a large probability mass on a single value --- it does not rain, no ride is requested, no part is ordered. We report what happens when such data is evaluated carefully. First, the standard rolling-origin protocol can score a model on a window whose atom structure bears no resemblance to the dataset: on one benchmark the dataset is $42\%$ zeros and the evaluation windows are $13\%$, on another $47\%$ against $5\%$. This is not a cosmetic problem --- it reversed one of our own conclusions, turning the strongest occurrence model in our study into what looked like a cautionary tale. Second, we give a control in which CRPS is invariant \emph{by construction} while the temporal coupling is destroyed, which measures exactly how much that coupling contributes to a chosen statistic. Third, benchmarking seven models on a matched protocol over five seeds, an autoregressive hurdle beats a conditional flow on five of six datasets, by up to a factor of $153$, while the flow's own occurrence statistics vary by up to $62\%$ across training seeds and every baseline is deterministic. Finally, the model ordering is not the same under five different occurrence statistics, and the two that do not share a construction agree with each other least.