这篇论文用M5竞赛数据测试了不同模型选择标准，发现短测试集时只有平均分靠谱。搞时间序列模型的人可以参考。
该论文探讨了在多个时间序列数据集上，使用适当评分规则（如平均分、中位分、平均排名）进行概率模型选择时出现的冲突问题。研究发现，这些统计量产生冲突决策的原因是评分分布的偏态性。随着每个时间序列测试集增大，不同模型选择标准会逐渐收敛到相同结论；但对于短测试集，只有平均分能正确识别真实模型。论文基于间歇性时间序列（包括M5竞赛数据集）进行了实验验证，强调了大规模测试集的重要性。
Model selection with proper scoring rules on data sets of time series
We consider the problem of model selection between probabilistic models on data sets of time series. Chosen a proper scoring rule, we denote by the term \textit{score} the average value of the scoring rule on the test of an individual time series. For model selection, we need aggregating the values of the scores across multiple time series. Three summary statistics are commonly used for model selection: mean score, median score, and mean rank. Results in previous papers show that these statistics can yield conflicting decisions; we show how the conflicting conclusions are due to the skewness of the distribution of scores. We also show that as the test set of each time series of the data set increases, the different model selection criteria progressively converge to the same conclusion. However, for short tests sets, only the mean score identifies the true model as the best. We illustrate these phenomena with an analysis on intermittent time series, including the data set of the M5 competition, where we underline the importance of having a large test set. In such experiments, we further notice that model selection based on mean ranks remains unchanged using different scaling factors.