UCR 时间序列分类基准遭审计：多个数据集任务与原始用途不符
Fifty-Two Shades of Pig: What the UCR Archive Really Asks Classifiers to Do
有人把 UCR 这个 TSC 标准基准逐个数据集扒了一遍，发现不少任务的原始用途被改掉了。跑基准刷分前建议先看这份审计。
一篇 arXiv 论文对 UCR Time Series Classification Archive 逐一溯源审计。其中 PigAirwayPressure 数据集原本用于检测猪失血前的气道压力变化，基准里却被改成识别 52 头猪个体。研究发现多个数据集被改变原始任务、来自玩具设置，或测试集划分存在缺陷。论文还质疑谱线、轮廓等有序非时间序列对真实 TSC 问题的代表性，审计代码已开源在 GitHub。
Fifty-Two Shades of Pig: What the UCR Archive Really Asks Classifiers to Do
The pigs were bleeding. That was the point of the experiment: could airway pressure reveal hemorrhage before it became critical? In the UCR Time Series Classification Archive, the de facto benchmark for Time Series Classification (TSC), the same recordings ask a different question: "which of 52 pigs produced this trace?". The dataset PigAirwayPressure is not an outlier. Although the archive is the standard benchmark, whether its tasks resemble the problems TSC is meant to solve has rarely been examined. We find that several datasets are repurposed from their original task, built from toy setups, or have flawed test splits. We further question how representative the archive's ordered non-temporal sequences (e.g. spectra, shape outlines) are of real TSC, and the potential downsides from the decomposition of multivariate problems. Our findings call for more careful interpretation of archive-wide scores and greater attention to deployment-relevant evaluation. The audit is at https://github.com/automl/ucr-provenance-audit.