CuraEval:评估混合技能机器人数据中的示范筛选方法
CuraEval: Evaluating Demonstration Curation on Mixed-Skill Robot Data
机器人模仿学习的数据筛选方法被系统性检验了,PSD 分数被证明就是轨迹长度,还给了代码,做机器人训练的可以看看。
CuraEval 是一个用于机器人模仿学习中示范数据筛选方法的评估框架,基于 mixed-skill robomimic 数据,使用 Diffusion Policy 和 BC-RNN 进行对照实验。研究发现 PSD 分数实际上只取决于轨迹长度和位置幅值的平方,在 square 任务上与轨迹长度的排序相关性达 ρ=0.999。用 Diffusion Policy 时,选取最短的 50% 示范可将 square 任务成功率从随机筛选的 70.7% 提升到 76.5%。但几何熵 LOO 选择器表现更差:25% 保留率下 can 任务成功率从 94.7% 降到 74.0%。经 Holm 校正后,没有任何筛选方法显著优于全量数据训练。
CuraEval: Evaluating Demonstration Curation on Mixed-Skill Robot Data
Robot imitation learning uses demonstrations from operators of varying skill, but selecting training data remains challenging. Existing methods score demonstrations using smoothness, diversity, or estimated influence on policy performance, but differences in selection and evaluation protocols make it difficult to determine whether they outperform simple baselines. We introduce \textbf{CuraEval}, a framework combining mathematical analysis of demonstration scores with controlled comparisons against fixed size-matched random subsets and full-data training on mixed-skill robomimic data using Diffusion Policy and BC-RNN. We show that the evaluated PSD score depends only on episode length and squared position magnitudes, and produces nearly the same ranking as episode length on \textit{square} ($ρ=0.999$). With Diffusion Policy, selecting the shortest 50\% of demonstrations improves \textit{square} success from 70.7\% under random selection to 76.5\%. Compared with random selection, our adapted geometric-entropy LOO selector reduces success at 25\% retention from 94.7\% to 74.0\% on \textit{can} and achieves 41.7\% on \textit{square}, below the mean success rates of all three independently sampled random subsets (59.8--62.5\%). On the same 3 \textit{square} Diffusion Policy training seeds, length-based selection at 75\% retention improves success over full-data training from 76.7\% to 86.7\% under the original protocol, but lowers it from 80.5\% to 79.5\% after checkpoint reporting, rollout count, and evaluation seed change jointly. No evaluated curated condition significantly outperforms full-data training after Holm correction, although confidence intervals still permit useful gains. Code is available at https://github.com/jiuyixu25/CuraEval.