数据组织是LLM训练中常被忽视的杠杆，STR和SAW方法几乎零成本就能提升训练效果，做预训练或SFT的团队值得一试。
该论文系统研究了数据组织对LLM训练的影响，提出四个关键准则：边界锐化、循环调度、课程连续性和局部多样性。基于这些准则，作者设计了两种新的数据排序方法STR和SAW，通过复用预计算的样本级分数，几乎不增加额外计算开销。实验表明，这些方法在预训练和SFT阶段均能提升训练稳定性和模型性能。对于追求训练效率的AI团队，这是一个低成本的优化方向。
Demystifying Data Organization for Enhanced LLM Training
Large Language Models (LLMs) have revolutionized various fields, yet their training efficiency is heavily reliant on effective data curation. While data selection has been widely studied, the strategic data organization for enhanced training remains an underexplored area, particularly since current LLMs are often trained for only one or a few epochs. This paper systematically explores the influence of data organization on LLM training by reusing pre-computed sample-level scores originally generated for data efficiency, thereby incurring minimal additional computational overhead. We identify and formalize four key guidelines for optimizing data organization: Boundary Sharpening, Cyclic Scheduling, Curriculum Continuity, and Local Diversity. Guided by them, we introduce two novel data ordering methods termed STR and SAW. Extensive experiments across different model scales and data sizes, encompassing both pre-training and SFT stages, validate the effectiveness of our summarized guidelines. They also demonstrate the robustness of our proposed data ordering methods in enhancing the stability and performance of LLM training. Github Link: https://github.com/microsoft/data-efficacy/