想省训练成本又想模型推理更强？PPL-Factory用10%数据在MATH上比全量微调高4.8，门槛低、效果还更炸。
PPL-Factory提出了基于困惑度（perplexity）的任务感知和预算感知数据选择框架，用于大语言模型微调。在GSM8K上仅用1%训练数据即超越现有最佳方法；用10%数据在GSM8K上超过全量微调0.9个点，在MATH上超出4.8个点。该方法通过区分语言建模与推理任务的不同学习目标，实现高效且可解释的数据筛选。
PPL-Factory: Task-Aware and Budget-Aware Data Selection from Language Modeling to Reasoning
Not all training samples contribute equally to large language model fine-tuning. Selecting informative training samples can reduce the computational cost while preserving downstream performance. Many existing data selection methods rely on indirect heuristics, such as data quality, diversity or reasoning trace length. However, the effectiveness of these fixed criteria is task-dependent and difficult to generalize across diverse downstream tasks. Perplexity-based data selection provides a simple and model-aware solution to estimate the sample difficulty, but existing approaches typically score the entire training sequence and ignore the difference in learning objectives of language modeling and reasoning tasks. In this paper, we propose PPL-Factory, a simple and interpretable data selection framework that combines task-aware perplexity-based scores and data budget-aware selection criteria. Experiments on GSM8K demonstrate that PPL-Factory outperforms other state-of-the-art data selection methods using only $1\%$ of the training set. With $10\%$ of the data, PPL-Factory exceeds full-data fine-tuning accuracy by 0.9 on GSM8K and 4.8 on MATH. Overall, our results demonstrate that task-aware and budget-aware perplexity-based selection provides an effective and applicable approach for efficient fine-tuning.