PLES让大模型训练调参效率提升10倍，用小规模实验预测大规模最优配置。
研究人员提出Power-Law Entropy Search (PLES)方法，用于高效估计大语言模型训练的最佳超参数缩放定律。PLES基于多保真度贝叶斯优化，通过自适应实验减少计算资源消耗。在合成基准、真实LLM训练数据和实际预训练任务中，PLES仅需传统网格搜索不到十分之一的计算量，就能收敛到准确的超参数缩放定律。
Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search
Optimal hyperparameter scaling laws describe how the best hyperparameters for large language model (LLM) training change with model and data scale, enabling practitioners to predict optimal configurations at production scales without expensive large-scale tuning. However, estimating these scaling laws conventionally requires exhaustive grid searches over thousands of training runs, consuming enormous computational resources. We introduce Power-Law Entropy Search (PLES), a computational cost-aware acquisition function built on multi-fidelity Bayesian optimization that efficiently estimates optimal hyperparameter scaling laws through adaptive experimentation. A key innovation in PLES is that it searches for candidates that reduce the overall uncertainty of a scaling law estimate, instead of optimizing a single objective function. At each iteration, PLES selects the candidate configuration that maximally reduces the uncertainty of the scaling law estimates per unit computational cost, naturally favoring informative small-scale experiments. We evaluate PLES on synthetic benchmarks, surrogate models fitted to real LLM training data, and actual LLM pre-training runs. Across all settings, PLES converges to accurate optimal hyperparameter scaling laws using less than one-tenth of the computational budget required by conventional grid search and other baselines.