用形式推导预训练，能在语言任务上少花36B token达标，剪枝33%还不掉点。做预训练或模型压缩的可以看看。
Logic-PPT 用形式推导作为预训练前置任务，为语言模型注入变量绑定、量词关联等结构偏置。在 100B token 规模下，该方法在语言任务上达到 80% 准确率，比标准初始化少用 36B token。相比 Dyck 和程序算法等基线，形式推导带来更持久的表征重组。其表征空间呈现更低秩、谱集中分布，在约 33% 稀疏度剪枝下仍能匹配稠密模型性能。
Logic Before Language: Pre-pretraining on Formal Derivations Fosters Skill Acquisition and Compressibility
Pre-pretraining language models (LMs) on symbolic data can accelerate and improve natural language acquisition. However, existing pre-pretraining tasks, such as Dyck and procedural algorithms, rely on narrow primitives that fail to capture the expressive capacity of natural language. Moreover, prior studies remain restricted to relatively small token budgets, offering limited insight into skill emergence and representational dynamics. To address these limitations, we propose logic pre-pretraining (Logic-PPT) as a principled initialization strategy, leveraging formal derivations to impart richer structural and linguistic biases. Formal derivations require abstract mechanisms that are central to natural language, simultaneously binding variables, connecting quantifiers and relational dependencies, and composing predicate-argument structures over long contexts. Scaling our evaluation to a 100B-token regime, logic pre-pretraining substantially accelerates skill acquisition in LMs, achieving 80\% accuracy on linguistic tasks with 36B fewer tokens than standard initialization, and outperforming alternative pre-pretraining baselines. Mechanistically, formal derivations induce persistent structural reorganization, distinctively characterized by a lower-rank, spectrally concentrated representation space. Crucially, we show that this internal geometry enables improved model compressibility via pruning, matching the dense baseline performance even at $\approx$33\% sparsity.