智谱GLM-5.3后训练提升编码能力

智谱联合创始人唐杰老师关于 GLM 5.3 以及模型训练的一些分享： 智谱 GLM-5.3 没有换基模底座，纯靠后训练让编码能力提升 50%。 GLM-5.3 模型的底座是 GLM-5.2，约 7...

智谱联合创始人唐杰老师关于 GLM 5.3 以及模型训练的一些分享： 智谱 GLM-5.3 没有换基模底座，纯靠后训练让编码能力提升 50%。 GLM-5.3 模型的底座是 GLM-5.2，约 743B 参数的混合专家（MoE）模型，每次推理只激活约 40B 参数。团队花了一个月，在长周期环境中做强化学习，编码能力比 GLM-5.2 提升了 50%。 这里解释一下 MoE：传统模型每次推理要跑完所有参数，MoE 相当于把模型拆成一群“专家”，每次只调用其中几个，推理速度更快、成本更低。 推文中特别区分了两个概念： 1. 总参数量决定模型学习了多少知识 2. 激活参数量和有效深度决定模型能想多深。 比如说，让模型去找安全漏洞，主要是依赖的是推理能力，能把一条二十步的推理链完整走到底，相对来说各种安全数据库是次要的。 至于为什么不换底座也能大幅提升呢？ AI 模型的扩展（scaling）不止堆参数这一条途径。 如果梳理下这些年模型训练的发展路线： - 2020 年 Kaplan 等人的研究建议参数增长要远快于数据，GPT-3、Gopher 都是这个思路的产物。 - 2022 年 DeepMind 的 Chinchilla 论文则认为参数和数据应该同步增长，参数大的模型反而是最浪费算力的。 - 再后来，大家发现模型上线后推理成本远超训练成本，最优解又变成了用更小的模型训更久，比如 Llama-2-7B 每个参数喂了约 290 个 token，Gemma-2-9B 更是喂了 889 个。 模型能力由很多因素决定：底座大小、预训练数据量、每次前向传播的计算量、后训练。 对于现阶段，模型后训练还有很大潜力可以挖掘。 jietang @jietang Thoughts About Scaling Law Scaling, but not only of parameters. Every model release now ends with the same question: how many parameters? It isn't a question that can be answered on its own. Parameter count is only meaningful alongside three others — how much data you have, where you intend to spend your compute, and who will run the model, under what conditions. The field learned this the hard way. Kaplan et al. (2020) fit an exponent that told everyone to grow parameters faster than data — roughly 2.7:1 — and the industry complied: GPT-3, Gopher, MT-NLG. Hoffmann et al. (2022) redid the experiment across four hundred models and found the compute-optimal split is closer to 20 tokens per parameter, and that with sufficient compute the two should grow at the same rate rather than drifting apart. The error in the earlier fit compounded with every order of magnitude of compute, which is why the largest models of that generation were the most misallocated. The trillion-parameter round was, in retrospect, a detour the whole field took together and then reversed. Chinchilla wasn't the end either. It optimized training compute for models that would be trained once and evaluated. Today a model is called billions of times a day and inference dominates lifetime cost. Put inference into the objective and the optimum moves toward smaller models trained far longer — deliberate over-training, which is what Llama-2-7B and Gemma-2-9B were doing at roughly 290 and 889 tokens per parameter. Sparsity moved the target again. In a MoE model two quantities have to be kept apart: total parameters govern roughly how much the model can hold — knowledge, facts, the long tail — while activated parameters and effective depth govern roughly how far it can think, how many steps of a causal chain it can carry before it comes apart. A dense 20:1 ratio does not transfer. And the ratio isn't a single number at all: Roberts et al. (2025) find the optimal tokens-per-parameter is task-dependent, with memorization favoring more parameters and reasoning favoring more data. Follow-up work on MoE observes that at fixed TPP, pushing total parameters higher actually degrades reasoning, while activating more experts reliably helps it. This matters for what we are building toward. Finding a vulnerability is not a retrieval problem. It doesn't come from having memorized more CVEs; it comes from carrying a twenty-step chain of inference to the end without losing the thread. That capability does not live in total parameter count. Which brings us to this release. Total parameters appear to matter up to a threshold — enough to hold the world — after which additional capability comes from scaling elsewhere: effective depth per forward pass, and above all post-training. GLM-5.3 is our controlled experiment on that claim. Same base, same architecture, same total and activated parameters as GLM-5.2. One month of scaling long-horizon environments and RL. The gains are not marginal. Well, scaling has more than one dial. We turned the post-training one this time because it had the most slack left in it — not because the others are finished. Base model size, pretraining data, compute spent per forward pass: all of them are still on the table, and we will come back to each. What this experiment taught us is that the dials do not have to be turned together, and that the one worth turning next is rarely the one that was worth turning last. We are not done scaling. Next time, maybe mid-training, pre-training, and even more. 🔗 View Quoted Tweet 💬 0 🔄 0 ❤️ 4 👀 1374 📊 1 ⚡