论文精选
Telescopic Language Models 发布
精选理由
新发布的 TLM 模型通过独特的前缀监督方法，实现了在任意深度都保持有效语言模型的能力。
Telescopic Language Model (TLM) 是一种嵌套能力 Transformer 模型，采用随机前缀监督训练。该模型在每一步训练中，随机截断能力轴的一个前缀，同时进行完整能力前向传播，确保在每个深度都成为有效的语言模型。
原文 · Tanishq Abraham (论文推介)
Telescopic Language Models
"We train a Telescopic Language Model (TLM) to be that continuum: a nested-capacity Transformer supervised by stochastic prefix supervision with a full anchor. At every step, one randomly truncated prefix of the capacity axis is trained against the full next-token target, alongside one full-capacity pass, so the trained artifact is a valid language model at every depth."
link: https://t.co/OnTEJpi5cM
