#Looped Transformer

5 stories · 1 in 7 days · 3 in 30 days

开源模型 3 推理模型 2 Foil 1 KV cache 1 KV Cache 1 LoopAhead 1 LoopVQ 1 MoE 1

Stories, products and papers tagged "Looped Transformer", newest first.

Nov 11

Models BuildersMulti-source02:14

Chinese contentNVIDIA Nemotron Labs 专家直播：详解 Dory Looped Transformer 模型

NVIDIA 的 Nemotron Labs 开了一场专家直播，专门讲 Dory 这个 Looped Transformer 模型的架构设计。想搞清楚循环式 Transformer 跟标准堆叠式架构差在哪、参数量怎么省的朋友，可以直接看回放。做模型结构研究或推理优化的人会有收获。

#Nemotron #NVIDIA #Looped Transformer #推理模型

Official accountNVIDIA AI@NVIDIAAISource

Nov 4

Papers Builders13:10

Chinese contentLoopVQ：面向 Looped Transformer 的亚 1-bit KV Cache 量化方法

.looped 模型参数省但 KV cache 爆炸，这篇提出 0.5bit 量化方案，SGLang 实测吞吐最高涨 6.1 倍，做部署的可以看看。

#LoopVQ #KV Cache #量化 #Looped Transformer

arXiv cs.LG@Minsoo Cheong, Sungjoo YooSource

Oct 16

Papers Builders09:40

Chinese contentLoopAhead：面向 Looped Transformer 的推测解码框架，加速最高 7.82 倍

用循环层的早期状态提前起草 token，解码最高快 7.82 倍，输出分布不变，研究解码优化的可以看看

#LoopAhead #Looped Transformer #推测解码 #推理加速

arXiv cs.LG@Zixuan Liu et al. (7)Source

Oct 8

Papers BuildersSelected02:15

Nanbeige 4.2 Looped Transformer Breakdown Compares Weight Reuse

The three reuse tricks all called 'looped transformers' differ in cache behaviour and cost, and picking the wrong one can burn a training week or budget a 3B decode like a 3B file. Builders planning efficient architectures can use this to decide whether extra loops justify their compute and whether a second cache is needed. Nanbeige 4.2 offers a concrete open reference point against standard Transformers.

#Nanbeige #looped transformer #open-source models #KV cache

Official accountAlphaSignal@AlphaSignalAISource

Sep 29

Papers Builders13:16

Chinese contentFoil 方法提出循环 MoE 训练新设计：扁平化专家并解绑注意力

一篇教你怎么把 MoE 和循环 Transformer 结合的论文，Foil 把专家层数减半、循环翻倍，同等算力下预训练损失更低，代码也开源了。

#MoE #Looped Transformer #Foil #开源模型

arXiv cs.LG@Shouren Wang et al. (9)Source