Yann LeCun 的团队提出目标传播算法，优化反向传播效率
Nice! This ends up being a version of what some of us have called "target prop": every layer's input...
Yann LeCun 团队提出的这个算法，能优化反向传播的效率，在稀疏自编码器场景下表现更好。
Yann LeCun 团队提出目标传播算法，将每一层的输入视为自由变量，作为前一层的目标。该算法通过将层间约束转化为惩罚项（输入与输出的差异）来优化，与反向传播优化相同目标但梯度计算方式不同。在稀疏自编码器中，目标传播在 L1 正则化下比反向传播更高效。
Nice! This ends up being a version of what some of us have called "target prop": every layer's input...
Nice! This ends up being a version of what some of us have called "target prop": every layer's input is a free latent variable that serves as a target for the previous layer. As this paper points out, this can be derived from an "augmented Lagrangian" formulation of backprop in which the constraints (input of layer k+1 = output of layer k) are turned into penalties (divergence between input of layer k+1 and output of layer k). I've always hoped more people would pick up on this idea. I'm happy this is happening! I must say though that target prop, in the end, optimizes the same criterion as backprop and does the same thing as backprop while evaluating the gradient in a different way, perhaps more biologically plausible. My lab did some work on this idea in the context of "sparse auto-encoders" in the late 2000s. It turns out when the code in an auto-encoder is regularized (e.g. with L1 to make it sparse) target prop seems more efficient than backprop. scholar.google.com/citations?view… 💬 0 🔄 1 ❤️ 0 👀 133 ⚡
