Target Prop algorithm discussed by Yann LeCun
For those interested in neural network optimization, this is a discussion of Target Prop, a backpropagation variant that could be more efficient in certain contexts like sparse auto-encoders.
Yann LeCun tweets about Target Prop, a backpropagation variant where each layer's input is a free latent variable serving as a target for the previous layer.
He notes it can be derived from an augmented Lagrangian formulation of backprop and that his lab explored it in sparse auto-encoders in the late 2000s.
He says it optimizes the same criterion as backprop but evaluates gradients differently, perhaps more biologically plausible, and can be more efficient when auto-encoder code is regularized.
Nice! This ends up being a version of what some of us have called "target prop": every layer's input is a free latent variable that serves as a target for the previous layer. As this paper points out, this can be derived from an "augmented Lagrangian" formulation of backprop in which the constraints (input of layer k+1 = output of layer k) are turned into penalties (divergence between input of layer k+1 and output of layer k). I've always hoped more people would pick up on this idea. I'm happy this is happening! I must say though that target prop, in the end, optimizes the same criterion as backprop and does the same thing as backprop while evaluating the gradient in a different way, perhaps more biologically plausible. My lab did some work on this idea in the context of "sparse auto-encoders" in the late 2000s. It turns out when the code in an auto-encoder is regularized (e.g. with L1 to make it sparse) target prop seems more efficient than backprop. scholar.google.com/citations?view…
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include target prop, backpropagation, neural networks, Yann LeCun.
Source attribution: curated from Yann LeCun (@ylecun).