STAPLE Adds Sparse Embedding Modules to Pretrained Checkpoints
Sparse upcycling usually means expensive retraining from scratch, so a method that grafts new capacity onto an existing checkpoint could cut the cost of scaling older models. The nuance matters: this is a lightweight FFN modification, not a full Engram architecture, so results depend on how much quality the added token-indexed embeddings recover. Researchers and engineers fine-tuning open-source checkpoints should follow the paper for the training recipe.
A new method called STAPLE (Static Sparse Upcycling of Pretrained Checkpoints via Embedding Modules) shows that a pretrained model checkpoint can be Engram-augmented directly. Instead of rebuilding the model, it adds a token-indexed embedding to the up-projection inside the existing FFN. The approach is described as a cheaper and faster form of sparse upcycling. The clarification notes it is not a full Engram replacement but an added embedding module within the current architecture.
You can in fact just Engram-augment a pretrained checkpoint. STAPLE: Static Sparse Upcycling of Pretrained Checkpoints via Embedding Modules (well, fine. not quite. it adds a token-indexed embedding to the up-proj inside the existing FFN. cheaper, faster sparse upcycling) https://t.co/WyxK5TMqnF
