Tri Dao on fixing SSM inference with better caching
The takeaway is that SSMs can pick up the same deployment optimizations as Transformers without changing the architecture, which weakens the usual cost argument for switching model families. Practitioners evaluating hybrid or SSM-based serving stacks should watch for IO-aware caching implementations and benchmark them against paged KV setups. Dao's framing also connects techniques across FlashAttention, Mamba and paged KV as one design principle.
Tri Dao argued that state space models lose out on inference optimizations that Transformers get by default, such as speculative decoding and quantization, because the wrong thing is being cached. He said the fix is an IO-aware caching approach in the same spirit as FlashAttention, the Mamba scan and paged KV caches. In his view, many large inference gains came not from new architectures but from treating memory IO as a first-class constraint.
SSMs had the nice asymptotics but kept missing out on the inference goodies Transformers get for free like spec decode and quantization. Turns out we were just not caching the right thing. Same IO-aware trick behind FlashAttention, Mamba scan, paged KV. A lot of the big wins weren’t new architectures, just people taking memory IO seriously.
