Twindex Offloads Indexer Cache for Long-Context Sparse-Attention Serving
The non-obvious insight is that in sparse-attention serving the indexer cache, not the main KV, becomes the memory ceiling once KV offloading is in place — a shift most serving teams have not accounted for. Engineers building long-context inference infrastructure can adopt Twindex without retraining or modifying model weights, since it integrates with CUDA graphs and existing offloading stacks. Teams running DeepSeek, Qwen or GLM deployments at million-token contexts should benchmark the reported 1.25x–3.03x concurrency gains against their own batching setups.