SlackTier Moves DeepSeek-V4.1 Engram Tables Off Host DRAM
Placing LLM Lookup Tables by Per-Layer Prefetch Deadlines
The non-obvious insight is that per-layer prefetch slack, not raw bandwidth, decides whether lookup tables fit on NVMe, DRAM, or remote memory — static hot sets transfer poorly for agent traffic. Inference engineers running DeepSeek-V4.1 or similar n-gram-augmented models on SGLang can reclaim roughly 180 GiB of host memory and convert it into KV cache, directly lifting throughput. Compared with simply scaling up DRAM, the RDMA option lets one table copy serve two instances, cutting per-node memory cost.
A new paper on arXiv (2611.13413) presents SlackTier, a table tiering system for SGLang that drastically reduces host memory used by DeepSeek-V4.1's Engram n-gram lookup tables. Those tables total 203 GB and currently occupy 190 of the 223 GiB a 4-GPU serving process consumes. Because row indices depend only on input tokens, lookups can start at each decode step and overlap with earlier layers. V4.1's two Engram layers have prefetch windows differing by 30x (0.3 ms vs 9.3 ms), and the paper shows this deadline, session reuse, and shared-device contention determine which storage tier each table can use.
SlackTier issues all lookups at step start inside the decode CUDA graph, reading the short-window layer first and splitting a small GPU row cache across layers; it is bit-exact with the original gather. With both tables on four NVMe disks, mean time per output token on agent and chat replays rises at most 1.4% over unmodified SGLang while host memory drops from 212 to 34 GiB. Reading over RDMA from another node's memory removes the need for any cache, letting one copy serve two instances.
For KV-constrained agent workloads, repurposing the freed memory as host KV cache raises median throughput by 36% and cuts median p90 time to first token from 38 s to 24 s. The result implies large embedded lookup tables need not pin scarce host DRAM, opening room for longer contexts or higher instance density per node.
Placing LLM Lookup Tables by Per-Layer Prefetch Deadlines
DeepSeek-V4.1's 203 GB of Engram n-gram tables sit in host DRAM today, taking 190 of the 223 GiB a 4-GPU serving process uses. Row indices depend only on input tokens, so reads can start with each step and overlap earlier layers. V4.1's two Engram layers have windows that differ by 30x (0.3 vs. 9.3 ms). We show that this deadline, together with session reuse and contention on a shared device, decides which tier each table can use and which layers need cache, and we give a model that predicts the resulting slowdown from measured read times. Agent and chat traffic reuse rows within sessions, and static hot sets transfer poorly. We build SlackTier, a table tier for SGLang that issues all lookups at step start. It reads the short-window layer first and splits a small GPU row cache across layers. The path runs inside the decode CUDA graph and is bit-exact with the original gather. Layer 14's table can go to NVMe uncached with no measurable decode cost. With both tables on four NVMe disks, mean time per output token on agent and chat replays rises at most 1.4% over unmodified SGLang, and host memory falls from 212 to 34 GiB. Both tables need no cache when read over RDMA (remote direct memory access) from another node's memory; one copy serves two instances. On a KV-constrained agent workload, using the freed memory as host KV cache at the same memory budget raises median throughput by 36% and cuts median p90 time to first token from 38 to 24 s.
This report is categorized under AITOP's "Papers" desk, focusing on China's AI industry and domestic foundation model ecosystem. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include DeepSeek, SGLang, Engram, KV cache.
Source attribution: curated from arXiv cs.LG (@Wen Huang, Jack Liu).
