Groq plans DRAM-stacked inference chip to fix SRAM capacity
The bonding method determines the outcome: 3DS adds capacity at DRAM speeds, while hybrid bonding delivers bandwidth much closer to SRAM, deciding how few chips are needed to serve large models fast. Inference infrastructure teams evaluating Groq-class LPUs versus Nvidia GPUs, and anyone tracking Cerebras, which plans DRAM stacking on its CS-6 wafer, should watch which route ships first. Capacity, not bandwidth, is emerging as the binding constraint across SRAM-first architectures.
Groq plans to stack DRAM on its next inference chip to overcome the small capacity of on-chip SRAM, according to industry sources. Today's Groq 3 LPU reads 500MB of on-chip SRAM at 150TB/s, roughly 7 times a Rubin GPU's HBM bandwidth, but a 256-chip rack holds only 128GB, less than one Rubin GPU's 288GB. Nvidia paid US$20B last December to license Groq's technology and hire its leaders, making the chip effectively Nvidia's next LPU. The reported design, known as 3DS, wires DRAM dies vertically through silicon at ordinary DRAM speeds, while earlier reporting described SK Hynix DRAM hybrid-bonded onto the logic die, promising roughly 10 times HBM's bandwidth density with 77% to 83% less energy per bit.
Groq plans to stack DRAM in its next inference chip to get around SRAM’s small capacity, industry sources say.
Nvidia paid $20B last December to license Groq’s technology and hire its leaders, so this chip is effectively Nvidia’s next LPU.
The big deal here is that, each generated token requires reading the model’s weights from memory, so bandwidth sets speed.
The big deal here is that each generated token requires reading the model’s weights from memory, so bandwidth sets speed.
Today’s Groq 3 LPU reads its 500MB of on-chip SRAM at 150TB/s, about 7 times a Rubin GPU’s HBM bandwidth, which is fast enough for thousands of tokens per second.
That memory is so small, though, that a full 256-chip rack holds 128GB, less than one Rubin GPU’s 288GB, and an 8-bit trillion-parameter model needs roughly 2,000 chips just for its weights.
Stacking DRAM on the logic die can put far more memory beside the compute, and hybrid-bonded versions promise roughly 10 times HBM’s bandwidth density with 77% to 83% less energy per bit.
That would let fast inference on large models run on far fewer chips.
The reported design, known as 3DS, is more modest: it wires DRAM dies vertically through the silicon and keeps a standard DRAM interface, adding capacity mainly at ordinary DRAM speeds.
Groq hasn’t announced it, and earlier supply-chain reporting described another route: SK Hynix DRAM fused directly onto the logic die by hybrid bonding, alongside SRAM.
Cerebras plans to stack DRAM on its CS-6 wafer too, a sign the capacity ceiling is binding across SRAM-first designs.
So the bonding method decides the outcome: 3DS mostly adds capacity at DRAM speeds, while hybrid bonding promises bandwidth much closer to SRAM.
