DeepSeek-V4-Flash on AMD GPUs: Correctness and Performance Engineering
DeepSeek-V4-Flash on AMD gfx90a: Correctness Recovery and Inference Performance Engineering
For users interested in optimizing large language model inference on AMD hardware, this paper provides detailed engineering insights and performance benchmarks.
This research details the engineering of DeepSeek-V4-Flash inference on AMD Instinct MI250 GPUs. It addresses a correctness issue with a fast execution path caused by a weight layout mismatch and implements fixes like output permutation and token checks. Performance improvements were achieved through techniques such as FP4 routed mixture-of-experts, INT8 activation quantization, and CDNA2-specific kernels, resulting in 74.5 tokens per second for decode and a 4,604-token prompt taking 2.061-2.062 seconds to process.
DeepSeek-V4-Flash on AMD gfx90a: Correctness Recovery and Inference Performance Engineering
We present the enablement, correctness recovery, and performance engineering of DeepSeek-V4-Flash inference on AMD Instinct MI250 GPUs using the gfx90a/CDNA2 architecture. The system integrates native safetensors loading, tensor and expert parallelism, FP4 routed mixture-of-experts computation, FP8 dense projections, sparse attention, HIP graph execution, and OpenAI-compatible serving within SGLang. An initially fast execution path was found to be numerically incorrect because of a routed-expert W2 layout mismatch. We identify the output permutation, repair the weight layout at load time, and establish fixed-token and hash-based correctness checks before further optimization. On the corrected path, decode performance is improved through packed FP4 weights, INT8 activation quantization, CDNA2 dot-product instructions, peer-read all-reduce, and topology-aware kernel geometry. Prefill is accelerated using CDNA2 MFMA kernels, improved packed-weight reuse, reduced sparse-attention overhead, larger chunks, and retuned expert sorting. On four MI250 GCDs, TP4/EP1 native autoregressive decode reaches approximately 74.5 tok/s, while a 4,604-token prompt reaches 2.061-2.062 s TTFT, or approximately 2,234 input tok/s. The results show that efficient DeepSeek-V4-Flash inference on CDNA2 is limited not only by memory bandwidth, but also by FP4 execution-format mismatch, low-M utilization, and per-layer synchronization costs.
