FlexNPU 解决了LLM推理中prefill和decode阶段资源冲突的痛点，做模型部署和推理优化的团队可以直接参考其动态调度思路，尤其适合使用华为Ascend NPU的开发者。
FlexNPU 提出了一种透明的用户空间虚拟化层，用于华为Ascend NPU，通过拦截AscendCL API并路由操作到设备守护进程，实现无需修改模型代码、AI框架或NPU驱动即可解耦应用与物理设备。该方案支持动态PD共置调度，根据prefill（计算密集）和decode（内存带宽受限）的互补资源特性灵活调整，解决了静态PD分离的资源不平衡和数据移动问题。在384卡Ascend 910C部署DeepSeek-R1时，相比静态PD分离，吞吐量提升5.15%和26.33%；在Qwen2.5-7B上，相比静态PD共置，TTFT降低超过92%且TPOT几乎不变。实验表明，透明NPU虚拟化是实现高效、响应式LLM服务的实用基础。
FlexNPU: Transparent NPU Virtualization for Dynamic LLM Prefill-Decode Co-location
Modern AI serving increasingly relies on NPUs for conventional inference and large language model serving. However, current NPU deployments commonly expose physical devices directly to applications, which limits runtime control over scheduling and makes it difficult to adapt execution to phase-level workload behavior. This limitation is particularly evident in LLM serving, where the prefill phase is compute-intensive while the decode phase is often constrained by memory bandwidth and KV-cache accesses. Static prefill-decode (PD) disaggregation reduces phase interference, but can introduce resource imbalance and unnecessary data movement. We present FlexNPU, a transparent user-space virtualization layer for Ascend NPUs. FlexNPU interposes on AscendCL APIs and routes NPU operations through per-device daemons, decoupling unmodified from physical NPU devices without modifying model code, AI frameworks, or NPU drivers. This runtime boundary allows FlexNPU to virtualize NPU objects, control operator dispatch, and support phase-aware scheduling for LLM serving. In particular, FlexNPU enables dynamic PD co-location, which adapts scheduling between prefill and decode according to their complementary resource characteristics. We implement FlexNPU on Huawei Ascend NPUs and evaluate it with typical LLM workloads. Compared with direct NPU passthrough, FlexNPU introduces no measurable inference overhead and slightly improves throughput in some scenarios. On a 384-card Ascend 910C deployment of DeepSeek-R1, FlexNPU improves throughput over static PD disaggregation by 5.15% and 26.33%. On Qwen2.5-7B, compared with static PD co-location, FlexNPU maintains comparable throughput while reducing TTFT by over 92% across tested workloads with nearly unchanged TPOT. These results show that transparent NPU virtualization is a practical substrate for efficient and responsive LLM serving.