Hibana：面向统一内存设备的本地智能体推理引擎
Hibana: Fast Agentic Inference on Unified-Memory Devices
想在本地跑大模型智能体的开发者可以看这篇。它专门针对 DGX Spark 这类统一内存设备优化，跑 125B 的 Qwen、Nemotron 比 vLLM、llama.cpp 快 3 到 4 倍，还能保存智能体状态随时恢复，代码也开源了。
Hibana 是一个专为统一内存设备设计的推理引擎，在单台 DGX Spark 上可服务最高 125B 参数的 Qwen 和 Nemotron 模型。相比 vLLM、NVIDIA Flare、llama.cpp 和 TensorFold，它每小时完成最多 3.4 倍的智能体任务，解码吞吐量最高提升 3.9 倍。现有主流引擎如 vLLM 和 SGLang 面向配备独立主存的数据中心 GPU，本地引擎则未能充分利用统一内存设备的内存层级。
技术上，Hibana 围绕统一内存系统级芯片的共享内存容量和有限带宽做了三项协同优化：通过模型手术（model surgery）修改训练后的模型架构以缩减内存占用和数据搬运；统一内存管理保留每个智能体跨轮次的状态，使长时运行的智能体可以即时恢复而无需重算；自适应投机解码在每步生成多个 token。代码已在 GitHub 的 sharar-project/hibana 仓库开源。
Hibana: Fast Agentic Inference on Unified-Memory Devices
Local AI promises capable personal assistants that run on users' own devices, keeping data and computation under their control. Open-weight models that fit on local devices can now handle practical tasks such as coding. Yet current inference engines fall short. Popular engines such as vLLM and SGLang target datacenter GPUs with dedicated host memory, while local engines underuse the memory hierarchy of unified-memory devices. We present Hibana, an inference engine for unified-memory devices that co-optimizes model representation, memory management, and decoding strategy. Model surgery, the process of editing a trained model's architecture to suit its hardware, changes how the model computes, shrinking its memory footprint and data movement. Unified-memory management preserves each agent's state across turns, so long-running agents can resume instantly instead of recomputing their state. Adaptive speculation generates several tokens per step. Hibana codesigns these features around the shared memory capacity and limited bandwidth of unified-memory systems-on-chip to support local multi-agent workloads. On a single DGX Spark serving Qwen and Nemotron models of up to 125B parameters, Hibana completes up to 3.4$\times$ more agentic tasks per hour and delivers up to 3.9$\times$ higher decoding throughput than other established frameworks such as vLLM, NVIDIA Flare, llama.cpp, and TensorFold. Code is available at https://github.com/sharar-project/hibana.
This report is categorized under AITOP's "Models" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include Hibana, vLLM, llama.cpp, DGX Spark.
Source attribution: curated from arXiv cs.AI (@Yahya Emara, Chi-Chih Chang, Anthony Fei, Ahmed F. AbouElhamayed, Chong-Yan Chen, Hung-Yueh Chiang, Kai-Chiang Wu, Mohamed S. Abdelfattah). Cross-source corroboration: 11 independent sources have covered this development (including IT之家, vLLM, techcrunch), confirming industry-wide signal strength.
