Qwen3-8B长上下文微调显存不够？HGA方案在16GB卡上跑16384令牌，比密集训练省显存还快一点，代码正在路上。
研究者提出分层全局注意力（HGA），结合分段反向传播和分层KV存储，仅保留当前段在VRAM中可微分，历史KV卸载到RAM或NVMe。在Qwen3-8B上使用4位QLoRA，PG19数据集，16GB Quadro RTX 5000上HGA达到16384训练长度，峰值显存15.28GB，而密集训练仅支持2048令牌。相同适配器可处理131072令牌推理。2K训练长度下，HGA与密集训练困惑度分别为2.7405和2.7383 nat，基线模型为2.9541；HGA训练速度略快（217.75 vs 207.02 tokens/s）。
Long-Context Fine-Tuning with Limited VRAM
Parameter-efficient fine-tuning reduces model and optimizer memory, but dense attention still makes long training sequences expensive. We combine Hierarchical Global Attention (HGA) with segment-wise backpropagation and tiered KV storage. Only the active segment remains differentiable in VRAM; older KV is detached into RAM or NVMe, and HGA loads a bounded set of exact historical tokens for each query block. On Qwen3-8B with 4-bit QLoRA and PG19, dense training on a 16 GB Quadro RTX 5000 fits 2,048 tokens but fails at 4,096, whereas HGA reaches 16,384 tokens with 15.28 GB peak VRAM. Under evaluation the same adapter runs through 131,072 tokens on this card; VRAM is not constant but grows gently with the resident chunk summaries, so RAM and NVMe capacity set the practical limit beyond these lengths. At the shared 2K training length, HGA-trained and dense-trained adapters obtain 2.7405 and 2.7383 nat under the same dense-attention readout, while the stock model obtains 2.9541. At this boundary HGA training is already marginally faster (217.75 vs. 207.02 tokens/s), and the HGA-to-dense throughput ratio improves from 1K to 2K; because HGA keeps the attended historical set per token approximately constant while dense work per token grows, we expect this lead to widen as context grows. Dense attention is used for the main quality and retrieval comparisons so that they measure the learned weights and remain compatible with standard generation frameworks. HGA can also be used for retrieval and generation; an optimized production-grade serving implementation is under development.