GEAR：用切向-法向分解改进 LoRA 动态秩分配
From What Is Used to What Is Missing: Rethinking Dynamic Rank in LoRA
GEAR 这个新微调方法把 LoRA 动态秩问题做成了几何问题，参数只用三分之一，分数还涨了近 2 分，搞微调的可以看看。
一篇 arXiv 论文提出微调方法 GEAR，重新思考 LoRA 的动态秩机制。论文指出因子梯度的一阶信息边界无法区分当前空间内的欠优化与空间外的任务相关方向，因此用切向信息优化现有自由度、用法向谱衡量新增秩的价值。GEAR 在 LLaMA3-8B 上将平均常识任务得分从 86.61 提升到 88.45，同时把可训练参数从 42.0M 降到 14.4M。在语言和视觉多个基准上均取得平均最优成绩。
From What Is Used to What Is Missing: Rethinking Dynamic Rank in LoRA
Parameter-efficient fine-tuning with LoRA is governed by rank, which controls parameter budget and locally accessible task updates. Existing dynamic-rank methods mainly focus on capacity allocation, leaving unresolved whether the current adaptation space is sufficient for the task. We characterize the first-order information boundary of standard factor gradients, showing that their unobservable subspace coincides with the normal space of the low-rank manifold. Thus, factor gradients alone cannot distinguish under-optimization within the current space from task-relevant directions beyond its first-order reach. This motivates a geometric view of rank evolution, where tangent information optimizes existing degrees of freedom and the normal spectrum quantifies the value of new ones. Under a local quadratic model, we show that leading normal singular directions yield the locally optimal rank expansion, with spectral energy determining its gain. Building on this characterization, we propose GEAR, which turns the tangent--normal decomposition into rank evolution through tangent-space optimization, matrix-free normal-spectrum estimation for rank expansion, and task-induced geometric initialization. Across language and vision benchmarks, GEAR achieves the best average performance in all evaluated settings with substantially fewer trainable parameters. On LLaMA3-8B, it raises the average commonsense score from 86.61 to 88.45 while reducing trainable parameters from 42.0M to 14.4M.
