AROMA：面向 Llama-3.2-11B-Vision 的冻结 VLM 可控干预框架
AROMA: Adaptive Representation-Oriented Multimodal Actuation for Vision-Language Model Control
用 Llama-3.2-11B-Vision 做的表示干预研究，把数数这类视觉能力拆成定位、干预和目标选择几个环节分别攻克，计数准确率提升 8 个百分点且不动权重。做模型可解释性和干预控制的开发者值得看，比常见的 attribution-steering 方法设计得更系统。
论文提出 AROMA（Adaptive Representation-Oriented Multimodal Actuation），一个在冻结视觉语言模型上做表示干预的控制框架，把干预拆成定位、几何与增益、可达性、目标选择、时机五个独立问题，不修改模型权重。
在 Llama-3.2-11B-Vision-Instruct 上的实验显示核心矛盾：一个因果验证过的 cross-attention 头的四个维度解释了 89.82% 的计数敏感性，却只占 9.52% 的激活能量。作者用幅度归一化的因果子空间干预把这一错配转化为有限控制，在未见过的程序化数据上将精确计数准确率提升 8.00 个百分点。配套的迭代式最小干预策略 MEA 在相同步数预算下命中 81.25% 的 oracle 目标，而冻结方向只有 15.18%。
在目标与门控于评测前冻结的设定下，targeted MEA 在 N=630 的自然 Stage-2 队列上把此前的冻结策略从 34.444% 提升到 38.730%（配对 p=4.63e-7）；在全部 7,992 个样本上为 65.741% 对 65.153%，仅实际编辑 63 个输入。论文还包含跨头与跨模型的复现实验，以及与 attribution-steering 基线的对比。
AROMA: Adaptive Representation-Oriented Multimodal Actuation for Vision-Language Model Control
Mechanistic localization does not by itself yield usable control: causal sensitivity may occupy little activation energy, an accurate local direction may fail at finite scale, and a reachable numeral target may be wrong. We introduce AROMA (Adaptive Representation-Oriented Multimodal Actuation), a frozen-VLM control framework separating where to intervene, which geometry and gain to use, how to reach a target, which target to pursue, and when to act. In Llama-3.2-11B-Vision-Instruct, four dimensions of a causally validated cross-attention head explain 89.82% of cardinality sensitivity but only 9.52% of median activation energy; magnitude-normalized causal subspace actuation converts this mismatch into finite control, improving exact-count accuracy on untouched procedural data by +8.00 pp. Iterative Minimal-Effort Actuation (MEA) reaches 81.25% of oracle targets versus 15.18% for a frozen direction under matched step budgets. With target and gate frozen before evaluation, targeted MEA improves the previous frozen policy from 34.444% to 38.730% on a new natural Stage-2 cohort (N=630; paired p=4.63e-7) and yields 65.741% versus 65.153% over all 7,992 examples while editing only 63 inputs. Prospective routing, replication across heads and models, and comparisons with attribution-steering baselines further characterize each stage, separating localization, actuation, reachability, target quality, and selective deployment as distinct control problems.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Llama-3.2-Vision, VLM, 模型可解释性, 表示干预.
Source attribution: curated from arXiv cs.LG (@Zhaoning Zhang).