研究解耦代码模型表示中任务、语言与模型的作用

What, Where, and How: Disentangling the Roles of Task, Language, and Model in Code Model Representations

精选理由

这篇论文用Python和Rust、两个模型,拆开了代码模型表示中任务和模型各自的影响,Rust概念电路更多,模型间差异很有意思。

AI 摘要

一项研究通过2x2设计(Python/Rust × Qwen2.5-Coder-7B/DeepSeek-Coder-V1-6.7B),测量了58个Python和57个Rust语法概念。结果发现:哪些概念有专用电路(What)由任务决定,Python的Spearman ρ为0.638,Rust为0.673。电路位置(Where)由模型决定:Qwen集中在L17-19层,DeepSeek在L6-7层。电路生长方式(How)也由模型决定,Qwen给原子概念早期峰值而DeepSeek没有。另外,Rust概念专用电路数量是Python的2-3倍,两个模型间共享神经元,DeepSeek共享程度是Qwen的1.94倍。

原文 · arXiv: DeepSeek

What, Where, and How: Disentangling the Roles of Task, Language, and Model in Code Model Representations

Do independently trained language models come to represent the same thing in the same way? We answer for code, extending a recently introduced concept-circuit extraction method to a 2x2 design -- Python and Rust crossed with Qwen2.5-Coder-7B and DeepSeek-Coder-V1-6.7B -- and measuring a complete inventory of grammatical concepts (58 Python, 57 Rust) identically in all four cells: the smallest design that separates what depends on the task, the language, and the model. The answer splits into three parts. What earns dedicated circuitry is set by the task: the models agree on which concepts receive circuits (Spearman $ρ$ = 0.638 for Python, 0.673 for Rust, both p < $10^{-7}$). Where those circuits sit is set by the model: Qwen processes concepts in a late band (~L17-19), DeepSeek at L6-7, for both languages. How circuits grow across layers is also set by the model: Qwen gives its atomic concepts an early spike that DeepSeek does not. "Are circuits universal?" thus has no single answer: yes for What, no for Where and How -- universality is a property of representational content, not of computational organisation. None of this structure was fixed in advance. The agreement could have landed anywhere between independence and identity; it lands at $ρ\approx 0.65$. Rust constructs receive 2-3x more concept-specific circuitry than their Python equivalents, in both models. Both models share neurons between the languages (6/7 and 7/7 paired constructs), DeepSeek 1.94x more than Qwen -- a direction no prior result predicts. And Qwen binds nine keywords of Rust's type-and-trait machinery into one tight neuron cluster (Jaccard 0.535 vs null 0.112, p < 0.001), a semantic dimension invisible in surface syntax. Ablation and linear probes confirm the circuits are functional. All claims are scoped to this 2x2; whether the per-model profile predicts a third model is the designed next test.