这篇论文用DeepSeek-V4-Flash生成了上万条验证过的Polkit规则,并通过消融实验证明结构化上下文不是锦上添花,词汇表才是提升语义质量的关键。
该研究扩展了Text2DSL自动生成领域特定语言代码的工作,用DeepSeek-V4-Flash作为教师模型,在结构化上下文(BNF语法、API规范、词汇表)下生成数据,经AST验证(使用esprima)和运行时验证(polkitd与pkcheck)的流水线,将PolkitBench语料从4,204对扩充到10,073对,其中AST有效性达100.0%、运行时通过率99.7%。在GigaChat-10B-A1.8B上对八种消融条件(C0-C7)评估发现:无上下文时语法有效从97.6%降至58.5%而完整上下文仅从98.6%降至97.4%,完整上下文C7在所有指标上最优,词汇表对语义质量贡献最大(组合得分+0.198),API和BNF分别对结构有效性贡献+24.7和+22.3个百分点。
Context-Aware Distillation and Ablation for Text2DSL
We extend our prior work on Text2DSL automatic generation of domain-specific language (DSL) code from natural language descriptions along two complementary axes. First, we replace prompt-only synthetic generation with context-aware distillation, in which a teacher large language model (DeepSeek-V4-Flash) operates under an explicitly defined structured context comprising a BNF grammar, an API specification, and a closed identifier vocabulary; the resulting corpus is verified by a two-tier pipeline combining AST validation through esprima and runtime acceptance through the production polkitd daemon and the pkcheck client. This scales the verified PolkitBench corpus from 4,204 to 10,073 natural-language-to-Polkit-rule pairs at 100.0% AST validity and 99.7% runtime pass rate. Second, we conduct the per-component factorial ablation of structured context that was identified as future work in the precursor study: eight conditions C0-C7 are evaluated on GigaChat-10B-A1.8B with the new corpus. Three findings emerge. (i) The new harder corpus collapses the baseline mode (Syntax Valid 97.6% -> 58.5%, Combined Score 0.482 -> 0.252), whereas the context-enhanced mode degrades only marginally (Syntax 98.6% -> 97.4%, Combined 0.801 -> 0.750), confirming that structured context is not a cosmetic improvement but a load-bearing mechanism. (ii) The best absolute condition is the full context C7 across all metrics, while the strongest partial conditions (C5 = BNF + Vocabulary, C6 = API + Vocabulary) both contain the vocabulary. (iii) A Shapley-style decomposition assigns the largest semantic-quality effect to the vocabulary (Combined +0.198), the largest structural-validity effects to API (+24.7 pp) and BNF (+22.3 pp).