Text2DSL: LLM-Based生成领域特定语言代码的新任务与PolkitBench数据集

Text2DSL: LLM-Based Code Generation for Domain-Specific Languages

精选理由

这篇论文定义了Text2DSL任务,带了一个4204条规则的数据集PolkitBench,还发现喂给模型语法规则能让代码生成质量暴增,不用微调。

AI 摘要

论文将自然语言到DSL代码生成定义为Text2DSL新问题,并引入PolkitBench数据集,含4204对自然语言-Polkit规则对。实验在GigaChat-10B-A1.8B(18亿活跃参数)和Nemotron-3-Nano-30B-A3B(30亿活跃参数)两个MoE模型上测试。提供结构化上下文(BNF语法、API说明、允许标识符词汇)后,语法有效性达98.6-99.4%,结构有效性提升9.7-35.5个百分点,CodeBLEU分数提升60-95%。

原文 · arXiv cs.AI

Text2DSL: LLM-Based Code Generation for Domain-Specific Languages

Domain-specific languages (DSLs) are widely used for managing operating system security policies, yet manually authoring rules in such languages demands high expertise and is error-prone. This paper formalises the task of automatic DSL code generation from natural language descriptions - Text2DSL - as a distinct problem class, separate from Text-to-SQL and general-purpose code generation. We introduce the PolkitBench dataset comprising 4,204 verified natural-language-to-Polkit-rule pairs, each validated through a three-level AST-based pipeline. Controlled prompt experiments on two MoE models of different scale and provenance - GigaChat-10B-A1.8B (1.8B active parameters) and Nemotron-3-Nano-30B-A3B (3B active) - demonstrate the critical role of structured context (BNF grammar, API specification, permitted identifier vocabulary) for LLM-based DSL code generation. Across both models, supplying context raises syntactic validity to 98.6-99.4%, structural validity by +9.7 to +35.5 pp, and the CodeBLEU score by +60% to +95%. The consistency of the effect across models of different scale and provenance indicates that, for the Text2DSL class of problems, injecting a formal target-language specification into the prompt context is a robust enabling factor for high-quality generation without model fine-tuning.