想知道怎么给LLM设计奖励函数来提升生成质量？这篇论文用38个指标和两个模型做了系统对比，结论很具体。
这篇论文研究了奖励函数设计对基于强化学习（RL）的流程模型生成质量的影响。研究者使用Llama 3.1 8B和Qwen 2.5 14B两个模型家族，在48种配置下训练，评估框架包含38个句法、语用和语义指标。实验发现RL能同时提升语用和句法质量，输出变异性降低6倍以上。等权重奖励始终优于定向加权，后者可能导致模型崩溃。设计选择与模型架构存在非平凡交互，例如无效惩罚对Llama 3.1必要但对Qwen 2.5无关。
Improving LLM-Generated Process Model Quality Through Reinforcement Learning: The Role of Reward Function Design
Large language models (LLMs) can generate BPMN process models from natural-language descriptions, yet supervised fine-tuning (SFT) limits their output quality to the patterns present in the training data. Reinforcement learning (RL) can optimize beyond this ceiling using external quality measures, but how the reward function should be designed when quality is multi-dimensional remains unexplored. We present a systematic investigation of reward function design for RL-based process model generation, training two LLM families (Llama~3.1 8B, Qwen~2.5 14B) under 48 configurations using Group Sequence Policy Optimization with rewards derived from an automated evaluation framework comprising 38 metrics across syntactic, pragmatic, and semantic quality. Three findings emerge. First, RL significantly improves pragmatic and syntactic quality while preserving semantic fidelity, reducing output variability by more than sixfold. Second, equal reward weighting consistently outperforms targeted weighting: emphasizing a specific dimension fails to improve it and can collapse the model into a low-quality mode. Third, design choices interact with model architecture in non-trivial ways: the invalidity penalty is essential for one model but irrelevant for the other, and SFT initialization is indispensable for one architecture but counterproductive for another. These results demonstrate that reward composition is a primary determinant of optimization outcomes, with effects as large as the decision to apply RL itself. The findings generalize to any structured generation task where quality is assessed along multiple automated dimensions. We release our implementation and experimental code at https://github.com/chlauer99/RL_for_process_modeling.