ROSTrans 基准发布：评估 LLM 完成 ROS1 到 ROS2 代码迁移
ROSTrans: Empirically Benchmarking LLMs for ROS1-to-ROS2 Migration
机器人圈的朋友看看，这个新基准专门测 LLM 做 ROS1 迁移 ROS2 的能力，六个主流模型 pass@1 都不到 11%，编译过了运行挂是重灾区。
ROSTrans 是一个针对 ROS1 到 ROS2 代码迁移的 LLM 评测基准，包含 108 个任务，取材自 70 多个开源 ROS1 仓库，并按接口、行为、系统三个层级组织正确性评估。每个任务配有可执行的 ROS2 参考实现、运行时测试套件和容器环境，候选模型无法篡改评测用例。六款前沿模型在每任务 5 次随机生成下 pass@1 均低于 10.9%，56.5% 的样本编译通过却在运行时失败，其中 32.0% 在迁移接口的运行时断言上出错。引入执行引导式修复后，Claude Opus 4.6 和 GPT-5.5 的任务成功率分别提升至 49.1% 和 47.2%。
ROSTrans: Empirically Benchmarking LLMs for ROS1-to-ROS2 Migration
Although large language models have shown strong capabilities in domain-specific code generation, their performance in robotics software development, especially for the Robot Operating System (ROS), remains largely unexplored. ROS code generation presents distinctive challenges, including complex inter-node coordination, middleware configuration, and lifecycle semantics that go far beyond standard coding benchmarks, and many important correctness properties become observable only once migrated nodes are launched and communicate. We present ROSTrans, a benchmark for evaluating LLMs on ROS1-to-ROS2 code migration, a representative and practically important setting for ROS code generation. ROSTrans comprises 108 scaffolded completion tasks drawn from a diverse set of 70+ open-source ROS1 repositories and organized into a hierarchical taxonomy assessing interface-level, behavioral-level and system-level correctness. For every task we establish an executable reference: a verified ROS2 implementation, a runtime test suite that launches the nodes and exercises their topics, services, actions and parameters, and a container ensuring reproducibility. Candidates are scored by injecting only their target files into an otherwise fixed copy of the reference package, so a candidate can neither supply nor weaken the tests it is judged by. Evaluating six frontier LLMs over five stochastic generations per task, we find pass@1 below 10.9%, and that 56.5% of all samples build successfully and then fail at runtime,and 32.0% fail runtime assertions on the migrated interfaces, a class of error no compile-only oracle can detect. Execution-guided repair further improves task success, reaching 49.1% and 47.2% for Claude Opus 4.6 and GPT-5.5 after three rounds, respectively.