做多智能体系统编排的团队终于有了一个低成本、高回报的训练方案——OrchRM 省去了人工标注和子智能体回滚，直接提升 8% 准确率，建议做 MAS 的开发者试试这个开源框架。
多智能体系统（MAS）依赖大语言模型（LLM）进行有效编排，但训练编排器面临监督信号稀缺和计算成本高的问题。本文提出OrchRM，一种自监督框架，通过多智能体执行过程中的中间产物构建胜负对，训练Bradley-Terry奖励模型，无需人工标注。相比依赖昂贵子智能体回滚的现有方法，OrchRM直接在编排层面操作，将训练效率提升10倍（以token使用量计），并将测试时扩展的准确率提升8%。该方法在数学推理、网页问答和多跳推理等多个领域均有效，代码已开源。
Reward Modeling for Multi-Agent Orchestration
Multi-Agent Systems (MAS) built on Large Language Models (LLMs) require effective orchestration to coordinate specialized agents, yet training such orchestrators is hindered by limited supervision and high computational cost. We propose Orchestration Reward Modeling (OrchRM), a self-supervised framework for evaluating orchestration quality without human annotations. OrchRM leverages intermediate artifacts from multi-agent executions to construct win-lose pairs for Bradley-Terry reward model training. Unlike existing MAS test-time scaling and orchestrator training frameworks that rely on costly sub-agent rollouts, OrchRM operates directly at the orchestration level, enabling efficient and high-performing reward-guided orchestrator training and MAS test-time scaling. OrchRM improves training efficiency by up to 10x in token usage while improving MAS test-time scaling performance by up to 8% in accuracy. These gains consistently transfer across multiple domains, including mathematical reasoning, web-based question answering, and multi-hop reasoning, demonstrating orchestration-level reward modeling as a scalable direction for robust multi-agent orchestration. Code will be available at https://github.com/Wang-ML-Lab/OrchRM.