Amazon Science 发布 Vendster：评估预测工具质量对 LLM 谈判智能体的影响
Vendster: An Evaluation Arena to Investigate How Forecasting Tool Quality Affects LLM Agent Behaviors in Buyer-Vendor Negotiation
Amazon Science 做了个开源评测场，专门测预测工具质量怎么影响 LLM 智能体谈判，跑了四个主流模型，结论挺反直觉。
Amazon Science 发布评测平台 Vendster，用于测量时间序列预测器的偏差与预测区间宽度如何影响两个 LLM 智能体之间的买卖折扣谈判结果。在基于 M5 数据集的 Vendster-M5 实例中，研究评测了 Claude Opus 4.6、GPT-5.6 Sol、DeepSeek-V3.2 和 Qwen3-235B-A22B 四个模型。结果显示模型生成的预测可将销售目标达成率提升最多 17 个百分点，但 relWQL 和 MASE 等传统指标无法显著预测谈判结果。受控合成实验发现，低预测偏差对反应更灵敏的智能体有显著收益（剂量-响应斜率 -0.387 和 -0.416，p<0.001），而偏差为零时区间宽度无显著影响。Vendster 支持完全自定义，以 CC BY-NC 4.0 协议开源。
Vendster: An Evaluation Arena to Investigate How Forecasting Tool Quality Affects LLM Agent Behaviors in Buyer-Vendor Negotiation
Time-series forecasters are ranked almost entirely by accuracy metrics, but a forecast's decision value, its effect on the downstream decision, need not track accuracy and is rarely measured. This problem compounds as large language model (LLM) agents increasingly consume forecasters as tools, mediating them through stochastic, opaque policies. We present Vendster, an evaluation arena measuring how a forecaster's bias and prediction-interval width change the outcome of a buyer--vendor discount negotiation between two LLM agents. In the reference instance Vendster-M5, sampling base demand patterns from the M5 dataset, we evaluate two frontier-closed LLM families (Claude Opus 4.6 and GPT-5.6 Sol) and two open ones (DeepSeek-V3.2 and Qwen3-235B-A22B). Model-generated forecasts raise sales target attainment by up to +17 percentage points (p<0.001), yet standard forecast evaluation metrics such as relWQL and MASE do not predict negotiation outcomes at a statistically significant level. To isolate the causal structure, we substitute in synthetic forecasts with controlled bias and interval width. Under-prediction bias significantly improves outcomes for agents that react more to the forecasts---dose-response slopes -0.387 and -0.416 (p<0.001). Interval width has no statistically significant effect at zero bias. These findings indicate that the direction of forecast bias and agent forecast-reactivity are more strongly associated with downstream outcomes than forecast accuracy. Vendster is fully customizable and released under CC BY-NC 4.0 at https://github.com/amazon-science/Vendster.