这篇论文用15000个专家评价告诉你, 测AI创意能力不能只看平均分, 分歧本身才是宝藏。
该论文提出人类创造力基准(HCB), 收集15000个专业判断覆盖5个创意领域和3个工作流阶段(构思/模型/细化)。HCB将评价分为收敛(如技术正确性)和发散(如美学方向), 发现专业分歧代表真实品味差异而非测量误差。单一质量指标会丢失关键信息: 模型在哪些维度必须正确、哪些维度应保持可操控性。
The Human Creativity Benchmark
Modern AI evaluation frameworks treat evaluator disagreement as noise to be resolved. In creative domains, professional disagreement reflects genuine differences in taste, not measurement error. We argue that evaluating creative AI requires preserving two distinct signals: convergence, where professionals align around shared best practices, and divergence, where individual taste legitimately varies. We present the Human Creativity Benchmark (HCB), a benchmark that operationalizes this separation by collecting pairwise preferences, scalar ratings on prompt adherence, usability, and visual appeal, and qualitative rationale from domain professionals. Across 15,000 professional judgments spanning five creative domains and three workflow phases (ideation, mockup, refinement), we find that convergence concentrates on verifiable dimensions like technical correctness and visual hierarchy, while divergence concentrates on taste-driven dimensions like aesthetic direction and conceptual risk. No model excels uniformly across all phases. Collapsing these signals into a single quality metric discards the most actionable information: where models must be correct versus where they should remain steerable.