想做办公自动化？这个新基准用真实成本告诉你：现在LLM虽便宜但还没人类靠谱，值得看看差距在哪。
新基准 OmegaUse-OfficeVal 包含 100 个办公套件任务，平均需 2.32 小时人工完成。每个任务提供人工时间与任务价格代理两个经济信号，用于对比人类成本与 LLM 推理成本。评估了 GPT-4、Claude 3.5 等前沿 LLM，结果显示它们虽更便宜更快，但交付质量尚未达到人类水平。该基准代码与数据集已开源。
OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding
Large language model (LLM) agents are increasingly expected to assist users in completing tasks. However, existing benchmarks provide limited support for evaluating whether agents can carry out office-suite workflows at a reasonable cost. We introduce OmegaUse-OfficeVal, a benchmark for evaluating LLM agents on long-horizon office-suite tasks with task-level economic grounding. The benchmark comprises 100 tasks derived from office-suite requests proposed by practitioners and adapted through a privacy-preserving process. On average, these tasks require 2.32 hours of human labor to complete. An important feature of the benchmark is that each task is paired with two economic signals: human labor time and task price proxy. These signals enable direct comparisons between human costs and LLM inference costs, as well as value-weighted evaluation. To support stable evaluation, we develop code-based verifiers from fine-grained rubrics. We evaluate several frontier LLMs together with a human baseline. Although all evaluated LLMs are substantially cheaper and faster than human workers, they have not yet approached human-level deliverable quality. The code and dataset are fully open-sourced, and more information is available on our project website: https://omegause-officeval.github.io.