Arena.ai Open-Sources Datasets for Interaction-as-a-Judge Evaluation
Evaluation shifts from comparing static code screenshots to judging how agents actually behave inside an app, which changes how generation quality can be measured for web development agents. Builders of coding agents and eval pipelines can use the two open-sourced trace datasets to train reward models or benchmark correctness without collecting their own user complaints data. Compared with human-preference-only arenas, this adds an automated signal that scales with computer-use capability.
Arena.ai researchers, including CMU PhD candidate Wayne Chi, describe Interaction-as-a-Judge, an evaluation paradigm in which computer-use agents directly interact with games and applications so their interaction traces can be scored by an LLM judge or trained reward model. The team reports this approach improves alignment with user preferences, a model's ability to detect errors in games, and downstream generation quality.
The workflow lets a computer-use agent navigate an application and produce interaction traces used for evaluation. Preference evaluation draws on real Code Arena battles, while correctness evaluation uses requirements mined from real user complaints in Agent Arena. The team also observes gains from inference-time scaling and interaction-only reinforcement learning, with judgment quality scaling alongside the underlying computer-use model's capability.
Arena.ai is open-sourcing two agent trace datasets, ArenaWebdevPreferences and ArenaWebdevCorrectness, to support future research. As agents increasingly recreate games and applications, evaluating them by direct interaction rather than static outputs becomes a practical signal for model builders.
Agents are demonstrating increasingly strong capabilities in recreating games and applications. One likely reason is their growing ability to directly interact with these applications. We studied Interaction-as-a-Judge, a growing evaluation paradigm where computer-use agents directly interact with games and applications to improve evaluation. Interaction-as-a-Judge improves: - alignment with user preferences. - a model’s ability to detect errors in games. - downstream generation. We see improvements in inference-time scaling and the ability for interaction-only reinforcement learning. It also scales with computer-use capabilities; the better the computer-use model and method, the better the judgments. We’re also open-sourcing two agent trace datasets, ArenaWebdevPreferences and ArenaWebdevCorrectness, to help support future research. Dive into the thread to learn more. Arena.ai @arena Arena research intern and PhD candidate at Carnegie Mellon University, Wayne Chi, explains Interaction-as-a-Judge: letting computer-use agents play games and use apps to better improve evaluation and downstream generation. A computer-use agent navigates an application, producing interaction traces for evaluation by an LLM judge or a trained reward model. The talk covers preference evaluation using real Code Arena battles and correctness evaluation using requirements mined from real user complaints in Agent Arena. Watch the full talk: youtube.com/live/lu--lrbzn… 🔗 View Quoted Tweet📊 4 ⚡
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Arena.ai, computer-use agents, agents, evaluation.
Source attribution: curated from lmarena.ai (@lmarena_ai).
