Wayne Chi Explains Interaction-as-a-Judge for Agent Evaluation
The core takeaway is that evaluation signal can be harvested from real user interactions instead of hand-written test suites, shifting agent benchmarking toward live platform data. Researchers and teams building computer-use agents can apply this pattern to score traces with LLM judges or reward models trained on user complaints. Code Arena and Agent Arena already supply the battle and complaint data, giving practitioners a concrete starting point for evaluation pipelines.
Wayne Chi, a research intern at Arena and PhD candidate at Carnegie Mellon University, presented Interaction-as-a-Judge, an evaluation approach in which computer-use agents play games and use apps to improve both evaluation and downstream generation. The talk was published by Arena, the organization behind community model comparison platforms.
The method works by having a computer-use agent navigate an application, producing interaction traces that are then judged by an LLM judge or a trained reward model. Chi covered two concrete use cases: preference evaluation using real Code Arena battles, where user preferences from head-to-head comparisons serve as judgment signal, and correctness evaluation using requirements mined from real user complaints in Agent Arena.
The approach addresses a known gap in agent evaluation, where static benchmarks often fail to capture how agents behave in real interactive environments. By grounding judgments in actual user complaints and battle outcomes from Arena platforms, the method ties agent quality measurement to authentic usage data rather than synthetic test cases.
Arena research intern and PhD candidate at Carnegie Mellon University, Wayne Chi, explains Interaction-as-a-Judge: letting computer-use agents play games and use apps to better improve evaluation and downstream generation. A computer-use agent navigates an application, producing interaction traces for evaluation by an LLM judge or a trained reward model. The talk covers preference evaluation using real Code Arena battles and correctness evaluation using requirements mined from real user complaints in Agent Arena. Watch the full talk: youtube.com/live/lu--lrbzn…📊 4 ⚡
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include Arena, computer-use agents, evaluation, reward models.
Source attribution: curated from lmarena.ai (@lmarena_ai).
