LMArena open-sources two web development evaluation datasets
Preference data of this scale for web app generation is scarce, and the pairing of subjective preference labels with verified functional requirements gives training and evaluation teams two complementary signal types. Model developers working on coding agents can benchmark against human judgment rather than proxy metrics. Comparison with existing web-dev benchmarks becomes straightforward since both datasets and the accompanying paper are public.
LMArena has open-sourced two datasets for evaluating AI-generated web applications: ArenaWebdevPreferences, containing 6,390 app pairs with human preference labels, and ArenaWebdevCorrectness, containing 189 game versions with 362 human-verified requirements. The release was announced alongside a paper on arXiv (2610.17895) and a blog post on arena.ai. The contribution team includes researchers such as Wayne Chi, Ion Stoica, Anastasios Angelopoulos, and Chou-Jui Hsieh, linking the effort to the UC Berkeley-affiliated LMArena team.
ArenaWebdevPreferences captures which of two generated apps human evaluators actually prefer, providing pairwise signals for training and benchmarking web development models. ArenaWebdevCorrectness takes a different approach: instead of preferences, it verifies whether generated apps meet specific, human-checked functional requirements, offering a stricter correctness metric.
Together the datasets address a growing evaluation gap for coding agents and web app generators, where benchmark scores often diverge from real user judgments. Teams building app-generation models can use the preference labels for reward modeling and the correctness set for requirement-level testing, both at no licensing cost since the data is open-sourced.
We're open-sourcing two datasets: ArenaWebdevPreferences (6,390 app pairs with human preference labels) and ArenaWebdevCorrectness (189 game versions, 362 human-verified requirements). Paper: arxiv.org/abs/2610.17895 Blog: arena.ai/blog/interacti… Contribution team: @iamwaynechi , @IHung_Hsu , @lichen225 , @infwinston , @istoica05 , @ml_angelopoulos , @atalwalkar , @chrisdonahuey , @cho_jui_hsieh
This report is categorized under AITOP's "Models" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include LMArena, ArenaWebdevPreferences, coding assistants, open-source models.
Source attribution: curated from lmarena.ai (@lmarena_ai).
