ExplorationBench AI Discovery Benchmark
This benchmark reveals that AI systems struggle with rule discovery even when rules are provided, highlighting a critical gap in current AI capabilities. For researchers and developers working on scientific discovery systems, this provides concrete metrics to evaluate and improve experimental design and learning from feedback in novel environments.
ExplorationBench is a new benchmark for measuring AI systems' exploration capabilities.
It features two sandboxes: AlienCode with 31 hidden rule changes and 70 tasks, and AlienLogic with 24 patched inference rules and 70 theorems.
The benchmark was tested on 10 frontier AI systems, showing that getting feedback is more effective than thinking alone, with the best system reaching 89.
0% after four rounds of feedback compared to 0.5-11.
0% without feedback.
New Research: We are releasing ExplorationBench, a benchmark for measuring how AI systems explore.
Scientific discovery begins where known problems end: a system has to frame hypotheses, design experiments, and learn from the results. Evaluating this is hard. Genuinely new answers cannot be checked quickly, and in familiar domains a model can simply recall what it has seen.
Addressing this challenge, researchers from Tencent Hy, Fudan University, and Tsinghua University built verifiable Alien Worlds. Their rules are executable, so every answer is checked exactly, and they conflict with familiar knowledge, so recall alone cannot solve the tasks.
🔹 Two sandboxes: AlienCode (31 hidden rule changes, 70 tasks) and AlienLogic (24 patched inference rules, 70 theorems) 🔹 A flawed manual, four rounds of self-designed probes, and closed-book tests after every round 🔹 Every answer graded by an interpreter or a proof checker, with no LLM judge
What we found across 10 frontier AI systems: 1️⃣ Getting feedback is more effective than thinking alone. No AlienCode run starts above 15.7%; after four rounds the best reaches 89.0%, while the same turns without feedback stay at 0.5–11.0%. 2️⃣ Designing the experiments matters. Replaying a system's own best probes gives it exactly the same evidence, yet in AlienCode 9 of 10 systems do worse than when they chose the probes themselves. 3️⃣ Knowing a rule is not using it. Even when every required rule is stated correctly, tasks are solved only 73.4% of the time. 4️⃣ One score hides a lot. The same system under the same budget ended anywhere from 5.7% to 79.0%, and rankings barely transfer between the two worlds.
CL-bench asked whether models can learn from context. ExplorationBench asks whether they can discover the rules themselves.
📄 Paper: https://t.co/eamwrsJyxQ 🌐 Website & leaderboard: https://t.co/gxSeoCfc2s 📝 Blog: https://t.co/jyrwH7ZnQH 💻 Code (coming soon): https://t.co/i7VlgErkcl
This report is categorized under AITOP's "Papers" desk, focusing on China's AI industry and domestic foundation model ecosystem. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include ExplorationBench, Tencent, AlienCode, AlienLogic.
Source attribution: curated from 腾讯混元 (@TencentHunyuan). Cross-source corroboration: 2 independent sources have covered this development (including Hunyuan), confirming industry-wide signal strength.
- Hunyuan09-30 13:29Source