OpenAI GPT-6 Astra Performance Analysis
Hot take on OpenAI GPT-6 Astra*, with a challenge to @gdb’s claims about it being AGI toward the end...
Astra's performance on ARC-AGI-3 raises questions about AGI claims and transparency concerns relevant to global AI development and safety standards.
OpenAI's GPT-6 Astra achieves 63% on ARC-AGI-3 benchmark and surpasses human performance on 96% of ARC-AGI-3 levels. The model builds precise symbolic models of novel environments, representing a genuine advance in AI capabilities. Marcus questions whether this constitutes AGI, noting the lack of transparency about how the system works and its reduced monitorability compared to prior systems.
Hot take on OpenAI GPT-6 Astra*, with a challenge to @gdb’s claims about it being AGI toward the end...
Hot take on OpenAI GPT-6 Astra*, with a challenge to @gdb ’s claims about it being AGI toward the end: • Looks to be pretty impressive. Multiple reports suggest it is a genuine advance. • As someone who has campaigned for nearly a decade for (neuro)symbolic world models, often to exceptional hostility, it is extraordinarily vindicating to see that a product from OpenAI explicitly creates and manipulate symbolic world models in the course of some of its computations. • What we don’t know is how robust that capability is. That is THE key question. • Success on ARC-AGI is great and impressive, but not —despite the name of the task—proof of AGI; I suspect we will see loads of problems with open-ended real world tasks. As with other recent models I would suspect best performance in verifiable domains. • And as a scientist, it’s disappointing that we don’t (yet) know much about how the system actually works. • As ever, enthusiasts got an advance look; skeptics did not. That’s a sound marketing strategy, but it often turns out to be misleading. What we have often seen is initial enthusiasm that gets tempered over time. I suspect we will see that here as well. • The new system appears to be *less* monitorable than prior systems, which is not great from a safety perspective. One really doesn’t want more capability in conjunction with less monitorability. • Would be great to see whether Astra can make progress on any of the ten tasks that Miles Brundage and I bet on at the end of 2024. (No AI to date has succeeded on any, AFAIK; link: garymarcus.substack.com/p/where-will-a… ) ——————- *This hot take is VERY tentative, pending more information about how it works and what its limitations are. ARC Prize @arcprize GPT-6 Astra by @OpenAI achieves SOTA on ARC-AGI: - Astra scores 63% on ARC-AGI-3, 99% via a new provider adapter harness - It surpasses human performance on 96% of ARC-AGI-3 levels - It builds the most precise symbolic model of novel environments we've seen Our analysis: 🔗 View Quoted Tweet 💬 17 🔄 8 ❤️ 131 👀 41822 📊 32 ⚡
