Vals AI Recursive Self-Improvement Index

The index evaluates pre-training, post-training, and engineering capabilities.Unlike direct self-improvement testing which is expensive and slow, this approach uses proxies to measure research abilities.

The index aims to provide an apples-to-apples comparison across different AI models.

Vals AI CEO Rayan Krishnan on how they score a model's ability to build its own successor: "One benchmark we released recently is our Recursive Self-Improvement Index. It's a topic which a lot of the big labs have been talking about and starting to report on in their model cards." "There isn't a shared language to talk about the RSI potential of models. So we created this as an apples-to-apples way to actually benchmark across the models." "In an ideal world, what you want to do is take a frontier model and have it train the next version of itself and see where the delta comes from. But obviously that's very expensive and slow." "What we're doing is forming a set of proxies for every part of the process it takes to build the next version of the model." "There's work around pre-training, post-training, harness-level engineering, and then seeing in which mechanisms and behaviors the models are able to do very good research work and build something new, and where they're struggling." @RayanKrishnan @JenniferHli

This report is categorized under AITOP's "Models" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include Vals AI, Recursive Self-Improvement Index, benchmarking, ai models.

Source attribution: curated from a16z (@a16z).