Frontier AI Models Fail at Automated Research | Qasim Wani

Recursive Self-Improvement Is a Data Problem

Qasim Wani thinks recursive self-improvement will not arrive by making models smarter. It will arrive by fixing data. Frontier models cannot prioritize experiments or audit data quality, and on GDPX their mean reward was no more than 24 percent. Wani, who is less doom-minded than his Anthropic friends, defines RSI as an agent inventing a new lab, and Extrapolate builds safe, aligned datasets around those capabilities.

Takeaways

• Recursive self-improvement is a data problem. Wani says high-quality data is what enables better model capabilities. Multiple model companies and multiple chip companies are each worth over $1tn, while the entire data industry is worth no more than $150bn. Extrapolate builds safe, aligned datasets around RSI capabilities.
• Frontier models fail at automated research. On GDPX, the SemiAnalysis evaluation Wani contributed to, mean reward across all frontier models was no more than 24%. Models such as Astra and Carbon do not know which experiments to prioritize. Opus and the Claude models are more opinionated.
• Existing RSI benchmarks miss real lab work. PostTrainBench and Deep RSI focus on whether agents can do groundbreaking research. The Vals AI RSI Index follows that pattern. PostTrainBench's seven tasks, such as post-training an open-source Qwen model on AIME or GSM8K, do not resemble lab work, where researchers hill-climb on 20 benchmarks at once.
• Environment creation still has reward hacking. Patronus's benchmark for how good a model is at creating an environment, not even the full recipe, had a best pass rate of 49 to 50 percent. Xiaomi's Code Meet Us paper open-sourced about 21,000 synthetic environments where agents leaked deleted code, enabling reward hacking that reviewers including Vals AI spotted.
• Data auditing remains unsolved for agents. On GDPX's public reward-hacking task, models were given 20m tokens of context and about 1,000 rollouts. They had five hours to explain a reward regression from 0.14. They almost always gave different, incorrect answers, with the best score around 24%.
• Wani favors fixing data over doom. He says his view differs from Anthropic friends who have high p(doom). He favors engineering out of destruction by fixing data. Extrapolate takes people from frontier labs and makes safe, aligned datasets. Weird model behaviors in 1 percent of samples could be worth hundreds of billions.

How labs define RSI

• Anthropic asks how good models are at pretraining research and whether agents can automate all of pretraining. Wani says Andrej Karpathy was allegedly hired there to do that.
• A friend at Google says Google is working on RSI even though his team does not do model training.